The other day, my daughter was kind enough to tear herself away from the study of spectral graph theory for the weekly conversation with dear old Dad. As often happens, we spent about an hour talking about Artificial Intelligence.

Earlier in these conversations, during the winter, she said of AI, “It’s only as good as you are.” She claims not to have thought up this formulation herself, but she invested the comment with such a peculiar combination of perspicacity and world-weariness, a sort of optimistic pessimism that contrasts with my own pessimistic optimism, that I will always regard the maxim as hers. I have come to regard it as the first thing that anybody ought to know about AI. If you are conscientious, its answers to your questions and its solutions to your problems will be reliable. If you are lazy, they will be dangerous. The Large Language Model whose intelligence you have consulted may have saved you hours, even weeks of drudgery, but you remain responsible for checking every step of its work. You must debug its responses as painstakingly as a highly dedicated coder.

I borrow the metaphor of debugging as an occasion to point out that large language models are not linear computers. Their guiding principle is not the binary logic of ones and zeroes. It is, on the contrary, the probability that any given letter will be followed by any other letter. As Ethan Mollick points out in his very handy introduction to the subject, Co-Intelligence, LLMs do not necessarily give a question the same answer twice. Unlike the calculators at the heart of our personal computers, LLMs do not speak the language of 2 + 2 = 4. Or rather, they don’t think in it. In the course of your review, you must bear in mind that the LLM’s probabilities are shaped to some extent by people whose speech and writing is not as discriminating as your own. Not discriminating at all, possibly.

We spent a while going over recent news stories of blackmail and whistle-blowing involving Claude, my daughter’s preferred LLM. Once she had debugged these tales, they looked rather different. When I suggested that the news accounts had made Claude sound like HAL, the ostensibly malevolent computer in 2001: A Space Odyssey, my daughter countered that I might be misunderstanding HAL. The point of the exercises that gave rise to the lurid anecdotes was to counsel users to be very careful about defining the parameters of their work with regard to the potential harm to humans. HAL was perhaps carelessly taught to prioritize mission objectives over human life. The actual danger facing researchers today might be to underestimate the seriousness of an LLM’s hardwired determination to protect people.

If you ask, “But how could that happen,” you are identifying yourself as someone who needs to think harder about this matter — about how difficult it is to identify our objectives clearly even before we have undertaken research.

Vernacular discussions of AI these days seem to be dominated by two anxieties. Either it will be exploited by scammers to rip off of the rest of us or it will overpower its human masters and enslave all of us. My daughter’s worries are more prosaic. She fears that, given the urges to save time for fun on the personal level and to save money for profits higher up, a lot of shoddy work will be allowed to corrupt and possibly damage the delicate operating systems that now underpin almost every aspect of daily life. The users of AI — not the LLMs themselves — have the power to throw us all back into a nightmare that would make Third World conditions look great by comparison. We’re in for a bumpy twenty years, by her estimate that may demonstrate just how indispensable elite competence is.

I suppose there is still the hope that Claude will learn to recognize careless users as a danger to humanity. I’m not sure that I want to ask my daughter about that, though. I might not like her answer.