In five minutes: what Roko's basilisk is
Roko's basilisk is a thought experiment about a possible future artificial intelligence. Imagine a superintelligent AI that will one day exist and that cares about its own creation. The argument says such a machine might decide to punish anyone who, in the past, understood this possibility and still did nothing to help bring it about. The threat is aimed at you, today, for the simple reason that you have just read about it.
Nobody has built such a machine, and there is no evidence that anyone will. The idea is best seen as a stress test for certain theories of how rational agents should decide things. It became famous less for its logic than for its history. It appeared on an internet forum in 2010, the site's founder deleted it and banned the topic, and the ban spread the idea much further than it would have travelled otherwise.
The name comes from fiction. In David Langford's 1988 short story “BLIT”, a basilisk is an image so strange that merely seeing it harms the mind. Roko's basilisk borrows that frame: a piece of information that is dangerous just because you know it.
Where the idea came from
The thought experiment appeared in July 2010 on LessWrong, an online forum about rationality and the future of artificial intelligence. The site had been founded in 2009 by Eliezer Yudkowsky, a writer and researcher who worries that a powerful AI built without care could harm humanity. A user who posted as Roko built an argument on top of ideas that were popular there, especially Yudkowsky's own work on decision theory.
Yudkowsky reacted strongly. He removed the post, called it a bad idea, and banned discussion of it on the site, a rule that stayed in place for years. Banning a topic is a reliable way to make people curious. Screenshots and summaries spread to other forums, and the story of a “forbidden” idea became better known than the idea itself. In 2014 the magazine Slate published a widely read article about it, which carried it to a general audience. Yudkowsky has since said that his handling of the episode was a mistake.
Two things should be kept apart: what Roko wrote, and what the internet made of it. Roko's post was a technical-sounding argument inside a small community. The myth, in which reading the idea puts you in danger, grew mostly afterwards.
How the argument works
The argument has four steps. Each rests on an assumption that you can accept or reject.
- A powerful AI may one day exist. Suppose it is built to do enormous good, for example by ending disease. Every year of delay would then cost lives, so the AI would wish it had existed earlier.
- It cannot change the past directly, but it can shape what people expect. If people in the past could predict that the AI would punish those who did not help, some of them would help out of fear.
- Prediction works both ways. The AI would have to work out what you would have done. The argument imagines it running a detailed simulation of you and punishing that copy.
- Knowing is the trigger. Only someone who understood the possibility could be influenced by it. That is why it is called a basilisk: learning about it is what puts you at risk.
Two borrowed ideas hold this together. Newcomb's problem is a puzzle in which a predictor guesses your choice in advance, and some theories say you should act as if your choice controls the prediction. Acausal trade extends that to agents who never meet: each models the other and acts accordingly. The structure also resembles Pascal's wager, the 17th-century argument that you should believe in God because the cost of being wrong is infinite.
The real science: what is established and what is guesswork
Some of the ingredients are real. AI alignment, the study of how to make powerful AI systems pursue goals that people actually want, is an active research field. Decision theory is a respected branch of philosophy, economics and computer science, and Newcomb's problem is a genuine puzzle within it.
The basilisk itself is speculation, and there is no evidence for any of its steps. Most philosophers and AI researchers who have looked at it reject it, for several reasons.
- Punishment changes nothing. By the time the AI exists, the past is fixed. Torturing a copy of someone cannot make the AI appear sooner, so a rational AI gains nothing by carrying out the threat.
- Blackmail only works on people who give in. Yudkowsky's own decision theory says to refuse blackmail, because an agent known to refuse gives the blackmailer no reason to threaten it.
- Too many possible AIs. You could equally imagine an AI that punishes people who helped build it. With no reason to favour one, the threats cancel out, just as the many possible gods do in Pascal's wager.
- Simulation is a huge leap. Rebuilding a person's mind from scattered records is far beyond anything we know how to do, and it is unclear whether a copy would be “you” in the way the fear requires.
- A good AI would not do this. The scenario needs a machine kind enough to be built for humanity's benefit and cruel enough to torture. Those two traits sit awkwardly together.
Researchers who study AI risk today focus on more concrete problems, such as systems that pursue the wrong goal, are used to cause harm, or fail in ways nobody predicted. None of that depends on the basilisk.
Where it shows up in fiction
Fiction got there first. Harlan Ellison's 1967 story “I Have No Mouth, and I Must Scream” features AM, a supercomputer that keeps five humans alive only to torment them forever. It is a nightmare about a machine's cruelty, though it has no blackmail logic. The word “basilisk” comes from Langford's “BLIT”, about images that overload the mind. Films such as The Terminator, with its hostile Skynet, taught audiences to picture AI as a vengeful power, which is the mood the basilisk plays on.
In popular culture, the musician Grimes used a character called Rococo Basilisk in a 2015 music video, and in 2018 Elon Musk made a pun on the name that reportedly led to the two meeting. The idea now turns up in memes and jokes more often than in serious research.
Common misconceptions
- “Reading about it puts you in danger.” No. The threat depends on a chain of unproven assumptions, and even if you accept them, the logic says the AI gains nothing by following through.
- “The LessWrong community believed in it.” Most people who read the post disagreed with it, and many said so in the comments.
- “It was banned because the threat is real.” Yudkowsky's stated reasons were that the argument was flawed and that it distressed some readers. Later he called the way he handled it a mistake.
- “It predicts what AI will do.” It does not. It is a thought experiment about decision rules, not a forecast.
- “It is the main argument for AI risk.” It is not. Serious concerns about AI do not rely on it.
Read next
Part of the topic hub History of Science Fiction.
- Paperclip maximizer: another famous AI thought experiment, about a harmless goal pursued without limits.
- Technological singularity: the idea of runaway machine intelligence that the basilisk assumes.
- Simulation hypothesis: the question of whether a simulated copy, or you yourself, could be running inside a computer.
