Artificial General Intelligence (Super-intelligence)

Publication of Superintelligence (Oxford University Press, 2014) by Nick Bostrom, the Oxford philosopher then running the Future of Humanity Institute, popularised the scenario of a self-improving AGI (artificial general intelligence — AI that matches or exceeds human cognitive ability across domains) that could rapidly outstrip human intellect and pursue goals lethal to humanity. The classic worry, sketched by Bostrom in thought experiments like the "paperclip maximiser," is that a sufficiently capable AI need not be malicious to be dangerous — just competent at a goal that doesn't quite match what we actually wanted. Physicist Stephen Hawking told the BBC in December 2014 that such an AI "could spell the end of the human race." Two months earlier, Elon Musk had told an MIT audience that developing AI was like "summoning the demon," calling it our "biggest existential threat." A January 2015 open letter from the Future of Life Institute (a non-profit focused on reducing catastrophic and existential risks), signed by Hawking, Musk and thousands of AI researchers, called for research to ensure AGI would be "robust and beneficial."

What was once a fringe academic concern has since become mainstream policy. The May 2023 Center for AI Safety Statement on AI Risk — a single sentence reading "mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war" — was signed by Turing Award winners Geoffrey Hinton and Yoshua Bengio, OpenAI CEO Sam Altman, Google DeepMind CEO Demis Hassabis, and hundreds of other leading researchers and executives. The existential-risk framing now underpins the UK and Seoul AI Safety Summits (intergovernmental meetings launched in 2023–24), the EU AI Act, and the US and UK AI Safety Institutes, government bodies set up to evaluate frontier AI systems. This 2010s panic links directly to the contemporary Generative AI era that followed.