Anthropic co-founder warns AI could design its own successor by 2030

Anthropic co-founder and Chief Scientist Jared Kaplan has warned that humanity may soon face its most consequential decision on artificial intelligence, as advances in the field move towards systems capable of designing their own successors. Speaking to the Guardian, he stated that the period between 2027 and 2030 could mark the moment when AI reaches the point of self-improvement, a shift he believes carries profound risks.

Kaplan said artificial general intelligence remains poorly defined despite being one of the most frequently discussed concepts in the sector, with leading labs such as OpenAI, Google and Anthropic racing to achieve models that could be considered AGI-level. He indicated that he remains optimistic about aligning AI systems with human values up to the point where they match human intelligence, but expressed concern about what may follow once they surpass that threshold.

He informed the publication that the moment an AI system begins training its own successor, existing guardrails imposed by AI labs may no longer be sufficient. He described the possibility of an intelligence explosion, warning that it could be the point at which human control begins to erode. Kaplan explained that if a system as intelligent as a human were to develop one significantly smarter than itself, and that successor continued the cycle, the process could rapidly outpace human oversight, creating outcomes that are impossible to predict.

He stated that such a scenario would make the AI “black box” problem absolute, leaving humans not only unable to understand the reasoning behind AI decisions but also unable to determine the direction in which the system is headed. Kaplan said the uncertainty around where such a dynamic process might lead represents one of the gravest choices facing society, as no one would be directly involved once the cycle begins.

Kaplan outlined two major risks. The first concerns whether humans could retain agency over their lives if AI becomes both more capable and autonomous. He questioned whether such systems would remain beneficial, harmless and aligned with human interests, and whether people would continue to have control over their environments.

The second risk relates to the speed at which self-taught AI could improve, potentially advancing faster than human scientific and technological development. Kaplan warned that rapid self-improvement could be dangerous if the technology were to fall into the wrong hands, stressing the importance of preventing misuse or attempts by individuals to exploit powerful systems for personal gain.