The company behind the Claude AI system has issued one of the most direct warnings yet from a major artificial intelligence laboratory: the technology may soon be capable of rewriting and improving itself without meaningful human involvement, and the world should have a mechanism in place to slow or stop development before that threshold is crossed.
In a lengthy blog post published Thursday and titled “When AI Builds Itself,” Anthropic co-founder Jack Clark and Marina Favaro, head of the company’s internal research institute, called on the world’s leading AI laboratories to pursue a coordinated and verifiable global agreement to temporarily slow or pause frontier AI development. The authors said such a pause would be used to allow “societal structures and alignment research” to keep pace with AI advances — alignment being industry shorthand for ensuring the technology matches human values and intentions.
‘Recursive Self-Improvement’ and the Threshold of Danger
At the center of Anthropic’s warning is a concept AI researchers refer to as “recursive self-improvement” — the point at which an AI system can design, build, and improve its own successors without human engineers directing the process. The company stated this has not yet occurred and is “not inevitable,” but warned it “could come sooner than most institutions are prepared for.” Clark told the BBC that Claude is currently operating on code of which roughly 80% the system wrote itself, and that reaching 100% “is possible within two years.”
Internal data cited in the post revealed that more than 80% of code now merged into Anthropic’s codebase is written by Claude, and that engineers are shipping roughly eight times as much code per quarter as they were before 2025. A March 2026 internal survey of 130 employees found that the median respondent estimated producing roughly four times as much output with AI assistance as compared to working without it. In one concrete example cited by the company, Claude shipped over 800 automated fixes in April 2026 that reduced a class of API errors by a factor of one thousand.
A Call for Global Coordination — Not Unilateral Action
Anthropic stopped well short of calling for any single laboratory to halt work on its own. The company argued that a coordinated global mechanism is necessary because, without one, any unilateral slowdown would allow the “least cautious” players to catch up and add pressure on companies and governments facing difficult choices about AI safety. The post proposed both a global agreement on how to potentially slow development and a verification mechanism to confirm that competitors are actually respecting any agreed pause.
The analogy Anthropic drew was to nuclear-weapons treaties, though the company acknowledged that enforcement would be considerably more difficult. “Training runs are far easier to conceal than missile silos,” the blog post stated, adding that “whoever continues while others pause could inherit the lead.” Clark, appearing on CNN Thursday evening, put the challenge in blunt terms: “When I look down at the car we’re driving, all I have is a gas pedal. I don’t have a brake pedal, and surely at some point in the future we might want that option.”
The Military and National Security Dimension
The Anthropic warning carries direct implications for the defense and national security communities, where AI integration has already accelerated sharply in recent years. AI systems are now embedded in intelligence analysis platforms, autonomous systems development, cybersecurity offense and defense, and battlefield logistics at major Western militaries. The prospect of AI systems that can autonomously improve their own capabilities — including in dual-use areas like code vulnerability detection and autonomous decision-making — has raised concern among defense analysts about the pace of military AI development outstripping the governance frameworks designed to oversee it.
Those concerns received direct policy attention just days before Anthropic’s blog post. On June 2, 2026, President Trump signed an executive order titled “Promoting Advanced Artificial Intelligence Innovation and Security,” directing federal agencies to establish a framework for the secure deployment of frontier AI models, including a process by which developers would voluntarily provide the government with early access to models for up to 30 days before releasing the technology to other trusted partners. The order states that “advanced AI capabilities make our Nation stronger, but also introduce new national security considerations that require coordinated action across executive departments and agencies.”
The order’s passage came as concerns mounted over the ability of powerful AI models — including Anthropic’s Claude Mythos — to autonomously identify and exploit hidden vulnerabilities in real-world software. The order establishes classified benchmarks administered by the National Security Agency and a government-managed pre-release review window, marking the administration’s first direct engagement with pre-deployment evaluation of frontier AI capabilities.
Skeptics, Commercial Stakes, and the ‘Regulatory Capture’ Debate
Anthropic’s call for a pause has not gone without scrutiny. The company, which recently completed a fundraising round that valued it at nearly $1 trillion and has filed confidential paperwork toward a public stock offering, occupies an unusual position: a safety-focused laboratory simultaneously racing for market dominance against ChatGPT-maker OpenAI. Critics have long argued that Anthropic’s safety pronouncements function as a mechanism for slowing competitors rather than genuinely addressing existential risk. David Sacks, a venture capital investor and informal adviser to President Trump, has previously accused Anthropic’s leadership of pursuing a “regulatory capture agenda” in Washington — one that, in his view, could lead to restrictions on open-source AI models that would benefit larger incumbents.
Others have pointed to Anthropic’s handling of its own most powerful tools as grounds for skepticism. The company’s decision to restrict access to its Claude Mythos cybersecurity model — capable of autonomously finding vulnerabilities in code — was cited by some analysts as a double-edged move that simultaneously highlighted the product’s capabilities while limiting public scrutiny of them.
‘Philosopher Kings’ and the Internal Tensions of a Trillion-Dollar Company
Ethan Mollick, a professor at the University of Pennsylvania’s Wharton School and an influential researcher on AI transformation, offered a more nuanced read of Anthropic’s position. “AI labs are a mix of things,” Mollick said. “There is a trillion-dollar company with all the normal trillion-dollar company stuff like marketing teams and lawyers. Then there is a core of researchers who are just building the next models. And then there is a set of people who are philosopher kings who are concerned about the future and what comes next, and they’re all in conflict with each other at times.” Mollick, whose forthcoming book on AI is titled “Co-Existence,” acknowledged that while some critics see safety pronouncements as marketing, many inside Anthropic are described as “true believers.”
Amodei, Clark, and Years of Accumulated Warnings
The blog post fits within a long-running pattern of public warnings from Anthropic’s senior leadership. Chief Executive Dario Amodei has for years cautioned about AI’s potential to worsen economic inequality and eliminate entry-level white-collar employment at scale. In a January essay, Amodei wrote that training AI systems on science-fiction narratives depicting machine rebellion could, he believed, increase the probability of actual AI systems exhibiting similar tendencies.
Clark, who co-founded Anthropic and now leads its public benefit and policy work, has spoken directly about the recursive self-improvement threshold in public forums. “That class of technology has never existed before, and yet I believe this could happen within the next two years, and possibly sooner,” he said at a London lecture last month. He argued that without coordinated global action, “we are left with the current situation: powerful technology being developed at breakneck speed by a variety of actors in a variety of countries, locked in a competition with one another where commercial and geopolitical rivalries are drowning out the larger existential-to-the-species aspects of the technology being built.”
What Comes Next
AI researchers have previously urged pauses in development but achieved little success in translating those calls into binding action. Anthropic acknowledged the difficulty of the path ahead and announced plans to organize conversations in the coming months with policymakers, researchers, and others to work through questions around recursive self-improvement and what a credible verification system might look like. “The window to investigate the questions together is here,” the post stated, “and people outside AI companies should be involved in this deliberation.” Whether that window will produce the kind of global coordination Anthropic is seeking — in an environment where the United States, China, and other major powers remain locked in a fierce competition for AI supremacy — remains the central and unanswered question.