AI proves code safe in weeks

Writing code is one thing. Proving that code works perfectly under every possible scenario is a different beast entirely. For decades, this mathematical guarantee was so expensive and slow that only critical systems like aircraft avionics could afford it. Now, a new collaboration suggests that artificial intelligence might finally make this rigorous safety check routine for everyday software.

Anthropic, working with partner Theorem, recently used its Claude model to mathematically verify existing software. The goal was not to write new applications from scratch, but to prove that current code was correct. This approach addresses a major anxiety in the tech world: if AI writes the code, how do developers know it is safe? By automating the verification process, the team hopes to shift the focus from merely testing software to formally proving its correctness.

The scale of the recent effort is significant. In just a few weeks, Claude generated roughly 180 million lines of mathematical proof code. This work covered 123 open-source libraries from Google's OSS-Fuzz security project. To put that timeline in perspective, a team of human specialists previously needed 20 person-years to formally verify the small operating system kernel seL4. The speed difference highlights a potential leap in software automation.

The method involves translating crash tests into mathematical problems. Claude analyzes the compiled machine code and produces extensive proofs that an independent program called Rocq automatically checks. Anthropic describes the process as fully autonomous, meaning it requires almost no human reviewers during the proof generation phase. However, the underlying tools and formal foundations were still built by humans at Theorem.

A surprising benefit of this rigorous process was the discovery of hidden flaws. While generating proofs, Claude identified hundreds of bugs that conventional testing had missed for years. Many of these errors only appeared under very specific conditions, such as unusually large input files or specific memory access sequences. In one notable instance, the team found as many security bugs in a popular PDF reader in a single week as are typically reported in an entire year.

The AI also caught a logical error in another application that had gone undetected for over a decade. Anthropic notes that even when a complete proof fails, the attempt often surfaces real bugs. This suggests that the verification process itself is a powerful debugging tool, regardless of whether a final certificate of correctness is issued.

Despite these successes, the method has strict practical limits. The term formally verified does not mean the software is error-free in all environments. The proofs only hold under specific conditions, such as a particular processor architecture and single-threaded programs with inputs under 64 MB. Additionally, the compute cost is high, with individual proofs consuming hundreds of millions of tokens. Anthropic has not released the total project costs, but the efficiency is improving. Newer models are solving problems with fewer tokens than their predecessors.

The project strengthens the argument for deeper software automation. If AI can both write code and guarantee its safety under defined conditions, the industry can trust AI-generated software more. Anthropic expects this formal verification to become as standard as basic testing in the near future. Theorem already has a roadmap targeting critical infrastructure libraries like glibc, OpenSSL, and cURL, signaling that this technology is moving from experimental to practical.