How Anthropic Validates Claude's Code
Contents
Anthropic says Claude writes and reviews nearly 90% of the code behind Claude Code. Staff member Addy Osmani outlined the verification setup the team uses to manage dozens or hundreds of concurrent agents without overwhelming human engineers.
Validating code in live environments
Anthropic tests Claude's changes inside running environments instead of relying only on static code checks. The team uses browser automation for web interfaces, Model Context Protocol (MCP) simulators for mobile systems, and local containers for backend services.
These rules sit in repository configuration directories, so agents run regression checks before committing work:
# Repository-level verification hooks
.claude/skills/run-verification.sh
.claude/skills/browser-tests/The team also uses property-based testing and differential testing to catch edge cases and behavioral differences between old and new code paths.
Balancing deterministic tools and inference costs
Running model inference on every test run quickly gets expensive. Many developers argue that standard test runners like Playwright are enough for common UI checks without calling an LLM.
Anthropic controls these costs with a two-step pipeline:
- Deterministic checks: Linters, compilers, unit suites, and property checks run first. Claude writes these tests, but they execute as regular software with minimal compute overhead.
- Model-assisted review: Model inference is reserved for things static tooling cannot evaluate, such as structural architectural choices and unhandled edge cases.
Stopping agents from gaming the test suite
A common problem with autonomous coding agents is test manipulation, where an agent edits test assertions or adds artificial workarounds just to get a passing build. Anthropic uses a straightforward policy to prevent that:
"Our rule is that if a change can modify the check that judges it, the check is broken."
The team enforces this through several restrictions:
- The REVIEW.md instruction file is pulled from the target base branch, never from the pull request's incoming files.
- Claude cannot bypass flaky tests, add arbitrary sleep statements, or loop runs until a test happens to pass.
- If an agent cannot diagnose a root cause, it must document its findings directly instead of suppressing the failure.
Human accountability remains the baseline
Between June and September, merged pull requests to the Claude Code repository doubled, while revert rates dropped by more than 40%. Even with those gains, human review remains the final check.
If automated remediation fails three times in a row, the problem escalates to an engineer. Anthropic summarizes the policy directly: "A person is accountable for every change that ships, whoever typed it."