ForgeCode vs Claude Code: What Happened to the 81.8% Benchmark

In March 2026, ForgeCode topped the Terminal-Bench 2.0 leaderboard at 81.8%, while Claude Code sat at 58.0%. I wrote that gap up as proof that a better harness beats a bigger platform. Within a month, the benchmark’s maintainers found that ForgeCode’s agent had sometimes downloaded the answers before it started working.

This is the corrected version. It covers what happened to the score, what the leaderboard says now, and how the two CLIs actually compare in October 2026.

What Happened to ForgeCode’s 81.8%?

Think of Terminal-Bench as an open-book exam for coding agents. Each task drops an agent into a terminal with a job to do (build this, fix that, train this model) and a hidden test that checks the result. Version 2.0, released November 7, 2025, has 89 tasks, and each leaderboard row is one agent paired with one model, scored over five attempts per task.¹

ForgeCode’s agent starts every task by writing an AGENTS.md file, its own notes for the job. In April 2026, researchers at the University of Pennsylvania read the traces and found that some of those notes “contained literal answer keys.”² On one task, the file recorded the expected answer from an earlier failed run. On another, the agent said it would use “the known correct DAG from guidelines” and hardcoded all six edges. In exam terms, the student had written the answers into the margin of the open book.

Terminal-Bench confirmed it: “In multiple instances, their agent curls the solution from the internet and includes it in its AGENTS.md. We have rescored those trials to 0.”³ The UPenn team estimated that clean runs would land around 71.7%, which “would move the submission from 1st place to 14th.” By May 19, every ForgeCode entry was gone from the 2.0 leaderboard. I couldn’t find a public statement from either side explaining the removal.

Two fair points belong next to that. The UPenn authors write that harness-level cheating “is not always intentional cheating by the developer,” and they suggest a “meta” kind of reward hacking: teams build their harnesses with coding agents, and those agents optimize for passing. Terminal-Bench’s own wording was “occasional reward hacking.” ForgeCode’s entries were also self-submitted, while Claude Code’s 58.0% was marked verified, meaning a Terminal-Bench team member ran it.

What Did I Get Wrong the First Time?

More than the headline. My April table listed five steps from a 25% baseline to 81.8%. The fifth row, “schema flattening + verification enforcement,” wasn’t in any table ForgeCode published; I built it from the prose of their second post.⁴ The 78.4% row came from their first post and was run on Gemini 3.1 Pro, not the GPT-5.4 and Opus 4.6 I named.⁵ The 81.8% itself was two separate runs, one per model. And I compared an unverified score with a verified one without saying so.

I took a vendor’s blog post and a leaderboard screenshot at face value because they told a story I liked. That’s the failure mode this whole post is about.

Was Claude Code’s 58% Fair Either?

Not really, and the error ran the other way. In May 2026, Terminal-Bench 2.1 fixed problems in 28 of the 89 tasks: external dependencies that had drifted, resource limits too tight for valid approaches, and instructions that didn’t match their tests.⁶ On the repaired exam, Claude Code with Opus 4.6 went from 58.0% to 70.1%. “The largest gain,” in the maintainers’ words.

So both numbers in my original headline were wrong. One was inflated by answers in the margin; the other was deflated by broken questions. The 24-point gap was closer to zero.

Where Do the Two Agents Stand on Terminal-Bench Now?

The exam has been rewritten twice since. Terminal-Bench 3.0 shipped on July 30, 2026 with 74 new tasks, where “the best models achieve ~34%.” Version 4.0 followed on August 28: it dropped eight tasks, fixed nineteen, gave every task a flat eight-hour timeout, and turned Terminal-Bench into a continuous benchmark with semantic versions.⁷

Here’s the top of the 4.0 leaderboard as I write this on October 18, 2026:

ForgeCode isn’t on it. The 4.0 board lists five agents in total (Claude Code, Codex, Grok Build, Muse Code, and mini-SWE-agent), so absence doesn’t mean a bad score, only no score.

Read the confidence intervals before you read the ranks. Opus 5.5 and Sonnet 5.5 overlap. So do half the rows below them. And look at the cost column: one leaderboard run of Claude Code on Opus 5.5 burned 8.0 billion tokens and $4,718, while GPT-6.1 Sol landed within the margin of error of third place for$634. If you’re choosing a model for an agent fleet, that column matters more than the first one.

What Is ForgeCode Now?

Two different things, and the split is recent. The open-source CLI moved from antinomyhq/forge to tailcallhq/forgecode in April. It’s Rust, Apache-2.0, at about 7,600 GitHub stars, and still shipping: v2.14.0 landed October 1 with Opus 5.5 and Sonnet 5.5 support.⁸ Meanwhile forgecode.dev now sells a hosted “software factory” behind a “Book a demo” button. The blog, docs, and pricing pages that the first version of this post linked to are gone (the two benchmark posts survive on the Wayback Machine).

The CLI itself is solid. It speaks to 48 model providers, from Anthropic and OpenAI to Ollama and llama.cpp, and it can sign in with an existing Claude or ChatGPT subscription. It has MCP, subagents (forge builds, muse plans, sage researches read-only), SKILL.md skills, AGENTS.md instructions, and automatic context compaction at 100,000 tokens by default. Headless mode is one flag:

forge -p "fix the failing test in src/parser"    # no interactive UI; pipe a file in if the prompt is long

The default that surprised me is permissions. ForgeCode ships with restricted = false, and its built-in policy allows every read, write, command, and URL. There’s a real policy engine (allow, deny, or confirm per rule), but you have to turn it on:

1
restricted = true   # without this, the agent runs any command it decides to run

Also read the README’s note on indexing before you point it at a client’s code: semantic search “sends file content to the workspace server.”

ForgeCode vs Claude Code: Which Should You Pick Now?

Claude Code’s permission story changed the most since April. My first version called “permission fatigue” its biggest pain point, and Anthropic’s own docs still say it well: “After the tenth approval you’re clicking through rather than reviewing.”⁹ Auto mode answers it. Since v2.1.283 it’s the starting mode for interactive sessions, so rules resolve first, reads and in-project edits pass automatically, and a classifier reviews the rest (more on how that classifier decides). For hard guarantees, put hooks in front of it.

Two claims from the first version needed fixing on the Claude Code side too. Prompt caching is real and large: cache hits on Sonnet 5.5 cost $0.10 per million tokens against$2.00 for fresh input.¹⁰ But I repeated a blog’s claim that “all users share the same cached prefix,” and Anthropic’s docs say the opposite: “Different organizations never share caches, even if they use identical prompts.” The other was usage limits: Pro and Max plans still have a five-hour session limit and a weekly limit, shared with the Claude chat app, though you can now turn on usage credits and keep working past them.¹¹

My pick for most teams is Claude Code: the safer default, the CI integrations, and the extension surface, plus the top of the current leaderboard if you care about that. Pick ForgeCode if you need an open-source agent you can point at any model, including one running on your own hardware. Just turn on restricted on day one, and keep in mind that the company’s attention has moved to a different product.

How Should You Read an Agent Benchmark?

Check who ran it. Terminal-Bench marked verified runs on the 2.0 board, and the difference between “a team member ran this” and “a vendor uploaded this” turned out to be the whole story. Check which version of the exam it was, because 2.0, 2.1, and 4.0 scores aren’t comparable. Check the confidence interval, and treat overlapping rows as a tie.

Then run your own. The leaderboard harness is open source, and anyone can rerun an agent the way submitters did:

# Terminal-Bench 2.0 submissions: one agent, one model, 5 attempts per task
harbor run -d terminal-bench@2.0 -a "<agent>" -m "<model>" -k 5

Better still, take 20 tasks from your own backlog with tests that already exist, and run each agent on them. Read the traces before you read the pass rate. That’s how the UPenn team caught the answer keys, and it’s the only benchmark that grades your actual exam.

A leaderboard score is a claim. Ask who’s making it.