Agent harnesses: two public evaluations, a clear cost gap and a more discreet pass-rate gap

Agent harnesses: two public evaluations, a clear cost gap and a more discreet pass-rate gap

Between November 5 and 18, 2026, two public evaluations compare agent harnesses on the same model: Garry Tan's, on Claude Opus 5.5 and GPT-6.1 Sol, then the Harness Leaderboard, on four open models. Cost per solved task varies by up to a factor of 2.2. On open models the average pass-rate gap is 10 points. The ranking also measures that the harness adds only 2.8 points of disagreement to the noise between tries.

In brief

  • On 52 coding tasks, five harnesses out of six pass within 15 points of the model's own harness. Cost per solved task varies by up to a factor of 2.2.
  • On four open models, the Harness Leaderboard measures an average of 42.7% to 52.5% of tasks solved depending on the harness, with the model held constant. The gap reaches 22 points on a single model.
  • Re-running the same harness changes the result of 24.1% of tasks, switching harness changes it for 26.9%: the harness adds only 2.8 points of disagreement to the noise between tries.
  • The "18.4-point gap" that is circulating does appear on the Terminal-Bench 2.0 ranking, but it pits a harness found by automatic search against Claude Code. According to an October 12 analysis, the median gap between a model's best and worst harness is 5.4 points on verified rows only.
  • Garry Tan also publishes that the skills of gstack, one of his own tools, were not followed by Opus 5.5 (0 sessions out of 46).
harnaisagentsevaluationopen-source

This is a short excerpt.

The full analysis and podcast are for subscribers. The account is free.

Sign in

No account yet?

← All the watch