You've got a Claude Max plan, a ChatGPT Pro plan, and a Mac Studio with a local model. (Mac Studio specs: M3 Ultra with 256 GBs memory) You'd like to run a real benchmark across all three, but the API estimate for a proper study came back much too expensive. So I routed the whole thing through the subscriptions. It mostly worked. The part that didn't is the useful part.
The Setup
This is the follow-up to my GVS5H pilot. GVS5H is a research project claiming that a manager-and-worker scaffold lets smaller open models match frontier ones on hard coding problems. My adaptation runs three modes: a single call, an iterative loop that revises after public tests, and a "ledger" mode where a manager plans, fresh workers execute, and notes persist on disk between calls.
The study froze 24 tasks before any generation: twelve hard LiveCodeBench problems (six AtCoder, six LeetCode) and twelve practical repair fixtures in Python and TypeScript (cancellation, atomic writes, retries, pagination). Three systems, three modes, three repetitions. That's 648 episodes.
The three systems were a local Qwen3.8-27B running 4-bit on MLX, GPT-6 Astra through the Codex CLI on a ChatGPT Pro login, and Claude Fable 5.1 through Claude Code on a Max login. No API keys were supplied to the harness, extra spending was disabled on both accounts, and a dry subscription meant waiting for the reset, not paid credit.
One caveat I'll keep repeating: this compares three configured systems, not three sets of model weights. Claude Code controls its own temperature and context, the Codex CLI has no verified provider-side output cap, and my adapter serializes history differently than the raw APIs do. Treat every cross-system number accordingly.
What's Actually Going On
The study finished with 602 of 648 episodes completed and 555 hidden-test passes. Additional API charge: zero dollars. Recorded quota waits: zero seconds on both backends, across roughly four hundred cloud episodes.
Astra through Pro was flawless: all 216 planned episodes were completed, and all 216 passed, in every mode and both families. The local Qwen also completed all 216 of its episodes, passing 177.
The missing 46 episodes all belong to Fable through Max, and 36 of those 46 are ledger-mode episodes. Exactly half of Fable's planned ledger runs were never counted.
Here's why. The protocol had a hard rule: no model substitution. Any other model in a response marks the episode operationally incomplete and excluded. That's the right rule; you can't credit Fable for an answer Fable didn't write.
Anthropic documents that its Fable and Opus models include safety classifiers that can decline a request, and that a declined request can be retried on a fallback model. In 32 of the 46 missing episodes, the response stream contained a fallback block naming Claude Opus 5, which is precisely the handoff those docs describe. Every one of those 32 was a ledger episode. Ledger mode makes many calls per episode (planning, ideation, manager, workers, and finalization), so it gets far more chances to trip a classifier. The other 14 exclusions were a subtler failure: the CLI emitted two assistant events under one message ID, and my strict identity guard refused those too.
The original run stopped after three consecutive Fable failures, with 479 episodes done. A predeclared continuation covered only the 154 never-attempted Fable episodes, with one change: a documented fallback handoff excludes that episode without halting unrelated tasks. Nothing else changed, and no original episode was retried.
Note what I'm not claiming. The refusal categories weren't captured, so I don't know what a classifier objected to in a competitive programming problem, and I'm not saying an Opus answer would have failed. Those episodes can't be scored.
What the Numbers Say
Read these results with coverage attached for full context.
On the hard problems, Qwen went from 55.6% with a single call to 72.2% iterative and 77.8% with the ledger. That's the one place the scaffold appears to help. In the paired analysis, the ledger beat the single call on five of twelve tasks and lost on none, but after Holm correction the adjusted p-value is 0.375. Suggestive, not significant. It also cost a lot: about 11.3 hours of active local inference for the 36 ledger episodes versus about 3.5 hours for the 36 single-call ones.
Astra needed no help. Its ledger mode spent about 3.6 times the active seconds of its single mode on the hard problems to land on the same 36 of 36.
Fable on the hard problems passed 25 of 36 single (five excluded), 30 of 36 iterative (five excluded), and 18 of 36 ledger. Count only completed episodes, and the ledger went 18 for 18. Count all planned cells, and it's 50%. Both numbers are true. Only the second one is honest as a coverage-aware score.
On the practical repairs, everyone sat near the ceiling. Qwen passed 33, 36, and 34 of 36 across the three modes. Astra passed all 36 in every mode. Fable passed all 36 single and iterative, and 17 of its 18 completed ledger episodes.
One adherence note: ten Qwen ledger episodes returned a manager section in the wrong format, and nine never invoked a worker. A passing answer doesn't prove the manager-and-worker loop ran.
Total recorded output was about 6.3 million tokens. Claude Code's API-equivalent estimate for the Fable portion was about $58, a display number rather than an invoice or a quota measurement. When I kicked off the continuation, the max usage meter read 14%; I didn't meter that independently.
Why This Matters
If you're planning to benchmark models through subscriptions, the money question answers itself. It works, and the bill is your existing plan. The question you should actually plan for is harness behavior.
A subscription CLI is a product, not a raw endpoint. It may hand your request to a different model, split one message into several events, or run auxiliary calls you didn't ask for. Your protocol has to decide in advance what each of those means. Mine decided "exclude," which protected the numbers and cost me half of one arm. I'd make that trade again, but next time I'd instrument the refusal category.
The other lesson is older. Every gain I've seen from the ledger scaffold arrives with a multiple on time and tokens, and the one statistically tempting gain here didn't survive multiple-comparison correction. Measure the whole configuration on the work you care about before you commit machine days to orchestration.
Quick Reference
Freeze tasks, order, prompts, and budgets before generation.
Decide the model-identity rule up front. "Any other model in the stream excludes the episode" is defensible. Write it down.
Report success over all planned cells and over completed cells, side by side, with the missing count.
Log every exclusion reason in machine-readable form. Fallback, quota, and double-event failures are different problems.
Capture the refusal category if the provider exposes it. I didn't, and I can't answer the obvious follow-up.
Multi-call modes multiply exposure to classifier declines. Expect exclusions to concentrate there.
Found this useful? I share practical lessons from my systems and AI engineering journey at As The Geek Learns.
Terminology and Definitions
The tools and models in the study
Claude Code-Anthropic’s command-line coding agent; it reads a codebase, edits files, and runs commands from natural-language instructions. The article ran Claude Fable 5.1 through it. (https://code.claude.com/docs/en/overview)
Codex CLI-OpenAI’s equivalent command-line coding agent, open source and built in Rust. The article ran GPT-6 Astra through it. (https://github.com/openai/codex)
Qwen3.8-27B-a locally-run open-weight model; “27B” means roughly 27 billion parameters (the internal numeric knobs the model tunes during training. More parameters generally means more capability but more memory to run). (https://huggingface.co/Qwen/Qwen3.8-27B)
MLX-Apple’s array/machine-learning framework built specifically for Apple Silicon (M-series chips), which is what let the study run a 27-billion-parameter model locally on a Mac. (https://github.com/ml-explore/mlx)
4-bit (quantization)-compressing a model’s numbers from their original higher-precision format down to 4 bits each, shrinking memory use (often ~75% smaller than 16-bit) so a large model fits and runs faster on consumer hardware, at some small cost to accuracy. (https://huggingface.co/blog/4bit-transformers-bitsandbytes)
Frontier model(s)-informal industry shorthand for the most capable models currently available from major labs (what GVS5H’s smaller open models are trying to match).
How models behave mid-response
Temperature-a setting that controls how random or deterministic a model’s output is; low temperature sticks to the most likely wording, high temperature varies more. Relevant here because the article notes Claude Code, not the raw API, controls this setting itself. (https://platform.claude.com/docs/en/about-claude/glossary)
Context (context window)-the total amount of text (measured in tokens) a model can “see” at once, covering everything fed in plus everything it generates back. (https://platform.claude.com/docs/en/build-with-claude/context-windows)
Tokens-the chunks (roughly word-pieces) a model reads and writes text in; usage, context limits, and API costs are all measured in tokens, which is why the article reports “6.3 million recorded output tokens.” (https://platform.claude.com/docs/en/about-claude/glossary)
Safety classifiers / fallback model-Anthropic’s Fable and Opus models can decline (refuse) a request via an automated safety check, and the product can silently retry that same request on a different Claude model. The article’s 32 excluded episodes trace to this documented behavior. (https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback)
Message ID / assistant event(s)-the identifiers and discrete chunks a coding agent’s API stream is broken into; the article’s other 14 exclusions came from the CLI emitting two separate response events under a single message ID, which its strict identity check treated as a red flag.
The benchmark and its scaffold
GVS5H-the open-source research project this study adapts: a “manager-and-worker” pattern where fresh instances of one model plan, divide, and execute work by coordinating through files on disk, aiming to let smaller open models match frontier ones on hard coding tasks. (https://github.com/slee-persis/GVS5H)
Manager-and-worker scaffold / multi-agent orchestration-running several model calls in coordinated roles (one planning, others executing) instead of one model answering in a single shot. What the article’s “ledger mode” is an implementation of.
Ledger mode-this study’s specific three-mode label for the GVS5H-style scaffold: a manager plans, separate “worker” calls execute, and notes persist on disk between calls within one episode.
Episode-this study’s unit of measurement: one complete attempt at one task, in one mode, by one system. 648 episodes = 24 tasks × 3 systems × 3 modes × 3 repetitions.
LiveCodeBench-a public benchmark dataset of competitive-programming problems (pulled from sites like LeetCode and AtCoder) used to test whether a model’s code actually passes hidden tests, not just looks plausible. LiveCodeBench dataset on Hugging Face (https://huggingface.co/datasets/livecodebench/code_generation_lite)
AtCoder-a Japanese competitive-programming contest platform; a source of some of the study’s “hard” problems. (https://atcoder.jp/)
LeetCode-a widely used platform for practicing coding-interview and algorithm problems; the other source of the study’s “hard” problems. (https://leetcode.com/)
Hidden tests-the test cases a coding benchmark uses to actually grade a submission, kept separate from any “public” example tests the model can see. So a passing score means the code generalizes, not that it memorized the visible example.
The statistics
Paired analysis-comparing two conditions (e.g., ledger mode vs. single-call mode) on the same set of tasks, task by task, rather than comparing overall averages, is a more sensitive way to detect a real difference.
Holm correction (Holm–Bonferroni method)-a statistical adjustment applied when you run several significance tests at once, to keep the odds of a false “it worked!” finding from stacking up across all those tests. It’s why the article’s p-value changes from 0.0625 raw to 0.375 adjusted. Wikipedia: Holm–Bonferroni method (https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method)
p-value / statistical significance-a p-value estimates the odds of seeing a result this strong by chance alone if there were actually no real effect; “significant” conventionally means that odds is low enough (commonly below 0.05) to trust the effect is real rather than noise. The article’s adjusted p-value of 0.375 is well above that bar, hence “suggestive, not significant.” Wikipedia: p-value (https://en.wikipedia.org/wiki/P-value)