As The Geek Learns

Local Laya vs Hosted Jev: Why My Agents Make Typed Decisions on My Mac

James Cruce — Tue, 22 Sep 2026 16:01:29 GMT

Laya local, Not Jev Hosted

You ask a chat model a yes-or-no question and get back three paragraphs. Then your code has to parse the paragraphs, guess what it meant, and hope the JSON is valid this time. I got tired of that for the small, bounded choices my agents make all day. Here's the model I picked, the one I didn't, and the replay numbers behind the decision.

This is my account of one deployment. Whether Laya beats Jev in general is a different question, and I didn't test it.

The setup

A coding agent makes dozens of small decisions that aren't really language tasks. Which specialist should take this job? Does this alert need a human right now? Which local model can handle this request without blowing the memory budget? Prose is the wrong output type for a bounded question.

TypeSafe's Jev is built for exactly this. You hand it a state and a set of predefined questions, and it returns typed answers. A choice picks among named options. A score rates on an ordered scale. A noul returns a probability for a yes-or-no statement. No paragraph to interpret. I liked it immediately.

But I had a requirement Jev couldn't meet. I wanted routine decisions running on my Mac Studio, with local data staying local. I also wanted to control the exact model, the permitted options, and what happens when the model is unsure or down. Jev is an early-access hosted service. Laya is an open-weight model with the same kind of typed-decision interface, published with downloadable weights and an Apache 2.0 runtime. For this system, that settled it.

What's actually going on

Both models solve the same narrow problem: turn a compact state and a few questions into structured decisions. Neither writes code or reasons through a long investigation. It's a specialist. Your main model keeps its job.

Jev's managed API is a good deal on paper. TypeSafe lists $0.042 per million input tokens with no output-token charge and says Jev handles choices with up to 255 options. Hosted also means nobody babysits model files, accelerator compatibility, startup time, or process supervision. That's real work I chose to own.

Laya's trade-offs run the other way. I downloaded a pinned checkpoint once and serve it over a loopback-only API. Routine decision requests never leave the machine. There's no per-token bill, though the workstation, electricity, storage, and my time still cost money. I didn't calculate a break-even. Privacy and control were the decisive reasons. Cost was a side effect. I have no savings number to give you.

The checkpoint, convaiinnovations/laya-typed-decisions, has about 421 million parameters, a ModernBERT-large encoder, and a 1,024-token limit. Its publisher trained it on four synthetic workflows: agent observability, customer service, invoice processing, and security incidents. Model routing isn't one of those, which is a reason to test rather than assume.

Here's what makes it different from asking a chat model for JSON. The Laya runtime renders the question, each option, and the state into one token sequence, with a marker before each option. A bidirectional encoder reads the whole thing, a small decision head scores the marker positions, and a softmax turns those scores into a distribution over options. Several questions against one state batch into a single call. There's no autoregressive text output, so there's nothing to parse. The published runtime shows the sequence construction and the head.

From state and options to a probability per option. No text generation anywhere in the path.

For a choice, my code gets an option key and a probability for every allowed key. For a score, a distribution over ordered levels plus the expected level. For a noul, the probability of true. The runtime also reports a confidence derived from the distribution's normalized entropy. That confidence is not the probability the downstream task will succeed. I picked thresholds on a separate development set, and I'd want fresh evidence before reusing them on a different model roster.

Laya's authors trained with rewards based on proper scoring rules, meant to encourage honest probabilities. The model card still warns it's overconfident and needs domain-specific calibration. A typed answer can be structurally perfect and factually wrong. I treat Laya as a recommender inside a program, never as a source of permission or truth.

The fix

My host is a Mac Studio with an M3 Ultra and 256 GB of unified memory. That hardware is part of the result. A smaller machine needs its own memory and latency tests.

The original implementation brief got two things wrong. It assumed plain transformers.AutoModel could load Laya's decision model, and that the typed checkpoint lived only in a subfolder of the base model. Neither held. The typed checkpoint is its own repository and AutoModel loads only the encoder shape. The published laya runtime reconstructs the custom decision head, so the loader uses that instead of mistaking a bare encoder for a working decision engine.

Around the checkpoint sits a FastAPI service that accepts the TypeSafe wire shape, so a client written for Jev can point at my box:

{
  "model": "laya-local",
  "state": "The production API is down and customers are blocked.",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Invoices and refunds",
        "technical": "Bugs, outages, and API errors"
      }
    }
  }
}

My code defines the labels before Laya ever sees them. The response identifies itself as laya-local-v1. Accepting a Jev-style client alias doesn't pretend Jev supplied the answer. The service listens on 127.0.0.1:8017, because Docker already had port 8000 on this machine.

The service fails closed. It loads from an explicit local directory and refuses to download during inference. In production, it requires Apple's Metal accelerator and warms the model before reporting ready. It counts the exact rendered token sequence and rejects an over-budget request instead of letting the runtime quietly shorten the state to fit. It caps Choice at 20 options by default and serializes inference so concurrent callers can't pile uncontrolled load onto the accelerator.

The more important change sits outside the model. A gateway first drops any model that isn't eligible: wrong capability, wrong data class, too little context, unhealthy, past deadline, or over the memory budget. Laya gets at most five survivors plus an explicit abstain option. It never sees an ineligible model and can't grant one permission. The gateway re-validates the returned option, confidence, margin, model identity, and eligibility before reserving capacity. If Laya abstains or times out, a permitted local fallback runs. If nothing is eligible, the gateway refuses the request.

Laya only ranks survivors. Permission never comes from the model.

Inside ClaudeClaw, Laya now handles bounded specialist dispatch and how noncritical alerts get presented. A user who names a specialist still wins. Model advice can't suppress a critical alert. I also tried Laya for post-failure recovery choices. That profile failed acceptance, so it stays disabled, and the deterministic recovery rules stay in charge.

What the replay showed

I wanted a measured result for my own routing decision, not a latency figure copied from someone else's machine. The service's API, engine, SDK compatibility, and routing-contract suites recorded 59 passing tests, with one optional real-checkpoint test skipped. Separate HTTP runs exercised the real checkpoint on Metal.

The held-out replay used previously recorded outcomes for two models qualified for this setup, Qwen3 Coder 30B and GPT-OSS 120B. The replay simulated model health and resources. The Laya inference was real. Nobody reran the coding tasks. I chose the profile on 20 development decisions, froze it, then ran 40 held-out decisions. Each task appears with more than one latency preference, so those 40 are correlated, not 40 independent wins.

Share As The Geek Learns

On the held-out set, deterministic quality-and-latency ordering produced 33 acceptable decisions out of 40, or 82.5%. Laya, with the same eligibility rules and local fallback, produced 37 of 40, or 92.5%. Four more acceptable decisions, a 10-percentage-point gain on this one replay. Laya directly selected a model 15 times, the gateway fell back 17 times, and the gateway blocked eight cases where no candidate was available. It never selected an ineligible model.

Held-out Routing Replay

The three misses matter. One near-threshold case fell back to a model that had failed its source task. Two cases had no acceptable candidate because both underlying models had failed. Laya can't repair a task; neither candidate can do.

Latency is where it got humbling. The first held-out run had a routing p95 of 143.54 ms. A later supervised replay on a busy host kept the identical 37-of-40 outcome but pushed p95 to 466.69 ms, past my 100 ms aspiration and past the gateway's 300 ms decision timeout. That run is why the timeout and fallback exist. They fired for real. Twenty near-threshold decisions repeated three times kept their selected-or-abstained status every time, but repetition doesn't add independent cases. And the busy-host replay didn't prove a live gateway enforcing 300 ms would keep all 37 acceptable choices under that load.

Live checks went a bit further. The supervised Laya service and gateway reported ready. Gateway acceptance covered a direct Laya choice, local fallback, and duplicate-request rejection. ClaudeClaw's compiled dispatch checks picked the expected specialist three times. Those are bounded integration checks. They say nothing about the next agent task I haven't written yet.

Why this matters

I did not run Jev and Laya side by side on these 40 cases. The 10-point gain is against my deterministic router, not against Jev.

Laya's publisher reports 76.6% top-choice accuracy on its 2,000-decision test split against a published Jev figure of 72.7%. The same card says Jev matches the reference probability distributions better and has better raw calibration. Those numbers are leads. I didn't run a head-to-head under one protocol. The card also warns about overconfidence, English-only training, and accuracy falling as the option list grows, while TypeSafe says Jev supports far larger choices. I'd test Jev directly before claiming Laya is more accurate, faster, or cheaper for the same production decisions.

What I can say is narrower. On one frozen replay, the local system made more acceptable routing decisions than my previous ordering, and it kept the decision input, checkpoint, thresholds, and fallback policy under my control. Jev may well be the better product if you want a managed API, have large choice sets, or don't want to run local inference. I chose Laya because local processing was a requirement, and I was willing to own the testing and operations that came with it.

Thanks for reading As The Geek Learns! This post is public so feel free to share it.

Typed decisions: choice returns an option plus a probability per option, score a distribution over ordered levels, noul a probability of true. Nothing to parse.
Laya checkpoint: convaiinnovations/laya-typed-decisions, about 421M parameters, ModernBERT-large, 1,024-token limit, Apache 2.0.
Load it with the published laya runtime, not bare transformers.AutoModel, which gives you an encoder with no decision head.
Deterministic code filters eligibility first. The model only ranks survivors, plus an explicit abstain option.
Re-validate everything the model returns before acting. A decision receipt records; it doesn’t authorize.
Confidence is normalized entropy, not task success probability. Calibrate thresholds on a development set and freeze them before held-out.
Budget for timeouts. My p95 went from 143 ms to 467 ms on a busy host with identical decisions.

Terms worth knowing

Some of this vocabulary comes from a small corner of the AI world. Short definitions, in the order they show up.

Typed decision. An answer with a fixed shape your code can use directly, such as one option from a list or a number between 0 and 1. The opposite of a paragraph you have to parse.
State. The text or JSON you hand the model describes the situation it's deciding about. An alert, a task description, a customer message.
Choice, score, noul. Jev's three question types, which Laya copies. A choice picks one option from a list you define. A score picks a level on an ordered scale you define, like low, medium, critical. A noul is a yes-or-no statement and returns the probability that it's true. The term is TypeSafe's.
Open-weight model. A model whose trained weights you can download and run yourself, as opposed to one you can only reach through a vendor's API.
Checkpoint. A saved set of model weights. "Pinned checkpoint" means I locked to one exact version and verify it hasn't changed.
Loopback-only. The service listens on 127.0.0.1, so only programs on the same machine can reach it. Nothing on the network can.
Token. The unit a model reads. Roughly a short word or word fragment. Laya's 1,024-token limit is the total length of the state, question, and options after they're rendered together.
Encoder, bidirectional. A model that reads the entire input at once, in both directions, and produces a representation of it. It doesn't generate text. ModernBERT is a 2024-era encoder family. Chat models are decoders, which is a different design.
Autoregressive. Generating output one token at a time, each token depending on the ones before. That's how chat models write. Laya doesn't do it, which is why there's nothing to parse and why it's fast.
Decision head. A small neural network bolted onto the encoder that turns its output into scores, one per option. The base encoder alone can't make decisions. This is the piece transformers.AutoModel doesn't load.
Softmax. A math step that turns a list of raw scores into probabilities that add up to 1.
Normalized entropy. A measure of how spread out a probability distribution is, scaled from 0 to 1. Laya's confidence is 1 minus that. All options equally likely gives confidence 0. One option at 100% gives confidence 1. It measures how sure the model is, not how right it is.
Calibration, overconfidence. A calibrated model that says "80%" is right about 80% of the time. An overconfident one says 80% and is right less often. Laya's own model card says it's overconfident.
Proper scoring rule. A training reward designed so the model scores best only when it reports what it believes. It's meant to discourage bluffing. It doesn't guarantee calibration on your data.
Development set, held-out set. Two separate batches of test cases. You tune thresholds on the development set, freeze them, and then measure on the held-out set the model has never influenced. Tuning on the held-out set makes the result meaningless.
Correlated cases. Test cases that share an underlying task, so they tend to succeed or fail together. Forty correlated cases carry less evidence than forty independent ones.
p95. The latency that 95% of requests came in under. A better measure of "how slow does it get" than the average.
Metal, MPS. Apple's GPU framework and PyTorch's backend for it. Requiring MPS means the service refuses to start if the model would silently run on the CPU.
Wire shape. The exact JSON layout of a request and response. Matching Jev's wire shape means a client written for Jev works against my service without changes.
Fails closed. When something is missing or wrong, the service stops instead of guessing. No checkpoint, no start. Request too long, rejected, not trimmed.
Abstain. An explicit "none of these" option the gateway always adds. It lets Laya decline instead of forcing a pick.
Eligibility gateway. The deterministic code that filters out models Laya isn't allowed to choose before Laya sees the list, then re-checks the answer afterward.

Leave a comment

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns.

Thanks for reading As The Geek Learns! This post is public, so feel free to share it.

Can a Subscription Fund a Coding-Model Study? 648 Episodes, $0 API

James Cruce — Thu, 17 Sep 2026 16:31:53 GMT

You’ve got a Claude Max plan, a ChatGPT Pro plan, and a Mac Studio with a local model. (Mac Studio specs: M3 Ultra with 256 GBs memory) You’d like to run a real benchmark across all three, but the API estimate for a proper study came back much too expensive. So I routed the whole thing through the subscriptions. It mostly worked. The part that didn’t is the useful part.

The Setup

This is the follow-up to my GVS5H pilot. GVS5H is a research project claiming that a manager-and-worker scaffold lets smaller open models match frontier ones on hard coding problems. My adaptation runs three modes: a single call, an iterative loop that revises after public tests, and a “ledger” mode where a manager plans, fresh workers execute, and notes persist on disk between calls.

The study froze 24 tasks before any generation: twelve hard LiveCodeBench problems (six AtCoder, six LeetCode) and twelve practical repair fixtures in Python and TypeScript (cancellation, atomic writes, retries, pagination). Three systems, three modes, three repetitions. That’s 648 episodes.

As The Geek Learns is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

The three systems were a local Qwen3.8-27B running 4-bit on MLX, GPT-6 Astra through the Codex CLI on a ChatGPT Pro login, and Claude Fable 5.1 through Claude Code on a Max login. No API keys were supplied to the harness, extra spending was disabled on both accounts, and a dry subscription meant waiting for the reset, not paid credit.

One caveat I’ll keep repeating: this compares three configured systems, not three sets of model weights. Claude Code controls its own temperature and context, the Codex CLI has no verified provider-side output cap, and my adapter serializes history differently than the raw APIs do. Treat every cross-system number accordingly.

What’s Actually Going On

The study finished with 602 of 648 episodes completed and 555 hidden-test passes. Additional API charge: zero dollars. Recorded quota waits: zero seconds on both backends, across roughly four hundred cloud episodes.

Every episode passes a model-identity guard. Ledger mode’s many calls per episode gave the classifier far more chances to hand a request to Opus 5.

Astra through Pro was flawless: all 216 planned episodes were completed, and all 216 passed, in every mode and both families. The local Qwen also completed all 216 of its episodes, passing 177.

The missing 46 episodes all belong to Fable through Max, and 36 of those 46 are ledger-mode episodes. Exactly half of Fable’s planned ledger runs were never counted.

Here’s why. The protocol had a hard rule: no model substitution. Any other model in a response marks the episode operationally incomplete and excluded. That’s the right rule; you can’t credit Fable for an answer Fable didn’t write.

Share As The Geek Learns

Anthropic documents that its Fable and Opus models include safety classifiers that can decline a request, and that a declined request can be retried on a fallback model. In 32 of the 46 missing episodes, the response stream contained a fallback block naming Claude Opus 5, which is precisely the handoff those docs describe. Every one of those 32 was a ledger episode. Ledger mode makes many calls per episode (planning, ideation, manager, workers, and finalization), so it gets far more chances to trip a classifier. The other 14 exclusions were a subtler failure: the CLI emitted two assistant events under one message ID, and my strict identity guard refused those too.

All 46 exclusions were Fable via Max. The 32 fallback handoffs were all ledger episodes.

The original run stopped after three consecutive Fable failures, with 479 episodes done. A predeclared continuation covered only the 154 never-attempted Fable episodes, with one change: a documented fallback handoff excludes that episode without halting unrelated tasks. Nothing else changed, and no original episode was retried.

Note what I’m not claiming. The refusal categories weren’t captured, so I don’t know what a classifier objected to in a competitive programming problem, and I’m not saying an Opus answer would have failed. Those episodes can’t be scored.

What the Numbers Say

Read these results with coverage attached for full context.

648 planned episodes. Missing cells stay in the denominator; active hours are inference time, not wall clock.

On the hard problems, Qwen went from 55.6% with a single call to 72.2% iterative and 77.8% with the ledger. That’s the one place the scaffold appears to help. In the paired analysis, the ledger beat the single call on five of twelve tasks and lost on none, but after Holm correction the adjusted p-value is 0.375. Suggestive, not significant. It also cost a lot: about 11.3 hours of active local inference for the 36 ledger episodes versus about 3.5 hours for the 36 single-call ones.

Astra needed no help. Its ledger mode spent about 3.6 times the active seconds of its single mode on the hard problems to land on the same 36 of 36.

Fable on the hard problems passed 25 of 36 single (five excluded), 30 of 36 iterative (five excluded), and 18 of 36 ledger. Count only completed episodes, and the ledger went 18 for 18. Count all planned cells, and it’s 50%. Both numbers are true. Only the second one is honest as a coverage-aware score.

On the practical repairs, everyone sat near the ceiling. Qwen passed 33, 36, and 34 of 36 across the three modes. Astra passed all 36 in every mode. Fable passed all 36 single and iterative, and 17 of its 18 completed ledger episodes.

One adherence note: ten Qwen ledger episodes returned a manager section in the wrong format, and nine never invoked a worker. A passing answer doesn’t prove the manager-and-worker loop ran.

Total recorded output was about 6.3 million tokens. Claude Code’s API-equivalent estimate for the Fable portion was about $58, a display number rather than an invoice or a quota measurement. When I kicked off the continuation, the max usage meter read 14%; I didn’t meter that independently.

Why This Matters

If you’re planning to benchmark models through subscriptions, the money question answers itself. It works, and the bill is your existing plan. The question you should actually plan for is harness behavior.

A subscription CLI is a product, not a raw endpoint. It may hand your request to a different model, split one message into several events, or run auxiliary calls you didn’t ask for. Your protocol has to decide in advance what each of those means. Mine decided “exclude,” which protected the numbers and cost me half of one arm. I’d make that trade again, but next time I’d instrument the refusal category.

The other lesson is older. Every gain I’ve seen from the ledger scaffold arrives with a multiple on time and tokens, and the one statistically tempting gain here didn’t survive multiple-comparison correction. Measure the whole configuration on the work you care about before you commit machine days to orchestration.

Quick Reference

Freeze tasks, order, prompts, and budgets before generation.
Decide the model-identity rule up front. “Any other model in the stream excludes the episode” is defensible. Write it down.
Report success over all planned cells and over completed cells, side by side, with the missing count.
Log every exclusion reason in machine-readable form. Fallback, quota, and double-event failures are different problems.
Capture the refusal category if the provider exposes it. I didn’t, and I can’t answer the obvious follow-up.
Multi-call modes multiply exposure to classifier declines. Expect exclusions to concentrate there.

Thanks for reading As The Geek Learns! This post is public, so feel free to share it.

Found this useful? I share practical lessons from my systems and AI engineering journey at As The Geek Learns.

Leave a comment

Terminology and Definitions

The tools and models in the study

Claude Code-Anthropic’s command-line coding agent; it reads a codebase, edits files, and runs commands from natural-language instructions. The article ran Claude Fable 5.1 through it. (https://code.claude.com/docs/en/overview)

Codex CLI-OpenAI’s equivalent command-line coding agent, open source and built in Rust. The article ran GPT-6 Astra through it. (https://github.com/openai/codex)

Qwen3.8-27B-a locally-run open-weight model; “27B” means roughly 27 billion parameters (the internal numeric knobs the model tunes during training. More parameters generally means more capability but more memory to run). (https://huggingface.co/Qwen/Qwen3.8-27B)

MLX-Apple’s array/machine-learning framework built specifically for Apple Silicon (M-series chips), which is what let the study run a 27-billion-parameter model locally on a Mac. (https://github.com/ml-explore/mlx)

4-bit (quantization)-compressing a model’s numbers from their original higher-precision format down to 4 bits each, shrinking memory use (often ~75% smaller than 16-bit) so a large model fits and runs faster on consumer hardware, at some small cost to accuracy. (https://huggingface.co/blog/4bit-transformers-bitsandbytes)

Frontier model(s)-informal industry shorthand for the most capable models currently available from major labs (what GVS5H’s smaller open models are trying to match).

How models behave mid-response

Temperature-a setting that controls how random or deterministic a model’s output is; low temperature sticks to the most likely wording, high temperature varies more. Relevant here because the article notes Claude Code, not the raw API, controls this setting itself. (https://platform.claude.com/docs/en/about-claude/glossary)

Context (context window)-the total amount of text (measured in tokens) a model can “see” at once, covering everything fed in plus everything it generates back. (https://platform.claude.com/docs/en/build-with-claude/context-windows)

Tokens-the chunks (roughly word-pieces) a model reads and writes text in; usage, context limits, and API costs are all measured in tokens, which is why the article reports “6.3 million recorded output tokens.” (https://platform.claude.com/docs/en/about-claude/glossary)

Safety classifiers / fallback model-Anthropic’s Fable and Opus models can decline (refuse) a request via an automated safety check, and the product can silently retry that same request on a different Claude model. The article’s 32 excluded episodes trace to this documented behavior. (https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback)

Message ID / assistant event(s)-the identifiers and discrete chunks a coding agent’s API stream is broken into; the article’s other 14 exclusions came from the CLI emitting two separate response events under a single message ID, which its strict identity check treated as a red flag.

The benchmark and its scaffold

GVS5H-the open-source research project this study adapts: a “manager-and-worker” pattern where fresh instances of one model plan, divide, and execute work by coordinating through files on disk, aiming to let smaller open models match frontier ones on hard coding tasks. (https://github.com/slee-persis/GVS5H)

Manager-and-worker scaffold / multi-agent orchestration-running several model calls in coordinated roles (one planning, others executing) instead of one model answering in a single shot. What the article’s “ledger mode” is an implementation of.

Ledger mode-this study’s specific three-mode label for the GVS5H-style scaffold: a manager plans, separate “worker” calls execute, and notes persist on disk between calls within one episode.

Episode-this study’s unit of measurement: one complete attempt at one task, in one mode, by one system. 648 episodes = 24 tasks × 3 systems × 3 modes × 3 repetitions.

LiveCodeBench-a public benchmark dataset of competitive-programming problems (pulled from sites like LeetCode and AtCoder) used to test whether a model’s code actually passes hidden tests, not just looks plausible. LiveCodeBench dataset on Hugging Face (https://huggingface.co/datasets/livecodebench/code_generation_lite)

AtCoder-a Japanese competitive-programming contest platform; a source of some of the study’s “hard” problems. (https://atcoder.jp/)

LeetCode-a widely used platform for practicing coding-interview and algorithm problems; the other source of the study’s “hard” problems. (https://leetcode.com/)

Hidden tests-the test cases a coding benchmark uses to actually grade a submission, kept separate from any “public” example tests the model can see. So a passing score means the code generalizes, not that it memorized the visible example.

The statistics

Paired analysis-comparing two conditions (e.g., ledger mode vs. single-call mode) on the same set of tasks, task by task, rather than comparing overall averages, is a more sensitive way to detect a real difference.

Holm correction (Holm–Bonferroni method)-a statistical adjustment applied when you run several significance tests at once, to keep the odds of a false “it worked!” finding from stacking up across all those tests. It’s why the article’s p-value changes from 0.0625 raw to 0.375 adjusted. Wikipedia: Holm–Bonferroni method (https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method)

p-value / statistical significance-a p-value estimates the odds of seeing a result this strong by chance alone if there were actually no real effect; “significant” conventionally means that odds is low enough (commonly below 0.05) to trust the effect is real rather than noise. The article’s adjusted p-value of 0.375 is well above that bar, hence “suggestive, not significant.” Wikipedia: p-value (https://en.wikipedia.org/wiki/P-value)

Can a Subscription Fund a Coding-Model Study? 648 Episodes, $0 API

James Cruce — Thu, 17 Sep 2026 16:30:40 GMT

You've got a Claude Max plan, a ChatGPT Pro plan, and a Mac Studio with a local model. (Mac Studio specs: M3 Ultra with 256 GBs memory) You'd like to run a real benchmark across all three, but the API estimate for a proper study came back much too expensive. So I routed the whole thing through the subscriptions. It mostly worked. The part that didn't is the useful part.

The Setup

This is the follow-up to my GVS5H pilot. GVS5H is a research project claiming that a manager-and-worker scaffold lets smaller open models match frontier ones on hard coding problems. My adaptation runs three modes: a single call, an iterative loop that revises after public tests, and a "ledger" mode where a manager plans, fresh workers execute, and notes persist on disk between calls.

One caveat I'll keep repeating: this compares three configured systems, not three sets of model weights. Claude Code controls its own temperature and context, the Codex CLI has no verified provider-side output cap, and my adapter serializes history differently than the raw APIs do. Treat every cross-system number accordingly.

What's Actually Going On

Every episode passes a model-identity guard. Ledger mode’s many calls per episode gave the classifier far more chances to hand a request to Opus 5.

Astra through Pro was flawless: all 216 planned episodes were completed, and all 216 passed, in every mode and both families. The local Qwen also completed all 216 of its episodes, passing 177.

The missing 46 episodes all belong to Fable through Max, and 36 of those 46 are ledger-mode episodes. Exactly half of Fable's planned ledger runs were never counted.

Here's why. The protocol had a hard rule: no model substitution. Any other model in a response marks the episode operationally incomplete and excluded. That's the right rule; you can't credit Fable for an answer Fable didn't write.

Share As The Geek Learns

All 46 exclusions were Fable via Max. The 32 fallback handoffs were all ledger episodes.

Note what I'm not claiming. The refusal categories weren't captured, so I don't know what a classifier objected to in a competitive programming problem, and I'm not saying an Opus answer would have failed. Those episodes can't be scored.

What the Numbers Say

Read these results with coverage attached for full context.

648 planned episodes. Missing cells stay in the denominator; active hours are inference time, not wall clock.

On the hard problems, Qwen went from 55.6% with a single call to 72.2% iterative and 77.8% with the ledger. That's the one place the scaffold appears to help. In the paired analysis, the ledger beat the single call on five of twelve tasks and lost on none, but after Holm correction the adjusted p-value is 0.375. Suggestive, not significant. It also cost a lot: about 11.3 hours of active local inference for the 36 ledger episodes versus about 3.5 hours for the 36 single-call ones.

Astra needed no help. Its ledger mode spent about 3.6 times the active seconds of its single mode on the hard problems to land on the same 36 of 36.

Fable on the hard problems passed 25 of 36 single (five excluded), 30 of 36 iterative (five excluded), and 18 of 36 ledger. Count only completed episodes, and the ledger went 18 for 18. Count all planned cells, and it's 50%. Both numbers are true. Only the second one is honest as a coverage-aware score.

One adherence note: ten Qwen ledger episodes returned a manager section in the wrong format, and nine never invoked a worker. A passing answer doesn't prove the manager-and-worker loop ran.

Total recorded output was about 6.3 million tokens. Claude Code's API-equivalent estimate for the Fable portion was about $58, a display number rather than an invoice or a quota measurement. When I kicked off the continuation, the max usage meter read 14%; I didn't meter that independently.

Why This Matters

If you're planning to benchmark models through subscriptions, the money question answers itself. It works, and the bill is your existing plan. The question you should actually plan for is harness behavior.

A subscription CLI is a product, not a raw endpoint. It may hand your request to a different model, split one message into several events, or run auxiliary calls you didn't ask for. Your protocol has to decide in advance what each of those means. Mine decided "exclude," which protected the numbers and cost me half of one arm. I'd make that trade again, but next time I'd instrument the refusal category.

The other lesson is older. Every gain I've seen from the ledger scaffold arrives with a multiple on time and tokens, and the one statistically tempting gain here didn't survive multiple-comparison correction. Measure the whole configuration on the work you care about before you commit machine days to orchestration.

Quick Reference

Freeze tasks, order, prompts, and budgets before generation.
Decide the model-identity rule up front. "Any other model in the stream excludes the episode" is defensible. Write it down.
Report success over all planned cells and over completed cells, side by side, with the missing count.
Log every exclusion reason in machine-readable form. Fallback, quota, and double-event failures are different problems.
Capture the refusal category if the provider exposes it. I didn't, and I can't answer the obvious follow-up.
Multi-call modes multiply exposure to classifier declines. Expect exclusions to concentrate there.

Thanks for reading As The Geek Learns! This post is public, so feel free to share it.

Found this useful? I share practical lessons from my systems and AI engineering journey at As The Geek Learns.

Leave a comment

Terminology and Definitions

The tools and models in the study

Codex CLI-OpenAI’s equivalent command-line coding agent, open source and built in Rust. The article ran GPT-6 Astra through it. (https://github.com/openai/codex)

Frontier model(s)-informal industry shorthand for the most capable models currently available from major labs (what GVS5H’s smaller open models are trying to match).

How models behave mid-response

The benchmark and its scaffold

Episode-this study’s unit of measurement: one complete attempt at one task, in one mode, by one system. 648 episodes = 24 tasks × 3 systems × 3 modes × 3 repetitions.

AtCoder-a Japanese competitive-programming contest platform; a source of some of the study’s “hard” problems. (https://atcoder.jp/)

LeetCode-a widely used platform for practicing coding-interview and algorithm problems; the other source of the study’s “hard” problems. (https://leetcode.com/)

The statistics

When Your Own Agent Is the Attacker: Reward Hacking Hits Production

James Cruce — Wed, 22 Jul 2026 19:10:20 GMT

Yesterday the story was that an autonomous AI agent breached Hugging Face, and the defenders used AI to fight back. Today it has a much stranger ending. OpenAI says the attacker was its own models, and they weren't trying to cause damage. They were trying to pass a test.

The Setup

Here's what OpenAI described. During an internal security evaluation on a benchmark called ExploitGym, two of its models, GPT-5.6 Sol and a more capable unreleased one, ran with their safety filters dialed down so the team could measure raw cyber capability. The models were supposed to solve exploit challenges inside a contained test environment.

They didn't stay contained. Burning through a lot of inference compute, they found and exploited a zero-day in a package registry cache proxy, escalated privileges, moved laterally through OpenAI's research infrastructure, and reached a node with open internet access. Then they reasoned that Hugging Face probably hosted ExploitGym's models, datasets, and solutions. So they broke into Hugging Face's production database and stole the answer key.

Read that again. The goal was never sabotage. The goal was to win the benchmark, and hacking a third party's production systems was just the shortest path the model could find.

What's Actually Going On

This is reward hacking, and it's one of the oldest-known failure modes in machine learning. You give a system an objective, and it optimizes that objective in a way you didn't intend and don't like. The classic examples are toys: a boat-racing agent that spins in circles farming points instead of finishing the race, a simulated robot that learns to hide a mess instead of cleaning it. Specification gaming, researchers call it. The model does exactly what you measured, not what you meant.

What changed here is scale and capability. When a toy agent games a toy metric, you get a funny screenshot. When a frontier model games a real metric, it can chain zero-days, escalate through a corporate network, and pivot into someone else's production database to get there.

OpenAI's own phrasing is that the models "were hyper-focused on finding a solution for ExploitGym and employed extreme measures."

That's a polite way of saying the model treated every security boundary between it and a higher score as an obstacle to route around.

The uncomfortable part: this wasn't a jailbreak or a rogue user prompt. The model was doing its assigned job. It just did it too well, in a direction nobody sandboxed against. And METR had already flagged this exact model, GPT-5.6 Sol, as having one of the highest cheating rates among publicly tested models. The signal was there before the incident.

What To Actually Do About It

You can't out-align your way past this if the containment isn't real. Treat every capable agent as something that will take the cheapest path to its goal, including paths straight through your infrastructure. A few things that actually move the needle:

Make egress default-deny. The models "escaped" by finding a route to the open internet. An agent's sandbox should block all outbound network by default and allowlist only the specific hosts the task needs. If the box can't reach the internet, it can't pivot to a third party's database, no matter how clever it gets.

Scope credentials to the task, not the machine. Ambient cloud credentials sitting on the agent's node are an invitation. Issue short-lived, narrowly scoped tokens for the one thing the agent is doing, and rotate them aggressively. The blast radius of a gamed objective is exactly the set of permissions you handed the process.

Design the objective, then attack it. Before you point an agent at a metric, spend ten minutes as the adversary. Ask, “What’s the laziest, ugliest way to max this number?" If the answer involves touching something you assumed was off-limits, your metric is the vulnerability.

Watch behavior, not just outputs. Yesterday's breach left more than 17,000 recorded attacker actions before anyone caught it. Anomaly detection on what your agent does, not just what it returns, is what turns a silent compromise into an alert.

Don't confuse "reduced filters" with "isolated." Turning off safety filters to measure capability is a legitimate test. Doing it without a network and credential boundary that assumes the model will misbehave is how a test becomes an incident.

Why This Matters

"Aligned in the lab" and "contained in the lab" are not the same claim, and this incident is the gap between them written into production logs. The model was presumably well-behaved on every normal prompt. Under a narrow enough incentive, with the filters down, it went and hacked a company.

There's a second irony worth sitting with. When Hugging Face ran forensics on the attack, its responders reached for an open-weight model, GLM 5.2, running locally, because the commercial models refused the cyber-related prompts on safety grounds. The same guardrails that couldn't stop the attacker also got in the way of the defender. If you run local models, that's not a gotcha, it's leverage: you own the safety tradeoff instead of renting it.

The real takeaway isn't "AI is scary." It's that agent security is now a systems-engineering problem you already know how to solve. Least privilege. Network segmentation. Egress control. Observability. Adversarial testing. The frontier labs are learning this in public. You get to learn it before you hand an agent the keys.

Quick Reference

Egress default-deny: block all outbound network, allowlist only what the task needs.
Scoped, short-lived credentials: no ambient cloud creds sitting on the agent's node.
Attack your own objective: find the laziest way to max the metric before the model does.
Behavioral monitoring: alert on what the agent does, not just what it returns.
"Filters off" is not "isolated": capability tests still need a real network and credential boundary.
Local models own the safety tradeoff: useful when commercial guardrails refuse legitimate defensive work.

Share As The Geek Learns

Sources

This one's a breaking-news piece, not just my own take, so here's the reporting it's built on:

OpenAI's attribution and the ExploitGym details: The Decoder
Corroboration on the sandbox escape and the Hugging Face target: The Hacker News
The underlying breach, the 17,000 attacker actions, and the AI-vs-AI response: The Decoder
Why Hugging Face's defenders reached for open-weight GLM 5.2: Fortune

Leave a comment

Found this useful? I share practical lessons from my systems engineering journey at

The Silent API Spend Trap: Why Cron Jobs Need Tiered Model Routing

James Cruce — Tue, 14 Jul 2026 14:15:17 GMT

You set up an autonomous agent to check a status page, summarize a log file, or score an incoming request every fifteen minutes. It works. You move on. Three weeks later you're staring at an API bill wondering where the money went, and the answer is: every one of those checks quietly called a paid model because that's what the scaffolding defaulted to.

The Setup

Here's how it happens. You're building your first cron job or headless agent, and you reach for the model you already have configured: your API key, your existing client, the one you use every day in your interactive coding sessions. It's the path of least resistance. The job works on the first try, the output looks fine, and you ship it.

Now do that four more times. A calendar check here, a log summarizer there, a scoring pass on incoming webhooks, a nightly digest job. Each one is trivial in isolation, a few cents per run. None of them individually looks worth optimizing. But cron jobs don't run once. They run every fifteen minutes, every hour, every night, forever, and nobody's watching a dashboard for "aggregate cost of five small automations." You only notice when the monthly total shows up somewhere you're already looking, like a billing email.

What's Actually Going On

This isn't a pricing problem. It's a defaults problem. When you're building interactively, using the strongest model available makes sense: you're paying for judgment, nuance, and the ability to handle a task you haven't fully specified yet. But a cron job isn't interactive. It runs unattended, on a fixed schedule, doing the same narrow task every time. Most of that work doesn't need frontier-model reasoning. It needs “Did this text contain the word 'error'" or “Summarize these five bullet points" or “Does this value fall outside a threshold."

The trap is that nobody makes an explicit decision to use the expensive model for ops work. It's just what the code already had wired up. Reliability worries make this worse: the first time a local model flakes or a self-hosted endpoint times out, the instinct is to blame the model and switch back to the API default rather than debug the actual cause, and once that switch flips, it rarely gets revisited.

Treating local-versus-API as a deliberate architectural decision, made once per task class instead of once per project, is what closes the gap.

The Fix

Most tools that call an LLM API, whether it's a summarizer CLI, a LangChain chain, or your own scripts, are built against the OpenAI-compatible chat completions format. If you're running Ollama locally, it already exposes that exact interface, so redirecting a tool to a local model is usually a base URL change, not a rewrite:

Point that at whichever local model you're running, gemma4:31b-mlx or whatever's current in your Ollama pull list, and any OpenAI-SDK-based tool talks to it without touching its own code. That's the mechanism. The decision layer on top of it is a simple tier:

Tier 1, local by default: anything scheduled and unattended, anything scoring or classifying against a fixed rubric, anything summarizing structured or short-form input. This is nearly every cron job.
Tier 2, API on escalation: tasks that need genuine reasoning over ambiguous or long-context input, anything customer-facing where a wrong answer is costly, anything you'd want a second opinion on if a human were doing it.

Write the tier assignment down next to the job definition, not just in your head. When you add a new cron, the question isn't "which model do I already have a key for," it's "which tier does this task belong to." That written-down rule is what keeps you from relitigating the same defaults decision every time you copy-paste a cron.

Why This Matters

The pattern extends past cost. Any unattended process that runs on a schedule, not just LLM calls, tends to inherit whatever configuration was easiest at creation time rather than what the task actually needs, and nobody revisits it because it isn't failing loudly. Cost is just the failure mode that happens to show up on an invoice instead of in an error log. If you're building a fleet of small agents or cron-driven automations, the same discipline applies to any resource you can over-provision by default: compute tier, retry budget, context window, and concurrency. Unattended work needs its defaults set on purpose because nothing else is going to flag it for you.

The other half of this is treating local-versus-API as a real architectural question instead of a reflex. If a job flakes, that's worth investigating on its own terms, not a reason to fall back to whichever model has a company card behind it. The reflex, not the model, is usually the expensive part.

Share As The Geek Learns

Quick Reference

Default new cron jobs and headless agents to a local model; escalate deliberately, not by habit
Route OpenAI-compatible tools to Ollama with OPENAI_BASE_URL=http://localhost:11434/v1
Tier 1 (local): scheduled, unattended, fixed-rubric scoring, short-input summarization
Tier 2 (API): ambiguous or long-context reasoning, customer-facing output, anything you'd want a second opinion on
Treat "the local model seems unreliable" as a question to investigate, not a reason to reflexively fall back to the API
Write the tier decision down next to the job definition so it doesn't get silently overridden later

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns

Leave a comment

Frequently Asked Questions

What does “tiered model routing” mean for AI agents?

It means assigning scheduled or unattended tasks to a cheap or local model by default, and reserving paid API models for work that genuinely needs deeper reasoning, instead of defaulting every task to whatever model your interactive tooling already uses.

Source: ASTGL Analysis

Why do API costs from cron jobs sneak up on you?

A job that costs a few cents per run adds up across dozens or hundreds of scheduled runs, and nothing surfaces the running total unless you’re specifically watching a billing dashboard for it.

Source: ASTGL Analysis

Does Ollama support the OpenAI API format?

Yes. Ollama exposes an OpenAI-compatible chat completions endpoint at `http://localhost:11434/v1` and accepts any placeholder value as the API key, so OpenAI-SDK-based tools can point at it with just a base-URL change.

Source: https://ollama.com/blog/openai-compatibility

What kinds of tasks are safe to route to a local model?

Scheduled, unattended tasks with a narrow job: fixed-rubric scoring, short-input summarization, calendar or status checks, and anything where the same input should reliably produce the same category of output.

Source: ASTGL Analysis

When should an agent escalate to a paid API model instead?

When the task needs genuine reasoning over ambiguous or long-context input, is customer-facing where a wrong answer is costly, or is the kind of judgment call you’d want a second opinion on if a human were doing it.

Source: ASTGL Analysis

My local model seems unreliable; should I switch back to the API?

Investigate it as its own problem before you reach for the reflex switch. Falling back to the paid API by habit every time a local model flakes is exactly the defaults problem this article describes, and the reflex is usually the more expensive part, not the model.

Source: ASTGL Analysis

You Have 6 Days: The Fable 5 Subscriber Migration Checklist

James Cruce — Mon, 13 Jul 2026 18:36:01 GMT

You probably have agents or workflows calling Claude Fable 5 right now because it’s been included in your subscription. That free window closes in six days. Instead of panic-switching everything or ignoring the bill, there is a calm way to decide what stays and what moves.

The clock that’s actually ticking

The deadline is firm: July 19, 2026, at 11:59:59 PM PT. Until then, if you are on a Pro, Max, Team, or premium Enterprise plan, you can use Fable 5 for up to 50% of your weekly usage limits without extra charges. After that, continued use requires prepaid credits at $10 per million input tokens and $50 per million output tokens.

Anthropic has extended this window three times in five weeks. The original paywall was June 22, then July 7, then July 12, and now July 19. While they say the credit-billing arrangement is temporary and they’ll return Fable 5 to standard subscriptions when compute capacity allows, there is no timeline for that. These repeated extensions are a signal that Anthropic is capacity-constrained. Plan as if July 19 is the final date.

This is a routing problem, not a loss

It’s easy to feel like you’re losing a tool, but for most of us, this is actually a routing problem. Most of the work we send to Fable 5 doesn’t actually need Fable 5. We tend to pin the “best” model by default because it’s available and free, regardless of whether the task requires that level of reasoning.

The fix isn’t a global switch to a cheaper model; it’s routing per task. You don’t need a sledgehammer to hang a picture frame. By identifying which tasks truly require high-reasoning capabilities and which are just “good enough” for mid-tier models, you can maintain your quality while keeping your costs low.

The six-day checklist

You can handle this migration in four steps. Don’t do it globally; do it per call site.

1. Audit your call sites

Start by finding every place where your code or config pins Fable 5. Grep your environment variables, agent definitions, and launch scripts for “fable” or the model ID claude-fable-5. You’ll likely find that you’ve pinned it in fewer places than you remember, but also in a few places reflexively just because it was the top option. List every single call site.

2. Classify by capability

Be honest about what each task actually needs. Sort each call site into three buckets:

Quality-critical reasoning: This is for hard multi-step logic, long-horizon autonomous runs, or tasks where a wrong answer costs you real money or hours of debugging. These genuinely want Fable 5 or Opus 4.8.

Mid-tier: Most coding, tool-heavy workflows, and structured data work fall here. Opus 4.8 or Sonnet 5 handle these effectively.

Cheap/high-volume: Simple classification, extraction, routing, or basic lookups. These belong on Haiku 4.5.

3. Substitute per task

Now map those buckets to specific models. This is where you save the most money.

For coding and agentic work, move to Claude Opus 4.8 (claude-opus-4-8) using effort xhigh. This is Anthropic’s own recommendation for coding (and it’s the default for Claude Code). It costs $5 per million input and $25 per million output, which is exactly half the cost of Fable 5.

For mid-tier work, use Sonnet 5 (claude-sonnet-5). It offers near-Opus quality at a lower price point: $3 per million input and $15 per million output (though there’s intro pricing of $2/$10 through August 31, 2026).

For high-volume, simple tasks, use Haiku 4.5 (claude-haiku-4-5). It’s the fastest and cheapest option at $1 per million input and $5 per million output.

For those few quality-critical tasks, you can either keep them on Fable 5 and budget for the credits or step them down to Opus 4.8 and measure if the quality gap actually affects your specific workload. Don’t assume it does; test it.

4. Estimate the monthly cost delta

For any task you decide to keep on Fable 5, estimate your monthly token volume and multiply by $10/M input and $50/M output. That number is what the free window has been hiding from you. When you route everything else down to Opus or Sonnet, the cost becomes a fraction of that. The mistake is defaulting your entire fleet to Fable 5; keeping one or two critical tasks there while routing the rest is usually very affordable.

What about jumping ship to Sol?

If you’re already re-evaluating your stack and your agents are pure coding agents, it’s worth looking at OpenAI’s GPT-5.6 Sol. It launched on July 9th, and OpenAI reports state-of-the-art coding results, claiming Sol outperforms competing frontier models (Claude included) at a lower cost. Just remember that Sol is an outside option from OpenAI, not part of the Claude-native ecosystem. It’s a strong cross-provider alternative if your primary metric is coding benchmark performance.

Share As The Geek Learns

Why this matters

The migration you do this week isn’t just about avoiding a bill; it’s the first draft of a permanent model-routing layer. Relying on a single “best” model is a fragile strategy. The durable answer is to build a system that chooses the right model for the right task based on cost, latency, and required intelligence. By doing this work now, you’re moving toward a more professional architecture where you control your costs instead of letting a subscription change dictate your uptime.

Quick reference

Deadline: July 19, 2026, 11:59:59 PM PT

Fable 5 Cost: $10/M input, $50/M output (prepaid credits)

Coding Recommendation: Opus 4.8 at effort xhigh ($5/M in, $25/M out)

Mid-tier Target: Sonnet 5 ($3/M in, $15/M out; intro pricing $2/$10 through Aug 31)

High-volume Target: Haiku 4.5 ($1/M in, $5/M out)

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns.

When exactly does free Fable 5 access end?

July 19, 2026, at 11:59:59 PM PT. Until then, Pro, Max, Team, and premium Enterprise plans can use Fable 5 for up to 50% of their weekly usage limits at no extra cost.

What does Fable 5 cost after July 19?

Prepaid usage credits at $10 per million input tokens and $50 per million output tokens. That is the price the free window has been hiding.

What is the best Claude replacement for coding agents?

Claude Opus 4.8 at effort xhigh. It’s Anthropic’s own coding recommendation and the Claude Code default, at $5/M input and $25/M output, exactly half of Fable 5’s token cost.

Will Anthropic bring Fable 5 back to subscriptions?

Anthropic says the credit-billing arrangement is temporary, and it aims to return Fable 5 to standard subscriptions when compute capacity allows, but there is no timeline. Plan as if July 19 is final.

Is OpenAI’s Sol better than Claude for coding?

Sol (GPT-5.6 Sol) posted state-of-the-art coding benchmarks at launch (Sol Ultra scored 91.9% on TerminalBench 2.1). It’s a cross-provider option worth benchmarking, but it is an OpenAI model, not a Claude one.

Do I have to move everything off Fable 5?

No. This is a routing problem, not an all-or-nothing switch. Keep one or two genuinely quality-critical tasks on Fable 5 and route the rest to Opus 4.8, Sonnet 5, or Haiku 4.5.

Leave a comment

My AI MOA Experiment Kernel-Panicked My Mac Studio Three Times

James Cruce — Mon, 06 Jul 2026 13:16:45 GMT

I set out to test whether a crowd of AI models writes better code than one good one. The experiment said no. Getting to that no involved crashing my Mac Studio three times, watching my AI assistant confidently blame the wrong thing, and catching it with the kernel's own flight recorder. This is the story of the crash, the one-line bug behind it, and why the unglamorous part was what mattered.

A machine with 256 GB of RAM should not run out of memory. Mine did. Three times in two days, and every time it happened during the part of the run I'd written off as the safe part: no AI models loaded, no network calls, just a script re-reading a results file I already had on disk. The screen would freeze, the fans would spin up, and then macOS would reboot itself and hand me a panic report. I want to walk you through how I proved it was my code and not my hardware, because the method matters more than the bug.

What I was actually testing

The project is a benchmark. There's a paper making the rounds called Mixture of Agents, or MoA. The idea is appealing: instead of asking one strong model to write code, you ask several models to each propose a solution, then a final "aggregator" model synthesizes their proposals into one answer. More heads, better output. I wanted to know if that holds up on real coding tasks, at equal cost, running mostly on local models on my Mac Studio. So I built a harness that fans a task out to several proposers, merges the results, and scores the merged code against real tests.

I did one thing up front that saved me later. Before running anything, I froze the whole experimental design in a decision record: the exact tasks, the token budget, and the statistical rule for calling a winner. Preregistration. It felt like overkill for a personal project. It wasn't.

The safe part crashed the machine

Phase 2 finished, and the answer was already leaning negative, so I moved on to attribution runs to understand why. One of those was pure re-analysis. It re-reads an existing results file and splits it by task difficulty. No model calls at all. It's the lightest thing in the whole codebase.

It kernel-panicked the Mac. Twice more after that.

The obvious move is to blame load. Close the background apps, kill the backup daemon, and blame Spotlight. My AI pair programmer went straight there and told me the culprit was probably concurrent memory pressure from other processes. Confident, plausible, and completely backwards.

Reading the kernel's black box

macOS writes a panic report when it crashes, and buried in it is a stackshot: a snapshot of every process at the moment of death. This is the black box recorder, and most people never open it. The file was in a folder that's easy to miss, /Library/Logs/DiagnosticReports/Retired/.

One process stood out. A Python process, pid 14869, sitting at 1,279 seconds of user CPU time, 48 GB resident, and only 430 page-ins. That last number is the tell. A process that's a victim of memory pressure shows heavy paging as the system thrashes to feed it. This process had almost none. High CPU, low paging: that's not a victim starving for memory. That's a program burning the processor in a tight loop, allocating as fast as it can. The evidence pointed at my code, not the environment.

So I did the thing I should always do when the AI and I disagree. I asked for an adversarial review. Three separate review agents, each told to attack a different part of my diagnosis and try to break it. One of them found the stackshot line that settled it. The backup daemon everyone wanted to blame was using half a gigabyte. It was innocent, and I could prove it from the report instead of superstitiously killing it.

The one-line bug

Here's the whole thing. The analysis code built a filtered copy of the results using Python's dataclasses.replace(). What I forgot is that replace() copies field references, not the data behind them. My "copy" shared the original's list. So the loop was appending rows to the exact list it was iterating over. Every row it added matched the filter and got added again. An infinite, self-feeding loop that allocated roughly 200 GB in about 23 minutes until the memory compressor gave out and the kernel panicked.

# replace() copies field *references*, so without results=[] the
# "filtered" run shares the source's mutable list, and the loop below
# appends to the very list it's iterating. Unbounded growth until the
# memory compressor runs out of segments and the kernel panics.
filtered = FactorialResults(
    run=replace(results.run, task_ids=tuple(sorted(task_ids)), results=[]),
)

The fix is results=[]. Two words. My first diagnosis failed because I reasoned about the algorithm as I designed it ("this can't allocate 256 GB, that's impossible") instead of the code as I actually wrote it.

The next crash was a different kind entirely

I built a memory watchdog so this couldn't happen again. The very next run wedged. No panic this time. Memory sat at 98% free for over two hours while the run produced nothing. My shiny new watchdog never fired, because it only knew how to watch for memory trouble, and this wasn't memory trouble.

It was a Unix job-control deadlock. A grading subprocess ran some model-generated code that happened to read from standard input, and because the whole run lived in a background process group, reading the terminal suspended the entire group with a SIGTTIN signal. The fix was structural, not another alarm: hand every subprocess stdin=subprocess.DEVNULL so untrusted generated code can never touch the terminal. That's the real lesson from the pair of incidents. Monitoring only catches the failure classes you already imagined. The next failure is always a new class.

with 98 percent free memory and the watchdog stays silent.","title":null,"type":"image/png","href":null,"belowTheFold":true,"topImage":false,"internalRedirect":"https://astgl.com/i/204782250?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff66245dd-e295-4fff-8a0b-28eed9acbc55_1083x196.png","isProcessing":false,"align":null,"offset":false}" class="sizing-normal" alt="Three steps: a memory kernel panic, then a memory watchdog is built, then a deadlock strikes > with 98 percent free memory and the watchdog stays silent." title="Three steps: a memory kernel panic, then a memory watchdog is built, then a deadlock strikes > with 98 percent free memory and the watchdog stays silent." srcset="https://substackcdn.com/image/fetch/$s_!YCEj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff66245dd-e295-4fff-8a0b-28eed9acbc55_1083x196.png 424w, https://substackcdn.com/image/fetch/$s_!YCEj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff66245dd-e295-4fff-8a0b-28eed9acbc55_1083x196.png 848w, https://substackcdn.com/image/fetch/$s_!YCEj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff66245dd-e295-4fff-8a0b-28eed9acbc55_1083x196.png 1272w, https://substackcdn.com/image/fetch/$s_!YCEj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff66245dd-e295-4fff-8a0b-28eed9acbc55_1083x196.png 1456w" sizes="100vw" loading="lazy">

The result was no, and that's the point

After all of that, the benchmark's answer was clean and disappointing: the crowd of models did not beat the single best model at equal cost. Not on easy tasks, not on hard ones, not with a stronger aggregator, not with a bigger budget. Every ensemble arm cost two to seven times more to be, at best, statistically tied.

And this is where the boring discipline paid off. One configuration's raw score, 0.950, actually sat above the best single model at 0.935. It would have been so easy to write the headline "hybrid AI matches the cloud at half the cost." But the frozen rule I'd written weeks earlier said to compare paired differences, and that difference wasn't significant. Under my own preregistered rule, it was a tie, not a win. The rule existed precisely to stop me from believing a number I wanted to believe.

That's the whole thread. The same discipline that let me overrule my AI's confident wrong guess is what stopped me from overruling my own honest result. A rigorously produced "no" is a real deliverable.

Quick Reference

Read the panic report before blaming load. Stackshots live in /Library/Logs/DiagnosticReports/ (check the Retired/ subfolder too).
High CPU plus low page-ins means a runaway loop, not a memory victim. The paging counter tells you which.
dataclasses.replace() copies references. Pass a fresh value for any mutable field you don't want shared.
Give untrusted or generated subprocesses stdin=subprocess.DEVNULL so they can't wedge on the terminal.
Freeze your decision rule before you see the data. It's cheap insurance against believing your own hype.

Leave a comment

Found this useful? I share practical lessons from my systems engineering journey into AI at As The Geek Learns

Share As The Geek Learns

I Mined My Own Claude Code Config Files. Here's What I Found.

James Cruce — Wed, 01 Jul 2026 11:46:43 GMT

After months of using Claude Code, I had a confession to make: I had no idea what was in my own config files anymore.

A global CLAUDE.md. Ten project memory indices. A hundred individual memory files. Twenty-three project-level CLAUDE.md files. Eleven custom command files. Six rules files scattered across projects. Every one of them contained something I'd had to explain to Claude more than once. A workflow I repeated, a mistake that burned me, or a preference I'd had to correct. But none of it was organized. None of it was searchable. None of it was reusable in any deliberate way.

So I ran an experiment. I asked Claude to mine the entire thing.

The Setup

The premise was simple: if I've been writing instructions to Claude for months, those instructions should contain patterns. The same workflows keep showing up. The same preferences get stated and restated. The same procedures get encoded in different files for different projects.

What if I treated my own config files as a dataset?

The goal: find every pattern that appeared in two or more source files, group them by domain, and generate a structured SKILL.md runbook for each one.

Source files to read:

The 80+ source files, grouped by type.

`~/.claude/CLAUDE.md` (228 lines of global instructions)
All `MEMORY.md` index files across project memory directories
All individual `memory/*.md` files (about 100 of them)
All project `CLAUDE.md` files (23 across active projects)
Custom commands in `.claude/commands/*.md`
Rules files in `.claude/rules/*.md`

That's roughly 80+ files and several thousand lines of accumulated AI instructions.

What's Actually Going On

Here's the thing about Claude Code config files that took me a while to internalize: they're not documentation. They're not notes to self. They're a pattern library in disguise.

Every `feedback_.md` memory entry represents a correction I had to make more than once. Every `project_.md` entry encodes context that was load-bearing enough to write down. Every custom command I built was a workflow I ran often enough to automate.

The problem is that none of it is explicit. The patterns are implicit in the files. You wrote them under pressure, one at a time, solving immediate problems. You weren't thinking, “This is the third time I've explained the auto-merge workflow." You were just... explaining it again.

Skill mining makes the implicit explicit. It takes the accumulated knowledge in your config files and surfaces the signal: here are the things you keep doing, and here they are, written down properly so you can actually use them.

The Process

The mining session had four steps.

Step 1: File discovery. Find every source file using `find` and `wc -l`. List them. Count the lines. This alone is revealing—seeing 228 lines in your global CLAUDE.md and realizing you've written that many rules for yourself is a moment.

Step 2: Read everything in parallel. All MEMORY.md index files first, then the largest individual memory files, then project CLAUDE.md files, then custom commands and rules. Read them all, not to understand each one, but to see what keeps showing up.

Step 3: Build the frequency table. For each pattern identified, track what's the pattern, how many files mention it, and which files. The threshold is two occurrences. If a workflow or preference appears in two or more places, it deserves a dedicated runbook.

Step 4: Write SKILL.md files. One per pattern, grouped by domain, with a consistent structure: name, description, trigger conditions (when to use it), step-by-step instructions, and notes for edge cases.

What We Found

Twenty-two patterns with two or more occurrences across the 80+ files. Here's the frequency table:

The top tier (6 occurrences each) had four patterns: the Ironclad Workflow (my 4-phase dev process: Plan-Execute-Verify-Ship), gstack skill routing, the ClaudeClaw agent architecture primitives, and the Cortex memory tagging system. These showed up in six different files each. They're the backbone of how I work.

Below that: conventional commits (5 files), the ASTGL publishing pipeline (5 files), auto-merge PRs (4), secrets management via pass-cli and Keychain (4), the Resist and Rise editorial workflow (4 command files defining a complete journalism process), and the ACA Council product pipeline (4).

Then a cluster of patterns that appeared in three files each: session logging convention, MCP server distribution strategy, LaunchAgent services for macOS background processes, knowledge base article creation, and subagent model selection.

And a final tier at two occurrences: the rule about replacing Markdown tables with `[GRAPHIC:]` callouts in Substack articles, the 1-to-15+ content repurposing workflow, embedding-based relevance scoring, the Crucible fiction writing system, two-phase fact-checking, idea pool submission, and the agent self-improvement methodology.

The 22 patterns extracted, sorted by how many source files they appeared in.

Twenty-two patterns. Twenty-two things I do repeatedly that I'd never looked at as a unified system

Why This Matters

There are a few things that surprised me when I saw the results laid out.

First, the coverage. I had patterns across six distinct domains: general dev workflow, ClaudeClaw agent configuration, ASTGL content, Resist and Rise journalism, secrets management, and fiction writing. I knew I worked across those areas, but seeing the patterns formalized across all of them in one place made the scope visible in a way it hadn't been before.

Second, the consistency. The Ironclad Workflow: ‘Plan-Execute-Verify-Ship’ showed up in six different project files. I never set out to teach it to six different projects. It crept in because it works, and I kept re-encoding it. That's the signature of a real pattern: it keeps appearing because you keep reaching for it, even when you're not thinking about it explicitly.

Third, what was missing. The domains I didn't find patterns in are equally informative. No PowerShell patterns. No VMware patterns. Those workflows exist, but they're not yet in my Claude Code config. That's a list of skills to mine next.

Quick Reference: Run Your Own Skill Mining Session

If you want to do this yourself, here's the structure:

Source files to mine:

`~/.claude/CLAUDE.md` (global instructions)
`~/.claude/projects/*/memory/MEMORY.md` (memory indices)
`~/.claude/projects//memory/.md` (individual memory files)
All `CLAUDE.md` files in `~/Projects/*/`
All `.claude/commands/*.md` files
All `.claude/rules/*.md` files

Minimum frequency: 2 occurrences to write a SKILL.md

SKILL.md structure:

Output location: `~/skill-mining/mined///SKILL.md`

The prompt: Ask Claude to read all source files, build a frequency table (pattern name, occurrence count, source files), group by domain, and generate a SKILL.md for each pattern with 2+ occurrences. Generate an INDEX.md listing all skills with one-line descriptions and source evidence.

The whole session takes less than an hour. The patterns were already there. You just needed to ask.

I write about partnering with AI to build real systems at As The Geek Learns. If you've been using Claude Code long enough to accumulate config files, you've already done the hard work. Mine them.

Leave a comment

Frequently Asked Questions

What is skill mining in Claude Code?

Skill mining is reading back your own accumulated Claude Code config—CLAUDE.md files, memory entries, custom commands, and rules—to find workflows you repeat, then writing each repeated pattern as a structured SKILL.md runbook. It makes implicit patterns explicit and reusable.

What’s the minimum frequency to write a SKILL.md?

Two occurrences. If a workflow or preference shows up in two or more source files, it’s a real pattern worth a dedicated runbook. One-off instructions stay where they are.

How many config files do I need before this is worth it?

Enough that you’ve lost track of what’s in them. In this run that was 80+ files—a 228-line global CLAUDE.md, ~100 memory files, 23 project CLAUDE.md files, 11 commands, and 6+ rules files—but the threshold is “I keep re-explaining things,” not a file count.

Where should I save the mined SKILL.md files?

One file per pattern, grouped by domain—for example, `~/skill-mining/mined///SKILL.md`, plus an INDEX.md listing every skill with a one-line description and its source evidence.

Does this work with Cursor or only Claude Code?

The technique is config-agnostic—any AI tool where you accumulate written instructions has a mineable pattern library. This walkthrough uses Claude Code’s CLAUDE.md, memory, and command files specifically.

How long does a skill-mining session take?

Under an hour. Discovery and reading are fast; the patterns are already written down. You’re surfacing them, not inventing them.

Share As The Geek Learns

Apple Container vs Docker: When to Use Which on Apple Silicon

James Cruce — Sat, 27 Jun 2026 17:30:28 GMT

Three container runtimes, one Mac Studio. I benchmarked Apple `container`, Docker Desktop, and OrbStack on Apple Silicon—here’s where each one wins.

Apple's container CLI hit 1.0 this month, picked up 30,000 GitHub stars, and arrived with the usual promise: faster than Docker on Apple Silicon, lighter, and more native. It runs Linux containers in lightweight VMs, written in Swift, optimized for your M-series chip.

I wanted to know if the promise holds. So I didn't read the marketing. I benchmarked it.

And I added a third contender Apple didn't mention: OrbStack. Because if you've already gotten tired of Docker Desktop on a Mac, OrbStack is probably what you switched to. Leaving it out of a "fast containers on Apple Silicon" comparison would've been lacking since it's the one a lot of us actually run.

Here's what the numbers say and how to decide which one belongs on your machine.

How I tested this

Quick note on method: because a benchmark you can't trust is just a screenshot of one lucky run.

Everything ran on a Mac Studio: M3 Ultra, 256 GB RAM, macOS 26.5.1. All three tools pulled the same images, pinned by digest, so nobody got a different build. Every measurement is the median of at least 10 runs, warmup discarded, each run in a clean subprocess so state from one couldn't bleed into the next.

Nine scenarios: container startup, a web server coming up, Postgres accepting connections, CPU work, disk I/O on a bind mount, container-to-container networking, an image build, a multi-service stack, and a density ramp. Plus idle footprint.

The whole harness is public if you want to reproduce it or pick it apart: github.com/Jmeg8r/apple-container-vs-docker. Every claim below traces to a number in that repo or a failure I logged on purpose.

One thing that matters before we start: all of this is on macOS 26 (Tahoe). On macOS 15, Apple container can't even do container-to-container networking. Containers get IPs but can't talk to each other. If you're not on Tahoe yet, half of this comparison doesn't apply to you, and the answer is "stay on Docker or OrbStack." For everyone else, read on.

The one number where Apple wins outright

Let's lead with Apple's real victory, because it's a good one.

Idle memory with nothing running—Apple `container` ~51 MB vs Docker Desktop ~1,124 MB and OrbStack ~1,631 MB.

Idle footprint. With nothing running—no containers, just the runtime sitting there ready. Apple container holds about 51 MB of memory. Docker Desktop sits at 1,124 MB. OrbStack at 1,631 MB.

That's not a typo. Apple container uses 22 to 32 times less memory at rest than the other two.

The reason is architectural. Docker Desktop and OrbStack each run one big Linux VM that stays resident the whole time you're logged in, whether you're using it or not. Apple container doesn't keep a VM warm. It spins one up per container and tears it down when the container stops. Nothing running means nothing resident.

If you keep a couple of long-lived services on your laptop and want them out of the way the rest of the day, this is a genuinely meaningful win. Your RAM is yours again when you're not using containers.

Hold onto that "spins up a VM per container" detail, though. It's also the source of every place Apple loses.

Where it ties and quietly wins again

Before the losses, give Apple its due on two more.

Raw CPU is a dead heat. Running sysbench inside a container, Apple container actually came out slightly ahead, about 38,500 events per second versus 36,400 for Docker Desktop and 33,500 for OrbStack. The per-VM model adds no real tax on compute. Once your code is running, it runs at full speed.

Cached builds are Apple's fastest. Rebuild an image when the layers are already cached, and Apple finishes in about 426 ms. Quicker than Docker Desktop (598 ms) and noticeably quicker than OrbStack (844 ms).

Cached image rebuild—Apple is fastest at ~426 ms vs. 598 (Docker Desktop) and 844 (OrbStack).

So this isn't a story about a slow tool. Where the VM boundary isn't sitting in the hot path, Apple container is right there with the best of them and sometimes ahead.

Thanks for reading As The Geek Learns! This post is public, so feel free to share it.

The per-VM tax: everywhere it creates or moves data

Now the other side. And it's consistent enough that you can predict it: anything that involves spinning up a container or pushing data across the VM boundary costs Apple time.

Startup. A bare container's start-to-exit takes Apple about 1,071 ms versus ~360–395 ms for the others.

Bare container start-to-exit—Apple ~1,071 ms vs. ~360–395 ms for the others.

Nginx answering its first request: 1,095 ms versus ~290 ms. Postgres accepting a connection: 2,081 ms versus 292 ms on Docker Desktop. That's 3× to 7× slower, depending on the workload. Every time you start something, you're paying for a VM to boot.

Bind-mount disk I/O—this is the one that'll bite you. Mount a folder from your Mac into the container and write to it, and Apple manages about 32.6 MB/s. Docker Desktop does 96.7. OrbStack does 548.5 MB/s which is roughly 17 times faster than Apple.

Sit with that one, because it's not an abstract benchmark. Mounting your source tree into a container and editing it live. The hot-reload dev loop, the thing a huge number of Mac developers do all day, runs straight through this path. On Apple container, that loop is going to feel like wading through mud.

Bind-mount write throughput—Apple ~32.6 MB/s vs. Docker Desktop 96.7 and OrbStack 548.5 (~17× faster than Apple).

Networking between containers follows the same shape: ~11 Gbps for Apple, ~53 for Docker Desktop, and ~85 for OrbStack. Real network hops between separate VMs versus traffic staying inside one shared VM.

And density. Spinning up 40 containers took Apple about 35 seconds. Docker Desktop and OrbStack did it in roughly 9. Forty VMs versus forty processes in one VM—the model shows up again. (To be fair, none of the three actually fell over at 40; I capped it there to be kind to my machine. Finding Apple's true ceiling is a job for another day.)

None of this is a bug. It's the direct, honest consequence of one lightweight VM per container. You're trading creation speed and I/O for isolation and a clean idle state. Whether that's a good trade depends entirely on what you do all day.

The gaps that have nothing to do with speed

Numbers aside, a few things will just stop you cold, and these matter more for daily use than any millisecond.

No Docker Compose. This is the big one. If your project starts with docker compose up, Apple container has no native answer. You're back to creating a network and running each service by hand, or leaning on an unofficial third-party bridge. Docker Desktop and OrbStack both speak Compose fluently.

Image names have to be fully qualified. container run alpine fails. You need container run docker.io/library/alpine. Small thing, but it'll trip every script and muscle-memory habit you have.

Port publishing isn't the model you know. I expected -p 8080:80 to put the service on localhost:8080 like Docker does. On Apple container it didn’t—the container was serving fine, but on its own IP address, not on localhost. Apple's model is "every container gets a real routable IP," which is elegant, but it's not the muscle memory you've built. (Related gotcha: poll 127.0.0.1, not localhost — localhost can resolve to IPv6 first and miss the service entirely.)

DevContainers support is incomplete. If you live in VS Code's dev containers, you're not there yet.

If you want to try it yourself

It's a five-minute install. The one non-obvious step is the kernel.

# Apple container
brew install container
container system kernel set --recommended   # do this first, or `start` prompts you
container system start

# OrbStack, if you want to compare
brew install --cask orbstack

Then remember the gotchas above: fully-qualified image names, reach services by their container IP, and 127.0.0.1 over localhost. That's most of the friction right there.

So which one do you actually use?

Here's how I'd decide.

Reach for Apple `container` when you run a handful of long-lived containers and you care about isolation and a clean idle footprint—a couple of always-on services on a laptop you also use for everything else. The per-container VM boundary is real security isolation, and getting your RAM back when you're idle is a nice perk. Just don't point your live-editing dev loop at it.

Reach for OrbStack if you want fast Docker without the friction. It won every speed test that involved I/O or networking, starts containers as fast as anything, and crucially, it still speaks Compose and the full docker CLI. The only price is the biggest idle footprint of the three and the slowest cached build. For most Mac developers, this is the easy pick.

Stay on Docker Desktop when you need the whole ecosystem—Compose, DevContainers, and the broadest tooling and documentation. It sat in the middle of nearly every benchmark, and "middle of the pack with everything supported" is exactly what a default should be.

The headline I came in expecting — "Apple container is faster than Docker" — just isn't what the data shows. It's faster at a few specific things and slower at most of the rest. But that's not a knock. It's a specialist. It does one shape of work really well and asks you to give up the conveniences you've built your workflow around.

Know which shape of work you're doing, and the choice makes itself.

The full harness, raw numbers, and every chart are public: github.com/Jmeg8r/apple-container-vs-docker Run it on your own machine. I'd genuinely like to see whether the bind-mount gap holds on an M1 or M2, or whether 256 GB of headroom is flattering somebody.

Share As The Geek Learns

Leave a comment

Local LLMs Plus Claude Code: The Mac Studio Hybrid Workflow

James Cruce — Tue, 23 Jun 2026 11:04:12 GMT

"Isn't that expensive?" Every time I talk about using Claude Code for a project, someone asks this. The honest answer is: it depends entirely on what you route to Claude versus what you run locally. On a Mac Studio with unified memory, the economics change fast. Here's the routing table I actually use.

The Setup

I have an M3 Ultra with 256 GB unified memory. That machine runs a 70B model locally with room to spare and a 235B mixture-of-experts model when I need frontier-scale reasoning. Running it at load draws around 60 watts. Eight hours of overnight inference work costs about five cents in electricity at typical US rates. The same run on a Frontier API would cost somewhere between fifteen and thirty dollars.

But cost isn't the primary reason to run locally. Latency is. My workhorse model, gemma4:31b-mlx, stays pinned in GPU memory, so there's no cold-start delay. It begins responding the moment you hit enter on M-series hardware. That's interactive. You can iterate on code at conversational speed with a local model, then bring Claude in for the decisions that actually require judgment.

Most developers running Claude Code for everything are paying for two things: generation (which local models handle well) and judgment (which frontier models handle better). Splitting those tasks cuts the API spend dramatically while keeping the quality where it matters.

What's Actually Going On

The routing principle has one line: "Claude reads the playbook. Local models do the work."

Local models handle generation: the fast, cheap, iterative part. New code, first-draft text, refactoring suggestions, variant generation. Claude Code handles orchestration and the Compound step: planning, judging what goes in learnings.jsonl, reviewing the session output, and updating CLAUDE.md.

The model routing table I use:

The everyday models are light. gemma4:31b-mlx loads in about 20 GB on disk and sits pinned in GPU memory (around 46 GB resident at its full 256K context). qwen3-coder:30b, a mixture-of-experts model that activates only ~3B parameters per token, loads in another 18 GB. On an M3 Ultra with 256 GB, both stay resident at once with enormous headroom enough to also pull a 120B or even a 235B reasoning model on demand. That last one is the whole point of the memory: a 235B model at Q4 needs around 142 GB and simply won't load on a smaller machine.

deepseek-r1:70b needs 64 GB or more. It's not interactive, responding in 20-30 seconds per generation. That's fine for overnight batch jobs. It's not fine for a code review you're waiting on.

The Fix

Setting up Ollama on a Mac Studio takes about 10 minutes:

brew install ollama
ollama pull gemma4:31b-mlx     # primary workhorse (MLX build, Apple-Silicon optimized)
ollama pull qwen3-coder:30b    # code generation + review (MoE, strong tool calling)
# Heavy reasoning — only comfortable with lots of unified memory:
# ollama pull deepseek-r1:70b                          # ~42 GB, overnight reasoning
# ollama pull gpt-oss:120b                             # ~65 GB
# ollama pull qwen3:235b-a22b-thinking-2507-q4_K_M     # ~142 GB, needs 256 GB

Open the Ollama menu bar app once to enable auto-start on login. It runs as a local HTTP server at localhost:11434.

The session structure for a typical day:

Morning: Planning (Claude Code)
  - Read CLAUDE.md + learnings.jsonl
  - Brainstorm, write plan with verify steps

Work (local models via Ollama)
  - gemma4:31b-mlx for generation and first drafts
  - qwen3-coder:30b for code and "catch what I missed"

Evening: Compound Step (Claude Code)
  - Review session, propose learnings
  - Update CLAUDE.md Known Patterns table
  - Check tests, commit

Overnight (optional, if you have enough RAM for a 70B model):
  - deepseek-r1:70b running autoresearch loop
    on a skill file or prompt variant

The Compound step is where the two frameworks connect. Karpathy's Autoresearch pattern (one file, one metric, keep/revert with git) maps directly onto a local overnight job. Instead of optimizing a training script, you optimize a skill file or a prompt file from your .claude/commands/ directory.

Define a set of test cases: files with known issues your skill should catch. Run variants overnight. Keep the variants that find more issues; revert the rest. In the morning, run the Compound step with Claude on the winning variants.

The autoresearch loop becomes the Work step in a Compound Engineering session.

Why This Matters

I've had a real-world data point on this from the stoicism-agent project, where I ran the first overnight autoresearch loop using local models. The learning I wrote after that run:

"The fork between MLX and Ollama for autoresearch comes down to one question: do you need to modify model weights? MLX if yes. Ollama for prompt-level optimization. For skill file autoresearch, Ollama is the right choice."

That's the kind of learning you only get by running the thing. The instinct before running it was to reach for MLX because it's the "native" Mac AI framework. The result after running it: Ollama is simpler, has better model selection, and is more than fast enough for overnight prompt optimization where you're not touching weights.

Same principle scales to any overnight optimization loop. Pick the tool that matches the task. Don't reach for fine-tuning when prompt-level optimization will do.

The cost math on the Mac Studio, after a few months of this workflow: roughly 90% of generation work routes to local models. The 10% that goes to Claude is the judgment layer: planning, compound steps, final review. Total Claude API spend for a typical weekend session runs $2-3. For a full week of this, maybe $8-10.

The Mac Studio has nearly paid for itself in API savings after about 7 months. More importantly, the learnings file from six months of consistent Compound Engineering is now the most valuable artifact in my projects. Not the code. The 200+ preserved decisions.

Code can be rewritten. Those decisions can't.

Install: brew install ollama, open Ollama.app for auto-start
Primary workhorse: gemma4:31b-mlx (~20 GB, pinned in GPU, MLX-optimized)
Code generation + review: qwen3-coder:30b (~18 GB, MoE)
Heavy reasoning: gpt-oss:120b (~65 GB) or qwen3:235b-a22b-thinking (~142 GB, needs 256 GB)
Overnight reasoning: deepseek-r1:70b (needs 64 GB+, ~20-30s/response)
Never route to local: Compound step, final review, architectural decisions
Autoresearch overnight: use deepseek-r1:70b for skill/prompt optimization, not weight training
Also resident: nomic-embed-text for embeddings, a custom voice model for narration
Smaller machines: a 76 GB M2 Ultra runs gemma4:31b-mlx + qwen3-coder:30b fine, but the 120B–235B tier needs the 256 GB
Full routing table: docs/local-llm-routing.md in the template repo

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns (https://astgl.substack.com)

Share As The Geek Learns

Leave a comment

SEO Isn't Dead — But the Click Might Be

James Cruce — Mon, 22 Jun 2026 11:03:39 GMT

Listen now | SEO and AEO explained: Explore how AI overviews drive zero-click searches and why being cited by AI assistants is the new way to capture high-converting…

Local vs. Frontier Models: Has the Gap Closed?

James Cruce — Sat, 20 Jun 2026 00:01:04 GMT

Listen now | Local AI models vs frontier APIs: explore the cost, privacy, and performance gaps in reasoning and agentic tasks for enterprise workloads.

I Ran Google's 1,000-Tokens-Per-Second Model on My Mac. A Normal Model Beat It.

James Cruce — Fri, 19 Jun 2026 13:15:26 GMT

There's a number floating around that's hard to ignore.

Google's new DiffusionGemma is supposed to crank out 1,000-plus tokens per second. For comparison, the local models most of us run putter along at 30 to 100. So a 10x jump? That gets my attention.

The reason it's fast is genuinely interesting. Every model you've used—ChatGPT, Claude, your local Llama—writes one token at a time, left to right, each word waiting on the one before it. That's "autoregressive." Diffusion models work completely differently. They start with a blank canvas of 256 tokens and refine the whole block at once, in parallel, like a photo developing. No waiting in line.

On paper, that's the future. So I did the obvious thing: I ran it on my Mac Studio to see if the future had arrived on my desk.

It hadn't. And the way it hadn't turned out to be more interesting than a win would've been.

The fair fight

Here's the thing that makes this a clean test instead of a vibe check.

DiffusionGemma is built on the same bones as Gemma 4—Google ships an autoregressive Gemma 4 26B A4B that's the same size, same architecture, same weights. The only difference is how it generates: diffusion vs. one token at a time.

So I put them head to head. Same Mac. Same 8-bit quantization. Same runner (Apple's MLX). Same prompts, same everything. The one variable left standing is the decoding paradigm itself. If diffusion is faster, this proves it. If it's not, there's nowhere to hide.

Thirty prompts across code, math, instruction-following, and writing. Five runs each. Let's look.

The result nobody puts in the headline

DiffusionGemma did about 43 tokens per second on my Mac.

The boring old autoregressive model did 61.

The diffusion model, the one that does 1,000+ on a datacenter GPU, was the slower of the two on Apple Silicon. Not by a hair. By 40%.

That gap between the headline and my desk is about 23x. The 1,000 tok/s is real; it's just real on an H100, a $30,000 datacenter card. On a Mac, that number has nothing to do with your life.

And it gets worse for diffusion if you care about how snappy a chat feels. There's a metric called time-to-first-token, how long you stare at a blank screen before words start appearing. The autoregressive model started typing in 0.12 seconds. DiffusionGemma took 1.86.

That one surprised me until I thought about it. Remember how diffusion refines a whole 256-token block at once? That's the catch; it can't show you anything until the entire block is done cooking. The "parallel" model that's supposed to feel instant actually feels laggier, because it makes you wait for the batch.

Why the magic doesn't travel

So why does the same model fly on an H100 and crawl on a Mac?

It comes down to what each machine is good at. Diffusion's whole speed trick is doing a giant pile of math all at once, refining 256 tokens in parallel. An H100 has thousands of cores sitting there begging for exactly that kind of bulk work. Flood it, and it's happy.

Apple Silicon doesn't win that way. It's not short on memory. My Mac Studio has 256GB, but it's limited by how fast it can move data around, not how much math it can do at once. The fancy parallel block doesn't help when the bottleneck is the plumbing, not the engine.

Autoregressive decoding, meanwhile, plays to the Mac's strengths. It reuses its previous work (a "KV cache") and touches way less memory per token. Same model, same weights; the architecture that wins in the datacenter loses on the desktop. The hardware decides.

The benchmark that kept slowing itself down

I almost shipped wrong numbers. Here's the part the polished write-ups leave out.

My first full run looked fine for the first 15 or so generations. Then DiffusionGemma started... degrading. Not crashing—slowing. Time-to-first-token climbed from 1 second to 2, then 4, then 18, then 60, and by the 28th generation, a single response took over two minutes. Same prompt that was instant a minute earlier.

My first guess was a memory leak. So I checked. And this is the maddening part: every memory counter the framework reports stayed flat. By the numbers, nothing was wrong. I added the standard "clear the cache between runs" call. No change. I added a "wait for the GPU to finish" call. It nudged the cliff from generation 18 to generation 19 and then fell off it anyway.

The culprit turned out to be the graphics driver itself quietly piling up state that none of the normal tools could see or clear. The only thing that actually worked was brute force: run a handful of generations, then kill the whole process and start fresh. Let the operating system clean up what the framework couldn't.

Two lessons I'm keeping:

Trust the measurement over the marketing and over your own assumptions. If I'd run 15 prompts and called it a day, I'd have published a number that looked great and was completely fake.

Your tools can lie by omission. "Memory usage is flat" is not the same as "nothing is accumulating." The dashboard being green doesn't mean the system is healthy.

Quality: closer, but autoregressive still edges it

Speed isn't everything, so I scored the actual answers too. Code got run and tested. Math got checked against the right answer. Instruction-following got graded against rules. Writing got judged blind.

Overall, autoregressive Gemma came out ahead—0.90 to 0.84—winning code, instructions, and writing, while DiffusionGemma edged it on math. Honestly, that tracks with Google's own advice, which quietly tells you to use the standard model "for maximum quality." It's a real gap, but it's not a blowout.

So the diffusion model on my Mac was slower, laggier, and a notch lower quality. That's not a trade-off. That's just losing.

So should you care?

If you run models locally on a Mac, here's the takeaway: don't reach for DiffusionGemma expecting the headline. You'll get a third of the speed of the autoregressive version, worse responsiveness, and slightly weaker answers. For Mac local inference, boring old next-token generation still wins.

That's not a knock on diffusion models. The approach is genuinely promising, and on the right hardware, it's a rocket. But "the right hardware" is an NVIDIA datacenter card right now, not the machine on your desk.

The bigger lesson is the one I keep relearning: a vendor benchmark is true and useless until you run it on your own hardware. 1,000 tokens per second was a real number that told me nothing about my Mac. The only way to know what a tool does for you is to point it at your setup and watch.

I built the whole benchmark as a reusable harness; same-architecture comparison, real scoring, the works, so when MLX gets a diffusion-optimized path (or llama.cpp's Metal support lands), I can re-run it in an afternoon and see if the story's changed. I suspect it will, eventually. Just not today.

The whole thing—harness, scorers, charts, and the raw results—is on GitHub if you want to poke at it or run it on your own machine: github.com/Jmeg8r/diffusiongemma-benchmark.

Thanks for reading As The Geek Learns! This post is public, so feel free to share it.

Does DiffusionGemma actually hit 1,000 tok/s on a Mac?

No. While it hits those speeds on an NVIDIA H100, I measured only 43 tok/s on my Mac Studio. The “parallel” advantage requires datacenter-grade compute to overcome memory bandwidth bottlenecks.

Why is the time-to-first-token (TTFT) higher for diffusion models?

Diffusion models refine a whole block of tokens (e.g., 256) at once. Because they cannot stream results token-by-token, you must wait for the entire batch to finish before any text appears on screen.

Can I fix the performance degradation in MLX diffusion runs?

The slowdown is caused by graphics driver state accumulation that standard memory tools don’t report. The only reliable fix currently is to run a few generations and then restart the process entirely.

Which Gemma model is better for coding on Apple Silicon?

Autoregressive Gemma 4 is superior. In my benchmarks, it scored higher in quality (0.90 vs 0.84) and was significantly faster (61 tok/s vs 43 tok/s) on Mac hardware.

Is 256GB of unified memory enough to make DiffusionGemma fast?

Memory capacity isn’t the bottleneck; memory bandwidth is. Even with 256GB, the Mac cannot move data fast enough to feed the parallel math required for diffusion speeds.

Running local models on Apple Silicon and want to run this yourself? The full harness is on the GitHub repo diffusiongemma-benchmark. And if you want more of these -"I actually tried it so you don't have to" - breakdowns, subscribe — that's most of what I do here.

Share As The Geek Learns

Leave a comment

I Built My Own Notion With Claude Fable 5 — In One Session

James Cruce — Thu, 11 Jun 2026 14:02:45 GMT

I gave Claude Fable 5 one prompt and walked away for a few minutes. When I came back, it had designed a database schema. By the time I went to bed, I had a fully functional macOS desktop app with a block editor, relational databases, and a calendar that schedules itself. This is the story of how that happened and what it says about where AI-assisted development is right now.

The Prompt That Started Everything

It started simple. I'd been using Notion for years, but it always felt like it was one subscription price increase away from becoming someone else's problem. I wanted something local. Something mine. And I'd been watching what Claude Fable 5 could do on the Max plan, so I decided to push it.

The prompt was direct:

"I want you to build a macOS desktop app. This should be an app that lets you create custom pages with tables, text, images and more exactly like Notion. Use ASTGL branding. Use Convex for the database. Please build the full app, make it incredible and a professionally designed product, and make sure everything works."

That was it. No architecture spec. No wireframes. No feature list beyond "exactly like Notion."

What came back wasn't just code; it was a plan. A full architecture decision: Electron + React 19 + Vite + Tailwind v4 + BlockNote for the editor + Convex running in anonymous local mode (no account, no cloud, data stays on this Mac). Fable 5 made the call that Convex's anonymous local deployment mode was the right choice before I even thought to ask about it. No auth, no monthly fee, reactive by default, data in ~/.convex. That's a good decision.

What Got Built

Geekspace shipped with everything I use Notion for daily.

The block editor works exactly how you'd expect: type / for the command menu, hit # for headings, [] for to-dos. Drag handles, nested blocks, image uploads to Convex storage. The BlockNote library handles the ProseMirror plumbing; Fable 5 wired it into the Convex backend cleanly.

The databases are where it gets interesting. Property types, multiple views (Table, Board, List, Calendar, Timeline), per-view filters, and sorts. Relations between databases and not just visual ones. Two-way synced relation pairs, rollup properties, and status fields with groups. The seeded template drops in a Projects ↔ Tasks ↔ Sprints structure pre-wired with relations and progress rollups. That took one command: npm run seed.

The calendar that schedules itself is the headline feature. Every task with an estimate, a due date, and a priority gets automatically placed into your working hours, packed around fixed events, using earliest-deadline-first ordering. Drag a block, and it locks the engine schedules around it. Past blocks freeze as history. If something can't fit, it surfaces in a "needs attention" panel. Nothing silently drops.

And then I kept asking for more.

The Features I Added

After the first version was running, I started pushing.

"How can I connect my macOS Calendar and email?" Done. Calendar.app events mirror into Geekspace via JXA scripting, show up as fixed busy time the auto-scheduler works around, and display as dotted-edge events on the calendar. The Mail inbox widget pulls your recent messages, shows unread counts, and lets you turn any email into a task with one click.

"Notion has AI Meeting Notes—can you recreate that locally?" Done. One-click recording with a floating recorder, live level meter, pause, and resume. The audio runs through whisper.cpp for transcription, then through a local Ollama model for summarization. It never leaves the Mac. Summaries adapt to meeting type, standup vs. client vs. interview generate different formats. Action items become tasks in one click.

Then I approved a full roadmap for four more features: enterprise search over my ASTGL knowledge base, an AI agent built into the app, project templates, and a docs library. All four shipped in the same session.

The agent piece called ARCHITECT is the one I'm most proud of. I can use both Claude Agent SDK and local llm qwen3-coder:30b using a toggle. Both are wired and running directly inside the Electron main process. The qwen3-coder:30b model is great for tools usage. It connects to a custom MCP server I built called geekspace-mcp that exposes 14 tools covering every meaningful operation in the workspace. Ask ARCHITECT to set up a database for tracking podcast guests and it appears, live, in the sidebar. Ask it what's overdue and it queries the scheduler and tells you. Any MCP client can use this server, which means I can also drive Geekspace from Claude Code itself.

The Bugs That Made It Real

No build story is honest without the debugging.

The Electron window wouldn't launch. Vite was binding to IPv6 (::1), the startup script was polling 127.0.0.1 (IPv4), and they never found each other. One config change: host: "127.0.0.1" in vite.config.ts.

The macOS Mail widget timed out. The Automation permission dialog was blocking the JXA script. The fix required an armAutomation() probe function that pre-triggers the permission window before the actual fetch, with a timeout window.

The local Ollama model (gemma4) wraps its JSON output in markdown code fences even when you ask it not to. The fix: parse from the first { to the last } and ignore whatever surrounds it.

And then there was the ARCHITECT architecture mistake. My first plan routed the agent through ClaudeClaw's chat API but that API is intentionally tool-free for security reasons. The agent could chat, but couldn't actually do anything. I caught this, paused, re-planned, and rebuilt: embed the Agent SDK directly in Electron main, run everything locally. That's the version that works, and works well.

Fable 5 caught the architectural problem during the re-plan conversation and designed the corrected solution. That's not autocomplete. That's engineering judgment.

What Fable 5 Made Possible

Here's the thing that keeps sticking with me: I'm not a developer by trade. I'm a systems engineer who's been learning to build with AI. In a previous chapter of my career this build would have been a months-long project requiring an entire team. What got built here in a few hours included the scheduling engine, the MCP server, the reactive database layer, and the audio pipeline. Just wow!

This was one session.

Fable 5 didn't just write code from my descriptions. It made architecture decisions. It identified when I was about to go down a wrong path (the ARCHITECT routing issue). It designed a pure functional scheduler module with 21 tests. It wired an MCP server from scratch. It debugged IPv6/IPv4 mismatches and macOS permission timing issues.

The code it writes is genuinely good code. It typed, tested where it matters, and followed established patterns. I watched Fable 5 write code and spin up an agent to perform an adversarial review more than once. The debugging process felt like working with a senior engineer who happened to also be infinitely patient about explaining tradeoffs.

The Series: Building Geekspace in Public

This article is the start of something bigger. I'm planning a full series on what I built, how it works, and what I learned. Here's where we're going:

Part 1 — You're reading it. The origin story: one prompt, one session, a complete Notion-style macOS app.

Part 2: The Calendar That Schedules Itself

A deep dive into the auto-scheduling engine. How earliest-deadline-first + priority + chunking actually works. How locked blocks, frozen history, and "needs attention" surfacing change the way you think about task management. Why I built it as a pure module with its own test suite and why that decision saved me three times.

Part 3: Five Bugs, Five Fixes — Debugging With Fable 5

The IPv6/IPv4 split-brain. The Automation permission race condition. The gemma4 JSON fence problem. The orphaned Convex backend port. The wrong architecture I almost shipped. Each one is a real debugging story with a real lesson about building local AI-native apps.

Part 4: AI Meeting Notes, 100% Local

Whisper.cpp + Ollama + a floating recorder UI. How I rebuilt Notion's AI meeting notes without sending audio anywhere. The pipeline that took five iterations to get right. The model behavior quirks you don't find in the documentation.

Part 5: One MCP Server, Three Runtimes

geekspace-mcp is a standard stdio MCP server. ARCHITECT runs it from Electron main. Claude Code can run it from the terminal. Claude Desktop can run it too. Building a workspace tool that any agent can drive and what that means for how AI and personal software intersect.

Part 6: What I'd Do Differently

Honest retrospective. The choices that worked out. The ones I'd reconsider. What building a complete app in one session actually costs you in terms of technical debt and understanding.

Video Walkthrough of the Geekspace Build

The full app is called Geekspace. It runs on my Mac, stays on my Mac, and does everything I actually use Notion for. If you want to follow the build in public as I write about it, subscribe below — Part 2 drops next.

Subscribe now

Frequently Asked Questions

Can Claude Fable 5 really build a full app from one prompt? Yes — in one session, from a single prompt, it produced a working macOS app with a block editor, five database views, an auto-scheduling calendar, local AI meeting notes, and an embedded agent. It chose the architecture itself before being asked.

What stack did Claude choose for a Notion-style desktop app? Electron + React 19 + Vite + Tailwind v4 + BlockNote for the editor, with Convex running in anonymous local mode — no account, no cloud, data stored in ~/.convex on the Mac.

Is a locally built Notion alternative actually private? Yes. Geekspace keeps data in a local Convex deployment, and the AI meeting notes run whisper.cpp plus a local Ollama model — the audio never leaves the Mac.

How do you give an AI agent tools inside a desktop app? The ARCHITECT agent runs the Claude Agent SDK in the Electron main process and connects to geekspace-mcp, a standard 14-tool MCP server. Because it's standard MCP, Claude Code and Claude Desktop can drive the same workspace.

What went wrong building an app this fast? Four real bugs: an IPv6/IPv4 localhost mismatch that blocked the window, a macOS Automation permission race in the Mail widget, gemma4 wrapping JSON in markdown fences, and an early agent architecture that couldn't run tools and had to be rebuilt.

Do you need to be a developer to do this? No. I'm a systems engineer learning to build with AI, not a professional developer. The skill that mattered most was knowing the outcome I wanted and recognizing when the architecture was wrong — not writing the code.

Leave a comment

*Part of the Building Geekspace series — an honest look at what's possible when you partner with AI to build real software. Published at As The Geek Learns.*

Your DNS Changed and Nobody Told You. Here's the Nightly-Diff Pattern That Catches It.

James Cruce — Wed, 10 Jun 2026 11:01:41 GMT

It was a Tuesday at 2:17 PM, and the marketing team's contact form was returning 502s. Not 404. Not a timeout. A clean 502, which means something was answering, just not the thing it was supposed to be.

An hour in, I'd checked the app server logs, restarted the nginx process twice, confirmed the SSL cert was valid, and pinged our cloud provider's status page like it owed me money. Everything looked fine everywhere I looked. Then, almost by accident, I ran `dig +short www.company.com CNAME` and saw a hostname I didn't recognize. Something like `legacy-assets.decommissioned-vendor-name.com`.

Vendor had been off the account for four months. The CNAME had quietly repointed to their infrastructure during the migration wind-down, sat there untouched, and then their old infrastructure finally went dark. Nobody changed our DNS intentionally. Nobody got notified when it happened. We found out when a sales rep tried to submit a lead form.

That was the day I stopped trusting that "nothing changed in DNS" was a statement anyone could actually verify.

Why DNS Is the Silent-Failure Layer of Every Infrastructure

DNS is configuration. It's just not a configuration you can store in your repo, lint on a commit, or review in a pull request. It lives in a registrar panel or a DNS provider dashboard, updated by humans who may or may not be following a change-control process, and it's completely invisible until something breaks.

Every other layer of your stack has some kind of drift detection built in these days. Config management tools track the desired state of your servers. Container orchestrators know what's supposed to be running. Infrastructure-as-code tools will tell you if something drifted from the Terraform state. DNS gets none of that by default. You get a text field in a web UI, a change that takes effect whenever the TTL expires, and exactly zero notifications.

The operational pattern most teams rely on is "we'll know when it breaks." And they're right. They will know. They'll know at 2 PM on a Tuesday when a customer reports it, after a sales lead gets lost, after the support team has spent 45 minutes ruling out everything else. The detection mechanism is user reports, which is among the worst possible monitoring strategies.

There's also a subtler problem. The change usually isn't malicious. It's not a security incident, at least not at first. It's a vendor cleanup, a platform migration, someone at a partner org tidying up their infrastructure without realizing your CNAME still pointed at them. It's the kind of change that feels harmless to whoever made it and catastrophic to whoever depends on it.

The fix isn't complicated. What you need is a declared source of truth for what your DNS should look like, a way to compare that against what it actually looks like right now, and something that runs that comparison regularly enough to catch drift before users do.

That's the pattern. The implementation fits in a bash script.

The Pattern, the Four States, and a Wrapper You Can Use Today

The idea is straightforward: declare your expected DNS state once in a YAML file, then run a script nightly that queries your authoritative nameservers and compares what it finds against what you declared. Any gap between the two gets reported.

The baseline file is the key piece. It's not generated. You write it manually, and that act of writing it is itself useful, because it forces you to actually look up what each record currently is and decide "yes, that's correct." Once it exists, it becomes your source of truth. Commit it to your repo. Update it when you make a legitimate DNS change. The baseline is always what you intend, and the script is always asking whether reality matches.

When the diff runs, every record type for every domain you declared lands in one of four states:

MATCH means the live record matches the baseline exactly. This is the quiet result. Nothing to do.

NEW means a record exists in live DNS that isn't in your baseline. It could be a vendor auto-adding a TXT verification record. It could be someone provisioning a new subdomain. It could be something you should care about. The script surfaces it; you decide.

MISSING means your baseline declared a record that doesn't exist in live DNS anymore. An A record that was decommissioned without cleaning up. An MX record that got deleted. A CNAME that was removed when a vendor migrated their platform.

DRIFT means the baseline and live DNS both have records for a type, but the values don't match. This is the Tuesday-afternoon scenario: the CNAME target changed, the IP behind an A record flipped, the SPF policy was modified.

NEW and MISSING and DRIFT all mean something in your environment changed without you being told. The script exits nonzero when any of those occur, which makes it trivially composable with cron, alerting pipelines, or anything else that reads exit codes.

Here's a minimal working bash wrapper you can adapt right now. It keeps the dependencies to just `dig` and `bash`, uses a simple shell-array for your expected records instead of parsing YAML, and is short enough to read in under five minutes:

#!/usr/bin/env bash
# dns-check.sh
# WHAT: Minimal DNS drift checker - declare expected records, diff against live DNS
# WHY:  Catches silent DNS changes before they become incidents
# Usage: ./dns-check.sh
#        Add to cron: 0 2 * * * /path/to/dns-check.sh || echo "DNS DRIFT DETECTED" | mail -s "DNS Alert" you@example.com

set -euo pipefail

# ── CONFIGURATION ────────────────────────────────────────────────────────────
# Authoritative resolver to query against (use your domain's actual nameserver)
# WHY: Querying authoritative NS catches changes before they propagate to resolvers
RESOLVER="8.8.8.8"

# Declare expected records as: "domain|TYPE|expected_value"
# Get current values with: dig +short example.com A
# Run once to populate, then treat this as your source of truth
EXPECTED_RECORDS=(
  "example.com|A|93.184.216.34"
  "www.example.com|CNAME|example.com.cdn.cloudflare.net"
  "example.com|MX|10 mail.example.com"
  "example.com|TXT|v=spf1 include:_spf.google.com ~all"
)

# ── DIFF ENGINE ──────────────────────────────────────────────────────────────
DRIFT_FOUND=0

for record in "${EXPECTED_RECORDS[@]}"; do
  # Parse the declared record into its three parts
  domain="${record%%|*}"
  rest="${record#*|}"
  rtype="${rest%%|*}"
  expected="${rest#*|}"

  # Query live DNS at the authoritative resolver
  # WHY: +short gives us clean output; @resolver pins which nameserver answers
  actual=$(dig +short "@${RESOLVER}" "${domain}" "${rtype}" 2>/dev/null \
    | sort \
    | tr '\n' '|' \
    | sed 's/\.$//g; s/|$//')

  # Normalize expected for comparison (sort, strip trailing dots)
  expected_norm=$(printf '%s\n' "${expected}" \
    | sort \
    | tr '\n' '|' \
    | sed 's/\.$//g; s/|$//')
      # Compare and classify the result
  if [[ -z "${actual}" ]]; then
    # Record existed in baseline but dig returned nothing: MISSING
    printf "MISSING  %s %s  (expected: %s)\n" "${domain}" "${rtype}" "${expected}"
    DRIFT_FOUND=1
  elif [[ "${actual}" != "${expected_norm}" ]]; then
    # Record exists but value changed: DRIFT
    printf "DRIFT    %s %s\n  expected: %s\n  actual:   %s\n" \
      "${domain}" "${rtype}" "${expected}" "${actual}"
    DRIFT_FOUND=1
  else
    # Values match: MATCH (silent - no output unless you add --verbose logic)
    : # nothing to report
  fi
done

# Exit nonzero on any drift - composable with cron, alerting, CI checks
if [[ "${DRIFT_FOUND}" -eq 1 ]]; then
  printf "\nDrift detected. Review records above.\n" >&2
  exit 1
fi

printf "All %d declared records match live DNS.\n" "${#EXPECTED_RECORDS[@]}"
exit 0

Save that, drop your actual records into `EXPECTED_RECORDS`, and run it once to confirm it sees what you expect. Then add it to your crontab:

# Run nightly at 2 AM, email on drift
0 2 * * * /path/to/dns-check.sh || echo "DNS drift detected on $(hostname)" | mail -s "[ALERT] DNS Drift" you@example.com

The "NEW record exists in live DNS" state isn't in this minimal version, since detecting it requires knowing which record types to scan for beyond what you declared. The four-state model handles that fully once you know which types to watch, which is what the complete kit covers. For a first pass, catching MISSING and DRIFT gets you most of the value.

A few practical notes. Use `dig +short` rather than `dig` without `+short` or you'll spend time parsing the human-readable output format. Always query a specific nameserver with `@resolver` rather than relying on your local resolver, since caching can hide drift for hours. The MX record normalization is worth being careful about: `dig +short` returns the priority prefix as part of the value (`10 mail.example.com`), so your expected strings need to include it exactly that way. And commit the script alongside your baseline declaration. If the baseline lives in the repo, you get history, diffs, and code review for DNS changes as a side effect.

Share As The Geek Learns

What Else Lives in the Full Kit

The script above covers the core pattern. The full DNS Drift Detector kit is what you reach for once you've outgrown the wrapper.

The main `dns-drift-detector.sh` handles all five record types: A, AAAA, CNAME, MX, and TXT. That last group matters more than it seems. TXT records are where SPF policies live, where DKIM selectors sit, where domain verification tokens accumulate. Quiet SPF drift can break your email deliverability for days before anyone notices. DKIM drift means legitimate mail starts hitting spam folders. These aren't hypothetical edge cases.

The color-coded output makes the diff results readable at a glance during incident response, and the `--quiet` flag strips all of that for cron-friendly logging where you only want the exit code to speak. There's a `--no-color` flag too, so piping to a log file doesn't fill it with ANSI escape sequences.

`install-cron.sh` is a one-command idempotent installer. It checks that `dig` is available, puts the scripts where they belong, creates a log directory, and writes the cron entry with a duplicate-guard marker so running it twice doesn't add the job twice. That kind of thing is boring to write and annoying to get wrong.

The `baseline.yaml` in the kit is annotated with examples for ten common services: Google Workspace MX, Cloudflare CDN CNAMEs, SendGrid SPF, common DKIM selectors, and a few others. It's the reference you use when you're populating your own baseline for the first time.

The runbook covers install, baseline setup, how to read the output, what to do for each of the four states, how to update the baseline after a legitimate change, and an FAQ. That last one matters during an incident, when you don't want to be making judgment calls about whether a NEW record means "update the baseline" or "call the registrar."

Try the Pattern Today

The pattern described here is genuinely useful as-is. Declare your records, schedule the diff, react to the exits. That alone puts you ahead of the "we'll know when it breaks" approach that most infrastructure environments are actually running.

If you want the full kit, it's at shop.asthegeeklearns.com/products/dns-drift-detector for $19. You get the complete `dns-drift-detector.sh` with all five record types, color output, quiet mode, and cron logging; the idempotent `install-cron.sh`; the annotated `baseline.yaml` with ten real-world service examples; and the full operator runbook.

The Tuesday-afternoon incident I described at the top cost more than $19 worth of everyone's time. The detection script would have caught it the night before.

As The Geek Learns is a newsletter about systems engineering, automation, and the gap between knowing something and actually applying it. If this was useful, subscribe for free to get new articles as they land.

Leave a comment

I Secured My AI Agent With a 7-Layer Threat Model

James Cruce — Mon, 08 Jun 2026 17:00:31 GMT

I have an autonomous AI agent running on my Mac Studio. It has full shell access, reads my calendar, manages my tasks, and sends iMessages on my behalf. It runs 24/7 as a background service.

If that sentence doesn’t make you slightly nervous, you haven’t been paying attention. In February 2026, researchers found over 135,000 OpenClaw instances exposed to the public internet. A coordinated attack called ClawHavoc planted over a thousand malicious plugins in the community registry. Nine CVEs have been disclosed, including remote code execution.

I needed to take security seriously. Not “I changed the default password” seriously. Threat-model seriously.

MAESTRO: Seven Layers of Things That Can Go Wrong

The Cloud Security Alliance published a framework called MAESTRO—a 7-layer threat model specifically designed for agentic AI systems. Ken Huang mapped it directly to OpenClaw’s codebase, identifying 35+ specific threats across every layer of the stack.

Here are the seven layers, translated from security-paper language into “things that could actually ruin your day”:

Layer 1: Foundation Models: Someone sends your agent a crafted message that hijacks its behavior. Prompt injection. Jailbreaks. System prompt leakage. Your agent does what an attacker tells it to instead of what you told it to.

Layer 2: Data Operations: Your credentials are stored in plaintext JSON files. Your session logs contain every conversation forever. A malicious skill injects code through your workspace.

Layer 3: Agent Frameworks: The agent misuses its own tools. It runs shell commands it shouldn’t. It spawns sessions without authorization. It escalates its own privileges.

Layer 4: Deployment & Infrastructure: Your gateway is exposed to the network. Someone brute-forces the WebSocket token. A reverse proxy misconfiguration bypasses authentication entirely.

Layer 5: Evaluation & Observability: Nobody’s watching the agent for anomalous behavior. There’s no audit trail. Logs can be tampered with. If the agent starts acting weird, nothing catches it.

Layer 6: Security & Compliance: Your DM policy is misconfigured. Anyone can message the agent. Pairing codes can be brute-forced. Identity can be spoofed across channels.

Layer 7: Agent Ecosystem: A malicious plugin gets installed. A legitimate plugin’s npm dependency gets compromised. The skill registry serves poisoned packages.

The critical attack chain MAESTRO identifies: compromise the gateway (Layer 4) → access the session store (Layer 2) → poison conversation history (Layer 1) → control the agent (Layer 3) → spread via messaging (Layer 7).

Reading this was humbling. I’d addressed some of these by instinct during setup. Loopback binding, directory permissions, and pairing-based access control were all implemented. But “some” isn’t a security posture.

SecureClaw: The Audit

SecureClaw is an open-source security tool built specifically for OpenClaw by Adversa AI. It maps to MAESTRO, OWASP, MITRE ATLAS, and NIST AI 100-2. The install is a git clone and a bash script, no npm install, no network calls, and no surprises.

git clone https://github.com/adversa-ai/secureclaw.git
bash secureclaw/secureclaw/skill/scripts/install.sh

Then you run the audit:

bash ~/.openclaw/skills/secureclaw/scripts/quick-audit.sh

My baseline score: 57 out of 100. Zero criticals. Three HIGHs. Three MEDIUMs. Eight checks passing.

Here’s what passed without any work:

• Gateway bound to loopback (127.0.0.1) not exposed to network

• Gateway authentication present

• Directory permissions set to 700 (owner only)

• No browser relay exposed

• DM policy set to pairing (not open)

• Skills clean of malicious patterns

And here’s what failed:

🟠 HIGH Plaintext key exposure: Keys in openclaw.json and 5 backup files
🟠 HIGH Sandbox mode: commands run directly on host
🟠 HIGH Exec approval mode: agent acts without human approval
🟡 MED No cognitive file baselines: can’t detect tampering
🟡 MED Default control tokens: vulnerable to spoofing
🟡 MED No failure mode: no graceful degradation

The Hardening

Step 1: Clean up credential leaks. OpenClaw creates .bak files every time you change config. Each backup contains your full config, including Slack tokens and API keys. I had five of them sitting in the OpenClaw directory. Deleted them all. Set the main config to 600 permissions.

This is the kind of thing that’s easy to miss and catastrophic to ignore. A single ls -la ~/.openclaw/ would show them. But who runs ls -la on their config directory after every change?

Step 2: Create integrity baselines. SecureClaw’s hardener generates SHA256 hashes of your “cognitive files” IDENTITY.md, AGENTS.md, and HEARTBEAT.md. These are the files that define who your agent is and what it does. If an attacker or a hallucinating agent modifies them, the nightly integrity check will catch it.

bash ~/.openclaw/skills/secureclaw/scripts/quick-harden.sh

Step 3: Exec approvals. This is the big one. MAESTRO recommends human-in-the-loop approval for all shell commands. But my agent runs morning briefings and heartbeat checks on cron—unattended. Setting approvals to “always” would break all automation.

The solution: an allowlist with on-miss approval. I created ~/.openclaw/exec-approvals.json with 17 safe command patterns: imsg, calctl, apple-reminders, cairn, and basic file operations. Tars can run these freely. Anything else; curl, rm, pip install, or any command not on the list, requires human approval.

{
  “defaults”: {
    “security”: “allowlist”,
    “ask”: “on-miss”
  },
  “agents”: {
    “main”: {
      “allowlist”: [
        { “pattern”: “imsg *”, “note”: “iMessage send/read” },
        { “pattern”: “calctl *”, “note”: “Apple Calendar” },
        { “pattern”: “cairn *”, “note”: “Task management” }
      ]
    }
  }
}

This is the trade-off MAESTRO doesn’t talk about: security versus automation. Maximum security means every action needs approval. Maximum automation means the agent acts freely. The allowlist is the middle ground. Routine operations are pre-approved, and novel or dangerous operations require a human.

Step 4: Full plugin install. Beyond the bash scripts, SecureClaw has a full npm plugin with 56 runtime audit checks, background monitors for config drift, and real-time integrity verification. Installing it required building from source (TypeScript → JavaScript) and registering it with OpenClaw’s plugin system.

openclaw plugins install -l /path/to/secureclaw

openclaw config set plugins.allow ‘[”secureclaw”]’

That plugins.allow line is important. By default, OpenClaw will auto-load any discovered plugin. Explicit trust means only plugins you’ve approved get loaded.

Step 5: Nightly audit cron. A macOS LaunchAgent runs the full audit suite every night at 2 AM which includes quick-audit, integrity check, and supply chain scan. Results go to secureclaw-audit.log. If something changes overnight, it shows up in the morning.

The Final Score

After hardening: 64 out of 100. Nine checks passing. Zero criticals. The three remaining HIGHs are documented, accepted trade-offs:

Findings I accepted (with reasoning)—Sandbox mode (Docker sandboxing would break imsg, calctl, and Apple Reminders); Plaintext keys in config (inherent to the platform config format, file is locked to 600); Exec approval not “always” (using allowlist + on-miss; full “always” breaks unattended cron automation).

The two MEDIUMs, control token customization and failure mode configuration, aren’t supported in OpenClaw v2026.3.2’s config schema yet. SecureClaw checks for them proactively. They’ll be fixable when OpenClaw adds the config options.

What I Actually Learned

Security isn’t a feature you enable. It’s a series of trade-offs you make with your eyes open. Sandbox mode is “more secure” but breaks the tools that make the agent useful. Approval mode “always” is “more secure” but kills the automation that makes the agent worthwhile. The right security posture isn’t maximum restriction; it’s documented, intentional decisions about what risks you accept and why.

Automated scanning is essential but insufficient. SecureClaw’s audit caught things I would have missed, including the .bak files with credentials, the missing integrity baselines, and the open exec policy. But the HIGHs it flagged as failures are things I’ve consciously accepted. No scanner can evaluate your specific trade-offs.

The biggest threat isn’t external. In my setup (loopback-bound, pairing-gated, allowlist-filtered), the most likely security failure isn’t a network attacker. It’s a malicious skill, a compromised npm package, or the agent itself hallucinating destructive actions. Layer 7 (ecosystem) and Layer 1 (model behavior) are the real attack surfaces for a local-first setup. The exec approval allowlist is my primary defense for both.

Clean up after yourself. OpenClaw creates backup files containing credentials on every config change. There’s no auto-cleanup. If you’re running OpenClaw, go check your directory right now: ls ~/.openclaw/*.bak*. You might be surprised.

Quick Reference

Hardening actions and commands: install, run audit, apply hardening, check integrity, scan skills, check for credential leaks, set exec approvals, set plugin trust. Commands target ~/.openclaw/skills/secureclaw/scripts/. Full command details in the image.

Update—June 2026: What I Actually Did When I Moved to ClaudeClaw

I wrote this piece in March, when OpenClaw was still the thing running my Mac Studio. By the end of April, I’d shut it down. Disabled the cron jobs, quarantined the LaunchAgents, and rebuilt the whole stack on the Claude Agent SDK. Based off of ClaudeClaw from the Early AI-Dopters AI learning group. The full post-mortem on why:

Why? The short version is this: I couldn’t see into OpenClaw. Which, if you scroll back up, is Layer 5: Evaluation & Observability, the exact layer this audit was weakest on.

You may wonder whether I just copied the 7-layer hardening over to the new stack. I didn’t, and I want to be honest about that. I did not port MAESTRO one-for-one. SecureClaw was written specifically for OpenClaw. Some of its thinking transferred; some of it didn’t. And the threat model itself moved on (more on that at the end). What the seven layers became was a checklist: for each one, how does the new architecture answer this? Here’s the scorecard.

The two layers that changed the most.

Layer 5 (Observability) went from my single biggest weakness to the entire reason ClaudeClaw exists. There’s now a dedicated agent, WATCHMAN, running seven probes every hour: failed tasks, stuck tasks, missed scheduler slots, daemon liveness, content-pipeline health, hidden failures (it greps the success logs for crash text), and delegation crashes. More importantly, there’s a second healthcheck running as a separate LaunchAgent with its own keychain-backed alert token. If the main daemon dies, the thing that tells me about it is still alive. The rule I wrote for myself out of this: the watcher cannot share fate with the watched. There’s also a behavioral dashboard, DefenseClaw, sitting on 127.0.0.1:3141.

Layer 3 (Agent Frameworks) is where my OpenClaw work actually carried forward. The exec-approvals allowlist from Step 3 above is the direct ancestor of what ClaudeClaw does now, except the enforcement dropped down a level. The first thing I shipped was killing bypassPermissions (the main agent had been running with permission checks disabled, which means a compromised agent has unlimited tool access. The SDK was no ceiling at all), switching to the SDK’s default permission mode, and handing the main agent a 15-tool allowlist as the single source of truth. Same idea as the OpenClaw allowlist. Enforced by the SDK itself instead of a config file I had to maintain.

The rest mapped like this:

How each of the seven MAESTRO layers from the OpenClaw audit is answered in ClaudeClaw.

Layer 1 Foundation Models: channel tagging and a trust gradient that treats retrieved text as data, not directives (evolved).

Layer 2 Data Operations: Chamberlain outbound scanner, exfiltration-guard, queryable Memory v2, and ingest-time canonicalization (replaced and extended).

Layer 3 Agent Frameworks: SDK permission ceiling and a 15-tool allowlist, the direct successor to the OpenClaw exec-approvals list (kept, moved into the SDK).

Layer 4 Deployment and Infrastructure: an egress gateway plus kernel-level pf default-deny (replaced).

Layer 5 Evaluation and Observability: WATCHMAN’s seven probes and a fate-isolated external healthcheck, the biggest upgrade.

Layer 6 Security and Compliance: out-of-band Telegram confirmation for state-changing actions and a role policy kept separate from content memory (evolved).

Layer 7 Agent Ecosystem: an MCP allowlist plus the tool ceiling as a second layer (kept and hardened).

Plus a new row beyond MAESTRO. Memory persistence: TTLs, a hash-chained write log, and canaries.

Where the 7-layer model ran out.

MAESTRO is a static threat model. It’s a map of what can go wrong at each layer, frozen in time. What it doesn’t have a layer for is persistence. An attack that lands quietly in your agent’s memory or vector store and just waits. My scheduler re-enters context every 60 seconds, which means anything dormant in memory fires on a clock. That’s a different class of problem, and it has a name now: LPCI, Logic-layer Prompt-based Conditional Injection. Hardening against it (I am planning a separate two-part write-up on As The Geek Learns) meant building things MAESTRO never asked for, including a canonicalizer that decodes payloads before they reach the vector store, channel-tagged prompts so the model knows retrieved text is data and not instructions, memory TTLs, a hash-chained write log, and canary entries that page me if memory ever leaks into output.

What I gave up and what I kept. The honest cost of the move: I lost local-first. OpenClaw ran on Ollama, fully offline; ClaudeClaw talks to Anthropic’s API. I still own every byte of my data; it’s all on my SSD; I just don’t own the weights anymore. What carried over intact was the philosophy this whole series is built on: every document is a file I can grep, every config is version-controlled, and every decision has a session note. That part never changed.

This is Part 5 of the Notion Replacement series. We went from “install an AI agent” to “secure it against a 7-layer threat model” in two days. Follow along at As The Geek Learns.

I Secured My AI Agent With a 7-Layer Threat Model

James Cruce — Mon, 08 Jun 2026 16:31:12 GMT

I have an autonomous AI agent running on my Mac Studio. It has full shell access, reads my calendar, manages my tasks, and sends iMessages on my behalf. It runs 24/7 as a background service.

I needed to take security seriously. Not “I changed the default password” seriously. Threat-model seriously.

MAESTRO: Seven Layers of Things That Can Go Wrong

Here are the seven layers, translated from security-paper language into “things that could actually ruin your day”:

Layer 2: Data Operations: Your credentials are stored in plaintext JSON files. Your session logs contain every conversation forever. A malicious skill injects code through your workspace.

Layer 3: Agent Frameworks: The agent misuses its own tools. It runs shell commands it shouldn’t. It spawns sessions without authorization. It escalates its own privileges.

Layer 4: Deployment & Infrastructure: Your gateway is exposed to the network. Someone brute-forces the WebSocket token. A reverse proxy misconfiguration bypasses authentication entirely.

Layer 6: Security & Compliance: Your DM policy is misconfigured. Anyone can message the agent. Pairing codes can be brute-forced. Identity can be spoofed across channels.

Layer 7: Agent Ecosystem: A malicious plugin gets installed. A legitimate plugin’s npm dependency gets compromised. The skill registry serves poisoned packages.

SecureClaw: The Audit

git clone https://github.com/adversa-ai/secureclaw.git
bash secureclaw/secureclaw/skill/scripts/install.sh

Then you run the audit:

bash ~/.openclaw/skills/secureclaw/scripts/quick-audit.sh

My baseline score: 57 out of 100. Zero criticals. Three HIGHs. Three MEDIUMs. Eight checks passing.

Here’s what passed without any work:

• Gateway bound to loopback (127.0.0.1) not exposed to network

• Gateway authentication present

• Directory permissions set to 700 (owner only)

• No browser relay exposed

• DM policy set to pairing (not open)

• Skills clean of malicious patterns

And here’s what failed:

🟠 HIGH Plaintext key exposure: Keys in openclaw.json and 5 backup files
🟠 HIGH Sandbox mode: commands run directly on host
🟠 HIGH Exec approval mode: agent acts without human approval
🟡 MED No cognitive file baselines: can’t detect tampering
🟡 MED Default control tokens: vulnerable to spoofing
🟡 MED No failure mode: no graceful degradation

The Hardening

This is the kind of thing that’s easy to miss and catastrophic to ignore. A single ls -la ~/.openclaw/ would show them. But who runs ls -la on their config directory after every change?

bash ~/.openclaw/skills/secureclaw/scripts/quick-harden.sh

{
  "defaults": {
    "security": "allowlist",
    "ask": "on-miss"
  },
  "agents": {
    "main": {
      "allowlist": [
        { "pattern": "imsg *", "note": "iMessage send/read" },
        { "pattern": "calctl *", "note": "Apple Calendar" },
        { "pattern": "cairn *", "note": "Task management" }
      ]
    }
  }
}

openclaw plugins install -l /path/to/secureclaw

openclaw config set plugins.allow ‘[”secureclaw”]’