Apple's container CLI hit 1.0 this month, picked up 30,000 GitHub stars, and arrived with the usual promise: faster than Docker on Apple Silicon, lighter, and more native. It runs Linux containers in lightweight VMs, written in Swift, optimized for your M-series chip.

I wanted to know if the promise holds. So I didn't read the marketing. I benchmarked it.

And I added a third contender Apple didn't mention: OrbStack. Because if you've already gotten tired of Docker Desktop on a Mac, OrbStack is probably what you switched to. Leaving it out of a "fast containers on Apple Silicon" comparison would've been lacking since it's the one a lot of us actually run.

Here's what the numbers say and how to decide which one belongs on your machine.

How I tested this

Quick note on method: because a benchmark you can't trust is just a screenshot of one lucky run.

Everything ran on a Mac Studio: M3 Ultra, 256 GB RAM, macOS 26.5.1. All three tools pulled the same images, pinned by digest, so nobody got a different build. Every measurement is the median of at least 10 runs, warmup discarded, each run in a clean subprocess so state from one couldn't bleed into the next.

Nine scenarios: container startup, a web server coming up, Postgres accepting connections, CPU work, disk I/O on a bind mount, container-to-container networking, an image build, a multi-service stack, and a density ramp. Plus idle footprint.

The whole harness is public if you want to reproduce it or pick it apart: github.com/Jmeg8r/apple-container-vs-docker. Every claim below traces to a number in that repo or a failure I logged on purpose.

One thing that matters before we start: all of this is on macOS 26 (Tahoe). On macOS 15, Apple container can't even do container-to-container networking. Containers get IPs but can't talk to each other. If you're not on Tahoe yet, half of this comparison doesn't apply to you, and the answer is "stay on Docker or OrbStack." For everyone else, read on.

The one number where Apple wins outright

Let's lead with Apple's real victory, because it's a good one.

Idle footprint. With nothing running—no containers, just the runtime sitting there ready. Apple container holds about 51 MB of memory. Docker Desktop sits at 1,124 MB. OrbStack at 1,631 MB.

That's not a typo. Apple container uses 22 to 32 times less memory at rest than the other two.

The reason is architectural. Docker Desktop and OrbStack each run one big Linux VM that stays resident the whole time you're logged in, whether you're using it or not. Apple container doesn't keep a VM warm. It spins one up per container and tears it down when the container stops. Nothing running means nothing resident.

If you keep a couple of long-lived services on your laptop and want them out of the way the rest of the day, this is a genuinely meaningful win. Your RAM is yours again when you're not using containers.

Hold onto that "spins up a VM per container" detail, though. It's also the source of every place Apple loses.

Where it ties and quietly wins again

Before the losses, give Apple its due on two more.

Raw CPU is a dead heat. Running sysbench inside a container, Apple container actually came out slightly ahead, about 38,500 events per second versus 36,400 for Docker Desktop and 33,500 for OrbStack. The per-VM model adds no real tax on compute. Once your code is running, it runs at full speed.

Cached builds are Apple's fastest. Rebuild an image when the layers are already cached, and Apple finishes in about 426 ms. Quicker than Docker Desktop (598 ms) and noticeably quicker than OrbStack (844 ms).

So this isn't a story about a slow tool. Where the VM boundary isn't sitting in the hot path, Apple container is right there with the best of them and sometimes ahead.

The per-VM tax: everywhere it creates or moves data

Now the other side. And it's consistent enough that you can predict it: anything that involves spinning up a container or pushing data across the VM boundary costs Apple time.

Startup. A bare container's start-to-exit takes Apple about 1,071 ms versus ~360–395 ms for the others.

Nginx answering its first request: 1,095 ms versus ~290 ms. Postgres accepting a connection: 2,081 ms versus 292 ms on Docker Desktop. That's 3× to 7× slower, depending on the workload. Every time you start something, you're paying for a VM to boot.

Bind-mount disk I/O—this is the one that'll bite you. Mount a folder from your Mac into the container and write to it, and Apple manages about 32.6 MB/s. Docker Desktop does 96.7. OrbStack does 548.5 MB/s which is roughly 17 times faster than Apple.

Sit with that one, because it's not an abstract benchmark. Mounting your source tree into a container and editing it live. The hot-reload dev loop, the thing a huge number of Mac developers do all day, runs straight through this path. On Apple container, that loop is going to feel like wading through mud.

Networking between containers follows the same shape: ~11 Gbps for Apple, ~53 for Docker Desktop, and ~85 for OrbStack. Real network hops between separate VMs versus traffic staying inside one shared VM.

And density. Spinning up 40 containers took Apple about 35 seconds. Docker Desktop and OrbStack did it in roughly 9. Forty VMs versus forty processes in one VM—the model shows up again. (To be fair, none of the three actually fell over at 40; I capped it there to be kind to my machine. Finding Apple's true ceiling is a job for another day.)

None of this is a bug. It's the direct, honest consequence of one lightweight VM per container. You're trading creation speed and I/O for isolation and a clean idle state. Whether that's a good trade depends entirely on what you do all day.

The gaps that have nothing to do with speed

Numbers aside, a few things will just stop you cold, and these matter more for daily use than any millisecond.

No Docker Compose. This is the big one. If your project starts with docker compose up, Apple container has no native answer. You're back to creating a network and running each service by hand, or leaning on an unofficial third-party bridge. Docker Desktop and OrbStack both speak Compose fluently.

Image names have to be fully qualified. container run alpine fails. You need container run docker.io/library/alpine. Small thing, but it'll trip every script and muscle-memory habit you have.

Port publishing isn't the model you know. I expected -p 8080:80 to put the service on localhost:8080 like Docker does. On Apple container it didn’t—the container was serving fine, but on its own IP address, not on localhost. Apple's model is "every container gets a real routable IP," which is elegant, but it's not the muscle memory you've built. (Related gotcha: poll 127.0.0.1, not localhost — localhost can resolve to IPv6 first and miss the service entirely.)

DevContainers support is incomplete. If you live in VS Code's dev containers, you're not there yet.

If you want to try it yourself

It's a five-minute install. The one non-obvious step is the kernel.

# Apple container
brew install container
container system kernel set --recommended   # do this first, or `start` prompts you
container system start

# OrbStack, if you want to compare
brew install --cask orbstack

Then remember the gotchas above: fully-qualified image names, reach services by their container IP, and 127.0.0.1 over localhost. That's most of the friction right there.

So which one do you actually use?

Here's how I'd decide.

Reach for Apple `container` when you run a handful of long-lived containers and you care about isolation and a clean idle footprint—a couple of always-on services on a laptop you also use for everything else. The per-container VM boundary is real security isolation, and getting your RAM back when you're idle is a nice perk. Just don't point your live-editing dev loop at it.

Reach for OrbStack if you want fast Docker without the friction. It won every speed test that involved I/O or networking, starts containers as fast as anything, and crucially, it still speaks Compose and the full docker CLI. The only price is the biggest idle footprint of the three and the slowest cached build. For most Mac developers, this is the easy pick.

Stay on Docker Desktop when you need the whole ecosystem—Compose, DevContainers, and the broadest tooling and documentation. It sat in the middle of nearly every benchmark, and "middle of the pack with everything supported" is exactly what a default should be.

The headline I came in expecting — "Apple container is faster than Docker" — just isn't what the data shows. It's faster at a few specific things and slower at most of the rest. But that's not a knock. It's a specialist. It does one shape of work really well and asks you to give up the conveniences you've built your workflow around.

Know which shape of work you're doing, and the choice makes itself.

Share

The full harness, raw numbers, and every chart are public: github.com/Jmeg8r/apple-container-vs-docker Run it on your own machine. I'd genuinely like to see whether the bind-mount gap holds on an M1 or M2, or whether 256 GB of headroom is flattering somebody.

Share As The Geek Learns

Leave a comment

"Isn't that expensive?" Every time I talk about using Claude Code for a project, someone asks this. The honest answer is: it depends entirely on what you route to Claude versus what you run locally. On a Mac Studio with unified memory, the economics change fast. Here's the routing table I actually use.

The Setup

I have an M3 Ultra with 256 GB unified memory. That machine runs a 70B model locally with room to spare and a 235B mixture-of-experts model when I need frontier-scale reasoning. Running it at load draws around 60 watts. Eight hours of overnight inference work costs about five cents in electricity at typical US rates. The same run on a Frontier API would cost somewhere between fifteen and thirty dollars.

But cost isn't the primary reason to run locally. Latency is. My workhorse model, gemma4:31b-mlx, stays pinned in GPU memory, so there's no cold-start delay. It begins responding the moment you hit enter on M-series hardware. That's interactive. You can iterate on code at conversational speed with a local model, then bring Claude in for the decisions that actually require judgment.

Most developers running Claude Code for everything are paying for two things: generation (which local models handle well) and judgment (which frontier models handle better). Splitting those tasks cuts the API spend dramatically while keeping the quality where it matters.

What's Actually Going On

The routing principle has one line: "Claude reads the playbook. Local models do the work."

Local models handle generation: the fast, cheap, iterative part. New code, first-draft text, refactoring suggestions, variant generation. Claude Code handles orchestration and the Compound step: planning, judging what goes in learnings.jsonl, reviewing the session output, and updating CLAUDE.md.

The model routing table I use:

The everyday models are light. gemma4:31b-mlx loads in about 20 GB on disk and sits pinned in GPU memory (around 46 GB resident at its full 256K context). qwen3-coder:30b, a mixture-of-experts model that activates only ~3B parameters per token, loads in another 18 GB. On an M3 Ultra with 256 GB, both stay resident at once with enormous headroom enough to also pull a 120B or even a 235B reasoning model on demand. That last one is the whole point of the memory: a 235B model at Q4 needs around 142 GB and simply won't load on a smaller machine.

deepseek-r1:70b needs 64 GB or more. It's not interactive, responding in 20-30 seconds per generation. That's fine for overnight batch jobs. It's not fine for a code review you're waiting on.

The Fix

Setting up Ollama on a Mac Studio takes about 10 minutes:

brew install ollama
ollama pull gemma4:31b-mlx     # primary workhorse (MLX build, Apple-Silicon optimized)
ollama pull qwen3-coder:30b    # code generation + review (MoE, strong tool calling)
# Heavy reasoning — only comfortable with lots of unified memory:
# ollama pull deepseek-r1:70b                          # ~42 GB, overnight reasoning
# ollama pull gpt-oss:120b                             # ~65 GB
# ollama pull qwen3:235b-a22b-thinking-2507-q4_K_M     # ~142 GB, needs 256 GB

Open the Ollama menu bar app once to enable auto-start on login. It runs as a local HTTP server at localhost:11434.

The session structure for a typical day:

Morning: Planning (Claude Code)
  - Read CLAUDE.md + learnings.jsonl
  - Brainstorm, write plan with verify steps

Work (local models via Ollama)
  - gemma4:31b-mlx for generation and first drafts
  - qwen3-coder:30b for code and "catch what I missed"

Evening: Compound Step (Claude Code)
  - Review session, propose learnings
  - Update CLAUDE.md Known Patterns table
  - Check tests, commit

Overnight (optional, if you have enough RAM for a 70B model):
  - deepseek-r1:70b running autoresearch loop
    on a skill file or prompt variant

The Compound step is where the two frameworks connect. Karpathy's Autoresearch pattern (one file, one metric, keep/revert with git) maps directly onto a local overnight job. Instead of optimizing a training script, you optimize a skill file or a prompt file from your .claude/commands/ directory.

Define a set of test cases: files with known issues your skill should catch. Run variants overnight. Keep the variants that find more issues; revert the rest. In the morning, run the Compound step with Claude on the winning variants.

The autoresearch loop becomes the Work step in a Compound Engineering session.

Why This Matters

I've had a real-world data point on this from the stoicism-agent project, where I ran the first overnight autoresearch loop using local models. The learning I wrote after that run:

"The fork between MLX and Ollama for autoresearch comes down to one question: do you need to modify model weights? MLX if yes. Ollama for prompt-level optimization. For skill file autoresearch, Ollama is the right choice."

That's the kind of learning you only get by running the thing. The instinct before running it was to reach for MLX because it's the "native" Mac AI framework. The result after running it: Ollama is simpler, has better model selection, and is more than fast enough for overnight prompt optimization where you're not touching weights.

Same principle scales to any overnight optimization loop. Pick the tool that matches the task. Don't reach for fine-tuning when prompt-level optimization will do.

The cost math on the Mac Studio, after a few months of this workflow: roughly 90% of generation work routes to local models. The 10% that goes to Claude is the judgment layer: planning, compound steps, final review. Total Claude API spend for a typical weekend session runs $2-3. For a full week of this, maybe $8-10.

The Mac Studio has nearly paid for itself in API savings after about 7 months. More importantly, the learnings file from six months of consistent Compound Engineering is now the most valuable artifact in my projects. Not the code. The 200+ preserved decisions.

Code can be rewritten. Those decisions can't.

Share

Quick Reference

• Install: brew install ollama, open Ollama.app for auto-start

• Primary workhorse: gemma4:31b-mlx (~20 GB, pinned in GPU, MLX-optimized)

• Code generation + review: qwen3-coder:30b (~18 GB, MoE)

• Heavy reasoning: gpt-oss:120b (~65 GB) or qwen3:235b-a22b-thinking (~142 GB, needs 256 GB)

• Overnight reasoning: deepseek-r1:70b (needs 64 GB+, ~20-30s/response)

• Never route to local: Compound step, final review, architectural decisions

• Autoresearch overnight: use deepseek-r1:70b for skill/prompt optimization, not weight training

• Also resident: nomic-embed-text for embeddings, a custom voice model for narration

• Smaller machines: a 76 GB M2 Ultra runs gemma4:31b-mlx + qwen3-coder:30b fine, but the 120B–235B tier needs the 256 GB

• Full routing table: docs/local-llm-routing.md in the template repo

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns (https://astgl.substack.com)

Share As The Geek Learns

Leave a comment

There's a number floating around that's hard to ignore.

Google's new DiffusionGemma is supposed to crank out 1,000-plus tokens per second. For comparison, the local models most of us run putter along at 30 to 100. So a 10x jump? That gets my attention.

The reason it's fast is genuinely interesting. Every model you've used—ChatGPT, Claude, your local Llama—writes one token at a time, left to right, each word waiting on the one before it. That's "autoregressive." Diffusion models work completely differently. They start with a blank canvas of 256 tokens and refine the whole block at once, in parallel, like a photo developing. No waiting in line.

On paper, that's the future. So I did the obvious thing: I ran it on my Mac Studio to see if the future had arrived on my desk.

It hadn't. And the way it hadn't turned out to be more interesting than a win would've been.

The fair fight

Here's the thing that makes this a clean test instead of a vibe check.

DiffusionGemma is built on the same bones as Gemma 4—Google ships an autoregressive Gemma 4 26B A4B that's the same size, same architecture, same weights. The only difference is how it generates: diffusion vs. one token at a time.

So I put them head to head. Same Mac. Same 8-bit quantization. Same runner (Apple's MLX). Same prompts, same everything. The one variable left standing is the decoding paradigm itself. If diffusion is faster, this proves it. If it's not, there's nowhere to hide.

Thirty prompts across code, math, instruction-following, and writing. Five runs each. Let's look.

The result nobody puts in the headline

DiffusionGemma did about 43 tokens per second on my Mac.

The boring old autoregressive model did 61.

The diffusion model, the one that does 1,000+ on a datacenter GPU, was the slower of the two on Apple Silicon. Not by a hair. By 40%.

That gap between the headline and my desk is about 23x. The 1,000 tok/s is real; it's just real on an H100, a $30,000 datacenter card. On a Mac, that number has nothing to do with your life.

And it gets worse for diffusion if you care about how snappy a chat feels. There's a metric called time-to-first-token, how long you stare at a blank screen before words start appearing. The autoregressive model started typing in 0.12 seconds. DiffusionGemma took 1.86.

That one surprised me until I thought about it. Remember how diffusion refines a whole 256-token block at once? That's the catch; it can't show you anything until the entire block is done cooking. The "parallel" model that's supposed to feel instant actually feels laggier, because it makes you wait for the batch.

Why the magic doesn't travel

So why does the same model fly on an H100 and crawl on a Mac?

It comes down to what each machine is good at. Diffusion's whole speed trick is doing a giant pile of math all at once, refining 256 tokens in parallel. An H100 has thousands of cores sitting there begging for exactly that kind of bulk work. Flood it, and it's happy.

Apple Silicon doesn't win that way. It's not short on memory. My Mac Studio has 256GB, but it's limited by how fast it can move data around, not how much math it can do at once. The fancy parallel block doesn't help when the bottleneck is the plumbing, not the engine.

Autoregressive decoding, meanwhile, plays to the Mac's strengths. It reuses its previous work (a "KV cache") and touches way less memory per token. Same model, same weights; the architecture that wins in the datacenter loses on the desktop. The hardware decides.

The benchmark that kept slowing itself down

I almost shipped wrong numbers. Here's the part the polished write-ups leave out.

My first full run looked fine for the first 15 or so generations. Then DiffusionGemma started... degrading. Not crashing—slowing. Time-to-first-token climbed from 1 second to 2, then 4, then 18, then 60, and by the 28th generation, a single response took over two minutes. Same prompt that was instant a minute earlier.

My first guess was a memory leak. So I checked. And this is the maddening part: every memory counter the framework reports stayed flat. By the numbers, nothing was wrong. I added the standard "clear the cache between runs" call. No change. I added a "wait for the GPU to finish" call. It nudged the cliff from generation 18 to generation 19 and then fell off it anyway.

The culprit turned out to be the graphics driver itself quietly piling up state that none of the normal tools could see or clear. The only thing that actually worked was brute force: run a handful of generations, then kill the whole process and start fresh. Let the operating system clean up what the framework couldn't.

Two lessons I'm keeping:

Trust the measurement over the marketing and over your own assumptions. If I'd run 15 prompts and called it a day, I'd have published a number that looked great and was completely fake.

Your tools can lie by omission. "Memory usage is flat" is not the same as "nothing is accumulating." The dashboard being green doesn't mean the system is healthy.

Quality: closer, but autoregressive still edges it

Speed isn't everything, so I scored the actual answers too. Code got run and tested. Math got checked against the right answer. Instruction-following got graded against rules. Writing got judged blind.

Overall, autoregressive Gemma came out ahead—0.90 to 0.84—winning code, instructions, and writing, while DiffusionGemma edged it on math. Honestly, that tracks with Google's own advice, which quietly tells you to use the standard model "for maximum quality." It's a real gap, but it's not a blowout.

So the diffusion model on my Mac was slower, laggier, and a notch lower quality. That's not a trade-off. That's just losing.

So should you care?

If you run models locally on a Mac, here's the takeaway: don't reach for DiffusionGemma expecting the headline. You'll get a third of the speed of the autoregressive version, worse responsiveness, and slightly weaker answers. For Mac local inference, boring old next-token generation still wins.

That's not a knock on diffusion models. The approach is genuinely promising, and on the right hardware, it's a rocket. But "the right hardware" is an NVIDIA datacenter card right now, not the machine on your desk.

The bigger lesson is the one I keep relearning: a vendor benchmark is true and useless until you run it on your own hardware. 1,000 tokens per second was a real number that told me nothing about my Mac. The only way to know what a tool does for you is to point it at your setup and watch.

I built the whole benchmark as a reusable harness; same-architecture comparison, real scoring, the works, so when MLX gets a diffusion-optimized path (or llama.cpp's Metal support lands), I can re-run it in an afternoon and see if the story's changed. I suspect it will, eventually. Just not today.

The whole thing—harness, scorers, charts, and the raw results—is on GitHub if you want to poke at it or run it on your own machine: github.com/Jmeg8r/diffusiongemma-benchmark.

Frequently Asked Questions

Does DiffusionGemma actually hit 1,000 tok/s on a Mac?

No. While it hits those speeds on an NVIDIA H100, I measured only 43 tok/s on my Mac Studio. The “parallel” advantage requires datacenter-grade compute to overcome memory bandwidth bottlenecks.

Why is the time-to-first-token (TTFT) higher for diffusion models?

Diffusion models refine a whole block of tokens (e.g., 256) at once. Because they cannot stream results token-by-token, you must wait for the entire batch to finish before any text appears on screen.

Can I fix the performance degradation in MLX diffusion runs?

The slowdown is caused by graphics driver state accumulation that standard memory tools don’t report. The only reliable fix currently is to run a few generations and then restart the process entirely.

Which Gemma model is better for coding on Apple Silicon?

Autoregressive Gemma 4 is superior. In my benchmarks, it scored higher in quality (0.90 vs 0.84) and was significantly faster (61 tok/s vs 43 tok/s) on Mac hardware.

Is 256GB of unified memory enough to make DiffusionGemma fast?

Memory capacity isn’t the bottleneck; memory bandwidth is. Even with 256GB, the Mac cannot move data fast enough to feed the parallel math required for diffusion speeds.

Running local models on Apple Silicon and want to run this yourself? The full harness is on the GitHub repo diffusiongemma-benchmark. And if you want more of these -"I actually tried it so you don't have to" - breakdowns, subscribe — that's most of what I do here.

Share As The Geek Learns

Leave a comment

I gave Claude Fable 5 one prompt and walked away for a few minutes. When I came back, it had designed a database schema. By the time I went to bed, I had a fully functional macOS desktop app with a block editor, relational databases, and a calendar that schedules itself. This is the story of how that happened and what it says about where AI-assisted development is right now.

The Prompt That Started Everything

It started simple. I'd been using Notion for years, but it always felt like it was one subscription price increase away from becoming someone else's problem. I wanted something local. Something mine. And I'd been watching what Claude Fable 5 could do on the Max plan, so I decided to push it.

The prompt was direct:

"I want you to build a macOS desktop app. This should be an app that lets you create custom pages with tables, text, images and more exactly like Notion. Use ASTGL branding. Use Convex for the database. Please build the full app, make it incredible and a professionally designed product, and make sure everything works."

That was it. No architecture spec. No wireframes. No feature list beyond "exactly like Notion."

What came back wasn't just code; it was a plan. A full architecture decision: Electron + React 19 + Vite + Tailwind v4 + BlockNote for the editor + Convex running in anonymous local mode (no account, no cloud, data stays on this Mac). Fable 5 made the call that Convex's anonymous local deployment mode was the right choice before I even thought to ask about it. No auth, no monthly fee, reactive by default, data in ~/.convex. That's a good decision.

What Got Built

Geekspace shipped with everything I use Notion for daily.

The block editor works exactly how you'd expect: type / for the command menu, hit # for headings, [] for to-dos. Drag handles, nested blocks, image uploads to Convex storage. The BlockNote library handles the ProseMirror plumbing; Fable 5 wired it into the Convex backend cleanly.

The databases are where it gets interesting. Property types, multiple views (Table, Board, List, Calendar, Timeline), per-view filters, and sorts. Relations between databases and not just visual ones. Two-way synced relation pairs, rollup properties, and status fields with groups. The seeded template drops in a Projects ↔ Tasks ↔ Sprints structure pre-wired with relations and progress rollups. That took one command: npm run seed.

The calendar that schedules itself is the headline feature. Every task with an estimate, a due date, and a priority gets automatically placed into your working hours, packed around fixed events, using earliest-deadline-first ordering. Drag a block, and it locks the engine schedules around it. Past blocks freeze as history. If something can't fit, it surfaces in a "needs attention" panel. Nothing silently drops.

And then I kept asking for more.

The Features I Added

After the first version was running, I started pushing.

"How can I connect my macOS Calendar and email?" Done. Calendar.app events mirror into Geekspace via JXA scripting, show up as fixed busy time the auto-scheduler works around, and display as dotted-edge events on the calendar. The Mail inbox widget pulls your recent messages, shows unread counts, and lets you turn any email into a task with one click.

"Notion has AI Meeting Notes—can you recreate that locally?" Done. One-click recording with a floating recorder, live level meter, pause, and resume. The audio runs through whisper.cpp for transcription, then through a local Ollama model for summarization. It never leaves the Mac. Summaries adapt to meeting type, standup vs. client vs. interview generate different formats. Action items become tasks in one click.

Then I approved a full roadmap for four more features: enterprise search over my ASTGL knowledge base, an AI agent built into the app, project templates, and a docs library. All four shipped in the same session.

The agent piece called ARCHITECT is the one I'm most proud of. I can use both Claude Agent SDK and local llm qwen3-coder:30b using a toggle. Both are wired and running directly inside the Electron main process. The qwen3-coder:30b model is great for tools usage. It connects to a custom MCP server I built called geekspace-mcp that exposes 14 tools covering every meaningful operation in the workspace. Ask ARCHITECT to set up a database for tracking podcast guests and it appears, live, in the sidebar. Ask it what's overdue and it queries the scheduler and tells you. Any MCP client can use this server, which means I can also drive Geekspace from Claude Code itself.

The Bugs That Made It Real

No build story is honest without the debugging.

The Electron window wouldn't launch. Vite was binding to IPv6 (::1), the startup script was polling 127.0.0.1 (IPv4), and they never found each other. One config change: host: "127.0.0.1" in vite.config.ts.

The macOS Mail widget timed out. The Automation permission dialog was blocking the JXA script. The fix required an armAutomation() probe function that pre-triggers the permission window before the actual fetch, with a timeout window.

The local Ollama model (gemma4) wraps its JSON output in markdown code fences even when you ask it not to. The fix: parse from the first { to the last } and ignore whatever surrounds it.

And then there was the ARCHITECT architecture mistake. My first plan routed the agent through ClaudeClaw's chat API but that API is intentionally tool-free for security reasons. The agent could chat, but couldn't actually do anything. I caught this, paused, re-planned, and rebuilt: embed the Agent SDK directly in Electron main, run everything locally. That's the version that works, and works well.

Fable 5 caught the architectural problem during the re-plan conversation and designed the corrected solution. That's not autocomplete. That's engineering judgment.

What Fable 5 Made Possible

Here's the thing that keeps sticking with me: I'm not a developer by trade. I'm a systems engineer who's been learning to build with AI. In a previous chapter of my career this build would have been a months-long project requiring an entire team. What got built here in a few hours included the scheduling engine, the MCP server, the reactive database layer, and the audio pipeline. Just wow!

This was one session.

Fable 5 didn't just write code from my descriptions. It made architecture decisions. It identified when I was about to go down a wrong path (the ARCHITECT routing issue). It designed a pure functional scheduler module with 21 tests. It wired an MCP server from scratch. It debugged IPv6/IPv4 mismatches and macOS permission timing issues.

The code it writes is genuinely good code. It typed, tested where it matters, and followed established patterns. I watched Fable 5 write code and spin up an agent to perform an adversarial review more than once. The debugging process felt like working with a senior engineer who happened to also be infinitely patient about explaining tradeoffs.

The Series: Building Geekspace in Public

This article is the start of something bigger. I'm planning a full series on what I built, how it works, and what I learned. Here's where we're going:

Part 1 — You're reading it. The origin story: one prompt, one session, a complete Notion-style macOS app.

Part 2: The Calendar That Schedules Itself

A deep dive into the auto-scheduling engine. How earliest-deadline-first + priority + chunking actually works. How locked blocks, frozen history, and "needs attention" surfacing change the way you think about task management. Why I built it as a pure module with its own test suite and why that decision saved me three times.

Part 3: Five Bugs, Five Fixes — Debugging With Fable 5

The IPv6/IPv4 split-brain. The Automation permission race condition. The gemma4 JSON fence problem. The orphaned Convex backend port. The wrong architecture I almost shipped. Each one is a real debugging story with a real lesson about building local AI-native apps.

Part 4: AI Meeting Notes, 100% Local

Whisper.cpp + Ollama + a floating recorder UI. How I rebuilt Notion's AI meeting notes without sending audio anywhere. The pipeline that took five iterations to get right. The model behavior quirks you don't find in the documentation.

Part 5: One MCP Server, Three Runtimes

geekspace-mcp is a standard stdio MCP server. ARCHITECT runs it from Electron main. Claude Code can run it from the terminal. Claude Desktop can run it too. Building a workspace tool that any agent can drive and what that means for how AI and personal software intersect.

Part 6: What I'd Do Differently

Honest retrospective. The choices that worked out. The ones I'd reconsider. What building a complete app in one session actually costs you in terms of technical debt and understanding.

Video Walkthrough of the Geekspace Build

The full app is called Geekspace. It runs on my Mac, stays on my Mac, and does everything I actually use Notion for. If you want to follow the build in public as I write about it, subscribe below — Part 2 drops next.

Subscribe now

Frequently Asked Questions

Can Claude Fable 5 really build a full app from one prompt? Yes — in one session, from a single prompt, it produced a working macOS app with a block editor, five database views, an auto-scheduling calendar, local AI meeting notes, and an embedded agent. It chose the architecture itself before being asked.

What stack did Claude choose for a Notion-style desktop app? Electron + React 19 + Vite + Tailwind v4 + BlockNote for the editor, with Convex running in anonymous local mode — no account, no cloud, data stored in ~/.convex on the Mac.

Is a locally built Notion alternative actually private? Yes. Geekspace keeps data in a local Convex deployment, and the AI meeting notes run whisper.cpp plus a local Ollama model — the audio never leaves the Mac.

How do you give an AI agent tools inside a desktop app? The ARCHITECT agent runs the Claude Agent SDK in the Electron main process and connects to geekspace-mcp, a standard 14-tool MCP server. Because it's standard MCP, Claude Code and Claude Desktop can drive the same workspace.

What went wrong building an app this fast? Four real bugs: an IPv6/IPv4 localhost mismatch that blocked the window, a macOS Automation permission race in the Mail widget, gemma4 wrapping JSON in markdown fences, and an early agent architecture that couldn't run tools and had to be rebuilt.

Do you need to be a developer to do this? No. I'm a systems engineer learning to build with AI, not a professional developer. The skill that mattered most was knowing the outcome I wanted and recognizing when the architecture was wrong — not writing the code.

Share

Leave a comment

*Part of the Building Geekspace series — an honest look at what's possible when you partner with AI to build real software. Published at As The Geek Learns.*

It was a Tuesday at 2:17 PM, and the marketing team's contact form was returning 502s. Not 404. Not a timeout. A clean 502, which means something was answering, just not the thing it was supposed to be.

An hour in, I'd checked the app server logs, restarted the nginx process twice, confirmed the SSL cert was valid, and pinged our cloud provider's status page like it owed me money. Everything looked fine everywhere I looked. Then, almost by accident, I ran `dig +short www.company.com CNAME` and saw a hostname I didn't recognize. Something like `legacy-assets.decommissioned-vendor-name.com`.

Vendor had been off the account for four months. The CNAME had quietly repointed to their infrastructure during the migration wind-down, sat there untouched, and then their old infrastructure finally went dark. Nobody changed our DNS intentionally. Nobody got notified when it happened. We found out when a sales rep tried to submit a lead form.

That was the day I stopped trusting that "nothing changed in DNS" was a statement anyone could actually verify.

Why DNS Is the Silent-Failure Layer of Every Infrastructure

DNS is configuration. It's just not a configuration you can store in your repo, lint on a commit, or review in a pull request. It lives in a registrar panel or a DNS provider dashboard, updated by humans who may or may not be following a change-control process, and it's completely invisible until something breaks.

Every other layer of your stack has some kind of drift detection built in these days. Config management tools track the desired state of your servers. Container orchestrators know what's supposed to be running. Infrastructure-as-code tools will tell you if something drifted from the Terraform state. DNS gets none of that by default. You get a text field in a web UI, a change that takes effect whenever the TTL expires, and exactly zero notifications.

The operational pattern most teams rely on is "we'll know when it breaks." And they're right. They will know. They'll know at 2 PM on a Tuesday when a customer reports it, after a sales lead gets lost, after the support team has spent 45 minutes ruling out everything else. The detection mechanism is user reports, which is among the worst possible monitoring strategies.

There's also a subtler problem. The change usually isn't malicious. It's not a security incident, at least not at first. It's a vendor cleanup, a platform migration, someone at a partner org tidying up their infrastructure without realizing your CNAME still pointed at them. It's the kind of change that feels harmless to whoever made it and catastrophic to whoever depends on it.

The fix isn't complicated. What you need is a declared source of truth for what your DNS should look like, a way to compare that against what it actually looks like right now, and something that runs that comparison regularly enough to catch drift before users do.

That's the pattern. The implementation fits in a bash script.

The Pattern, the Four States, and a Wrapper You Can Use Today

The idea is straightforward: declare your expected DNS state once in a YAML file, then run a script nightly that queries your authoritative nameservers and compares what it finds against what you declared. Any gap between the two gets reported.

The baseline file is the key piece. It's not generated. You write it manually, and that act of writing it is itself useful, because it forces you to actually look up what each record currently is and decide "yes, that's correct." Once it exists, it becomes your source of truth. Commit it to your repo. Update it when you make a legitimate DNS change. The baseline is always what you intend, and the script is always asking whether reality matches.

When the diff runs, every record type for every domain you declared lands in one of four states:

MATCH means the live record matches the baseline exactly. This is the quiet result. Nothing to do.

NEW means a record exists in live DNS that isn't in your baseline. It could be a vendor auto-adding a TXT verification record. It could be someone provisioning a new subdomain. It could be something you should care about. The script surfaces it; you decide.

MISSING means your baseline declared a record that doesn't exist in live DNS anymore. An A record that was decommissioned without cleaning up. An MX record that got deleted. A CNAME that was removed when a vendor migrated their platform.

DRIFT means the baseline and live DNS both have records for a type, but the values don't match. This is the Tuesday-afternoon scenario: the CNAME target changed, the IP behind an A record flipped, the SPF policy was modified.

NEW and MISSING and DRIFT all mean something in your environment changed without you being told. The script exits nonzero when any of those occur, which makes it trivially composable with cron, alerting pipelines, or anything else that reads exit codes.

Here's a minimal working bash wrapper you can adapt right now. It keeps the dependencies to just `dig` and `bash`, uses a simple shell-array for your expected records instead of parsing YAML, and is short enough to read in under five minutes:


#!/usr/bin/env bash
# dns-check.sh
# WHAT: Minimal DNS drift checker - declare expected records, diff against live DNS
# WHY:  Catches silent DNS changes before they become incidents
# Usage: ./dns-check.sh
#        Add to cron: 0 2 * * * /path/to/dns-check.sh || echo "DNS DRIFT DETECTED" | mail -s "DNS Alert" you@example.com

set -euo pipefail

# ── CONFIGURATION ────────────────────────────────────────────────────────────
# Authoritative resolver to query against (use your domain's actual nameserver)
# WHY: Querying authoritative NS catches changes before they propagate to resolvers
RESOLVER="8.8.8.8"

# Declare expected records as: "domain|TYPE|expected_value"
# Get current values with: dig +short example.com A
# Run once to populate, then treat this as your source of truth
EXPECTED_RECORDS=(
  "example.com|A|93.184.216.34"
  "www.example.com|CNAME|example.com.cdn.cloudflare.net"
  "example.com|MX|10 mail.example.com"
  "example.com|TXT|v=spf1 include:_spf.google.com ~all"
)

# ── DIFF ENGINE ──────────────────────────────────────────────────────────────
DRIFT_FOUND=0

for record in "${EXPECTED_RECORDS[@]}"; do
  # Parse the declared record into its three parts
  domain="${record%%|*}"
  rest="${record#*|}"
  rtype="${rest%%|*}"
  expected="${rest#*|}"

  # Query live DNS at the authoritative resolver
  # WHY: +short gives us clean output; @resolver pins which nameserver answers
  actual=$(dig +short "@${RESOLVER}" "${domain}" "${rtype}" 2>/dev/null \
    | sort \
    | tr '\n' '|' \
    | sed 's/\.$//g; s/|$//')

  # Normalize expected for comparison (sort, strip trailing dots)
  expected_norm=$(printf '%s\n' "${expected}" \
    | sort \
    | tr '\n' '|' \
    | sed 's/\.$//g; s/|$//')
      # Compare and classify the result
  if [[ -z "${actual}" ]]; then
    # Record existed in baseline but dig returned nothing: MISSING
    printf "MISSING  %s %s  (expected: %s)\n" "${domain}" "${rtype}" "${expected}"
    DRIFT_FOUND=1
  elif [[ "${actual}" != "${expected_norm}" ]]; then
    # Record exists but value changed: DRIFT
    printf "DRIFT    %s %s\n  expected: %s\n  actual:   %s\n" \
      "${domain}" "${rtype}" "${expected}" "${actual}"
    DRIFT_FOUND=1
  else
    # Values match: MATCH (silent - no output unless you add --verbose logic)
    : # nothing to report
  fi
done

# Exit nonzero on any drift - composable with cron, alerting, CI checks
if [[ "${DRIFT_FOUND}" -eq 1 ]]; then
  printf "\nDrift detected. Review records above.\n" >&2
  exit 1
fi

printf "All %d declared records match live DNS.\n" "${#EXPECTED_RECORDS[@]}"
exit 0

Save that, drop your actual records into `EXPECTED_RECORDS`, and run it once to confirm it sees what you expect. Then add it to your crontab:

# Run nightly at 2 AM, email on drift
0 2 * * * /path/to/dns-check.sh || echo "DNS drift detected on $(hostname)" | mail -s "[ALERT] DNS Drift" you@example.com

The "NEW record exists in live DNS" state isn't in this minimal version, since detecting it requires knowing which record types to scan for beyond what you declared. The four-state model handles that fully once you know which types to watch, which is what the complete kit covers. For a first pass, catching MISSING and DRIFT gets you most of the value.

A few practical notes. Use `dig +short` rather than `dig` without `+short` or you'll spend time parsing the human-readable output format. Always query a specific nameserver with `@resolver` rather than relying on your local resolver, since caching can hide drift for hours. The MX record normalization is worth being careful about: `dig +short` returns the priority prefix as part of the value (`10 mail.example.com`), so your expected strings need to include it exactly that way. And commit the script alongside your baseline declaration. If the baseline lives in the repo, you get history, diffs, and code review for DNS changes as a side effect.

Share As The Geek Learns

What Else Lives in the Full Kit

The script above covers the core pattern. The full DNS Drift Detector kit is what you reach for once you've outgrown the wrapper.

The main `dns-drift-detector.sh` handles all five record types: A, AAAA, CNAME, MX, and TXT. That last group matters more than it seems. TXT records are where SPF policies live, where DKIM selectors sit, where domain verification tokens accumulate. Quiet SPF drift can break your email deliverability for days before anyone notices. DKIM drift means legitimate mail starts hitting spam folders. These aren't hypothetical edge cases.

The color-coded output makes the diff results readable at a glance during incident response, and the `--quiet` flag strips all of that for cron-friendly logging where you only want the exit code to speak. There's a `--no-color` flag too, so piping to a log file doesn't fill it with ANSI escape sequences.

`install-cron.sh` is a one-command idempotent installer. It checks that `dig` is available, puts the scripts where they belong, creates a log directory, and writes the cron entry with a duplicate-guard marker so running it twice doesn't add the job twice. That kind of thing is boring to write and annoying to get wrong.

The `baseline.yaml` in the kit is annotated with examples for ten common services: Google Workspace MX, Cloudflare CDN CNAMEs, SendGrid SPF, common DKIM selectors, and a few others. It's the reference you use when you're populating your own baseline for the first time.

The runbook covers install, baseline setup, how to read the output, what to do for each of the four states, how to update the baseline after a legitimate change, and an FAQ. That last one matters during an incident, when you don't want to be making judgment calls about whether a NEW record means "update the baseline" or "call the registrar."

Try the Pattern Today

The pattern described here is genuinely useful as-is. Declare your records, schedule the diff, react to the exits. That alone puts you ahead of the "we'll know when it breaks" approach that most infrastructure environments are actually running.

If you want the full kit, it's at shop.asthegeeklearns.com/products/dns-drift-detector for $19. You get the complete `dns-drift-detector.sh` with all five record types, color output, quiet mode, and cron logging; the idempotent `install-cron.sh`; the annotated `baseline.yaml` with ten real-world service examples; and the full operator runbook.

The Tuesday-afternoon incident I described at the top cost more than $19 worth of everyone's time. The detection script would have caught it the night before.

As The Geek Learns is a newsletter about systems engineering, automation, and the gap between knowing something and actually applying it. If this was useful, subscribe for free to get new articles as they land.

Share

Leave a comment

If that sentence doesn’t make you slightly nervous, you haven’t been paying attention. In February 2026, researchers found over 135,000 OpenClaw instances exposed to the public internet. A coordinated attack called ClawHavoc planted over a thousand malicious plugins in the community registry. Nine CVEs have been disclosed, including remote code execution.

I needed to take security seriously. Not “I changed the default password” seriously. Threat-model seriously.

MAESTRO: Seven Layers of Things That Can Go Wrong

The Cloud Security Alliance published a framework called MAESTRO—a 7-layer threat model specifically designed for agentic AI systems. Ken Huang mapped it directly to OpenClaw’s codebase, identifying 35+ specific threats across every layer of the stack.

Here are the seven layers, translated from security-paper language into “things that could actually ruin your day”:

Layer 1: Foundation Models: Someone sends your agent a crafted message that hijacks its behavior. Prompt injection. Jailbreaks. System prompt leakage. Your agent does what an attacker tells it to instead of what you told it to.

Layer 2: Data Operations: Your credentials are stored in plaintext JSON files. Your session logs contain every conversation forever. A malicious skill injects code through your workspace.

Layer 3: Agent Frameworks: The agent misuses its own tools. It runs shell commands it shouldn’t. It spawns sessions without authorization. It escalates its own privileges.

Layer 4: Deployment & Infrastructure: Your gateway is exposed to the network. Someone brute-forces the WebSocket token. A reverse proxy misconfiguration bypasses authentication entirely.

Layer 5: Evaluation & Observability: Nobody’s watching the agent for anomalous behavior. There’s no audit trail. Logs can be tampered with. If the agent starts acting weird, nothing catches it.

Layer 6: Security & Compliance: Your DM policy is misconfigured. Anyone can message the agent. Pairing codes can be brute-forced. Identity can be spoofed across channels.

Layer 7: Agent Ecosystem: A malicious plugin gets installed. A legitimate plugin’s npm dependency gets compromised. The skill registry serves poisoned packages.

The critical attack chain MAESTRO identifies: compromise the gateway (Layer 4) → access the session store (Layer 2) → poison conversation history (Layer 1) → control the agent (Layer 3) → spread via messaging (Layer 7).

Reading this was humbling. I’d addressed some of these by instinct during setup. Loopback binding, directory permissions, and pairing-based access control were all implemented. But “some” isn’t a security posture.

SecureClaw: The Audit

SecureClaw is an open-source security tool built specifically for OpenClaw by Adversa AI. It maps to MAESTRO, OWASP, MITRE ATLAS, and NIST AI 100-2. The install is a git clone and a bash script, no npm install, no network calls, and no surprises.

git clone https://github.com/adversa-ai/secureclaw.git
bash secureclaw/secureclaw/skill/scripts/install.sh

Then you run the audit:

bash ~/.openclaw/skills/secureclaw/scripts/quick-audit.sh

My baseline score: 57 out of 100. Zero criticals. Three HIGHs. Three MEDIUMs. Eight checks passing.

Here’s what passed without any work:

• Gateway bound to loopback (127.0.0.1) not exposed to network

• Gateway authentication present

• Directory permissions set to 700 (owner only)

• No browser relay exposed

• DM policy set to pairing (not open)

• Skills clean of malicious patterns

And here’s what failed:

🟠 HIGH Plaintext key exposure: Keys in openclaw.json and 5 backup files

🟠 HIGH Sandbox mode: commands run directly on host

🟠 HIGH Exec approval mode: agent acts without human approval

🟡 MED No cognitive file baselines: can’t detect tampering

🟡 MED Default control tokens: vulnerable to spoofing

🟡 MED No failure mode: no graceful degradation

The Hardening

Step 1: Clean up credential leaks. OpenClaw creates .bak files every time you change config. Each backup contains your full config, including Slack tokens and API keys. I had five of them sitting in the OpenClaw directory. Deleted them all. Set the main config to 600 permissions.

This is the kind of thing that’s easy to miss and catastrophic to ignore. A single ls -la ~/.openclaw/ would show them. But who runs ls -la on their config directory after every change?

Step 2: Create integrity baselines. SecureClaw’s hardener generates SHA256 hashes of your “cognitive files” IDENTITY.md, AGENTS.md, and HEARTBEAT.md. These are the files that define who your agent is and what it does. If an attacker or a hallucinating agent modifies them, the nightly integrity check will catch it.

bash ~/.openclaw/skills/secureclaw/scripts/quick-harden.sh

Step 3: Exec approvals. This is the big one. MAESTRO recommends human-in-the-loop approval for all shell commands. But my agent runs morning briefings and heartbeat checks on cron—unattended. Setting approvals to “always” would break all automation.

The solution: an allowlist with on-miss approval. I created ~/.openclaw/exec-approvals.json with 17 safe command patterns: imsg, calctl, apple-reminders, cairn, and basic file operations. Tars can run these freely. Anything else; curl, rm, pip install, or any command not on the list, requires human approval.

{
  “defaults”: {
    “security”: “allowlist”,
    “ask”: “on-miss”
  },
  “agents”: {
    “main”: {
      “allowlist”: [
        { “pattern”: “imsg *”, “note”: “iMessage send/read” },
        { “pattern”: “calctl *”, “note”: “Apple Calendar” },
        { “pattern”: “cairn *”, “note”: “Task management” }
      ]
    }
  }
}

This is the trade-off MAESTRO doesn’t talk about: security versus automation. Maximum security means every action needs approval. Maximum automation means the agent acts freely. The allowlist is the middle ground. Routine operations are pre-approved, and novel or dangerous operations require a human.

Step 4: Full plugin install. Beyond the bash scripts, SecureClaw has a full npm plugin with 56 runtime audit checks, background monitors for config drift, and real-time integrity verification. Installing it required building from source (TypeScript → JavaScript) and registering it with OpenClaw’s plugin system.

openclaw plugins install -l /path/to/secureclaw

openclaw config set plugins.allow ‘[”secureclaw”]’

That plugins.allow line is important. By default, OpenClaw will auto-load any discovered plugin. Explicit trust means only plugins you’ve approved get loaded.

Step 5: Nightly audit cron. A macOS LaunchAgent runs the full audit suite every night at 2 AM which includes quick-audit, integrity check, and supply chain scan. Results go to secureclaw-audit.log. If something changes overnight, it shows up in the morning.

The Final Score

After hardening: 64 out of 100. Nine checks passing. Zero criticals. The three remaining HIGHs are documented, accepted trade-offs:

Findings I accepted (with reasoning)—Sandbox mode (Docker sandboxing would break imsg, calctl, and Apple Reminders); Plaintext keys in config (inherent to the platform config format, file is locked to 600); Exec approval not “always” (using allowlist + on-miss; full “always” breaks unattended cron automation).

The two MEDIUMs, control token customization and failure mode configuration, aren’t supported in OpenClaw v2026.3.2’s config schema yet. SecureClaw checks for them proactively. They’ll be fixable when OpenClaw adds the config options.

What I Actually Learned

Security isn’t a feature you enable. It’s a series of trade-offs you make with your eyes open. Sandbox mode is “more secure” but breaks the tools that make the agent useful. Approval mode “always” is “more secure” but kills the automation that makes the agent worthwhile. The right security posture isn’t maximum restriction; it’s documented, intentional decisions about what risks you accept and why.

Automated scanning is essential but insufficient. SecureClaw’s audit caught things I would have missed, including the .bak files with credentials, the missing integrity baselines, and the open exec policy. But the HIGHs it flagged as failures are things I’ve consciously accepted. No scanner can evaluate your specific trade-offs.

The biggest threat isn’t external. In my setup (loopback-bound, pairing-gated, allowlist-filtered), the most likely security failure isn’t a network attacker. It’s a malicious skill, a compromised npm package, or the agent itself hallucinating destructive actions. Layer 7 (ecosystem) and Layer 1 (model behavior) are the real attack surfaces for a local-first setup. The exec approval allowlist is my primary defense for both.

Clean up after yourself. OpenClaw creates backup files containing credentials on every config change. There’s no auto-cleanup. If you’re running OpenClaw, go check your directory right now: ls ~/.openclaw/*.bak*. You might be surprised.

Quick Reference

Hardening actions and commands: install, run audit, apply hardening, check integrity, scan skills, check for credential leaks, set exec approvals, set plugin trust. Commands target ~/.openclaw/skills/secureclaw/scripts/. Full command details in the image.

Update—June 2026: What I Actually Did When I Moved to ClaudeClaw

I wrote this piece in March, when OpenClaw was still the thing running my Mac Studio. By the end of April, I’d shut it down. Disabled the cron jobs, quarantined the LaunchAgents, and rebuilt the whole stack on the Claude Agent SDK. Based off of ClaudeClaw from the Early AI-Dopters AI learning group. The full post-mortem on why:

Why? The short version is this: I couldn’t see into OpenClaw. Which, if you scroll back up, is Layer 5: Evaluation & Observability, the exact layer this audit was weakest on.

You may wonder whether I just copied the 7-layer hardening over to the new stack. I didn’t, and I want to be honest about that. I did not port MAESTRO one-for-one. SecureClaw was written specifically for OpenClaw. Some of its thinking transferred; some of it didn’t. And the threat model itself moved on (more on that at the end). What the seven layers became was a checklist: for each one, how does the new architecture answer this? Here’s the scorecard.

The two layers that changed the most.

Layer 5 (Observability) went from my single biggest weakness to the entire reason ClaudeClaw exists. There’s now a dedicated agent, WATCHMAN, running seven probes every hour: failed tasks, stuck tasks, missed scheduler slots, daemon liveness, content-pipeline health, hidden failures (it greps the success logs for crash text), and delegation crashes. More importantly, there’s a second healthcheck running as a separate LaunchAgent with its own keychain-backed alert token. If the main daemon dies, the thing that tells me about it is still alive. The rule I wrote for myself out of this: the watcher cannot share fate with the watched. There’s also a behavioral dashboard, DefenseClaw, sitting on 127.0.0.1:3141.

Layer 3 (Agent Frameworks) is where my OpenClaw work actually carried forward. The exec-approvals allowlist from Step 3 above is the direct ancestor of what ClaudeClaw does now, except the enforcement dropped down a level. The first thing I shipped was killing bypassPermissions (the main agent had been running with permission checks disabled, which means a compromised agent has unlimited tool access. The SDK was no ceiling at all), switching to the SDK’s default permission mode, and handing the main agent a 15-tool allowlist as the single source of truth. Same idea as the OpenClaw allowlist. Enforced by the SDK itself instead of a config file I had to maintain.

The rest mapped like this:

How each of the seven MAESTRO layers from the OpenClaw audit is answered in ClaudeClaw.

Layer 1 Foundation Models: channel tagging and a trust gradient that treats retrieved text as data, not directives (evolved).

Layer 2 Data Operations: Chamberlain outbound scanner, exfiltration-guard, queryable Memory v2, and ingest-time canonicalization (replaced and extended).

Layer 3 Agent Frameworks: SDK permission ceiling and a 15-tool allowlist, the direct successor to the OpenClaw exec-approvals list (kept, moved into the SDK).

Layer 4 Deployment and Infrastructure: an egress gateway plus kernel-level pf default-deny (replaced).

Layer 5 Evaluation and Observability: WATCHMAN’s seven probes and a fate-isolated external healthcheck, the biggest upgrade.

Layer 6 Security and Compliance: out-of-band Telegram confirmation for state-changing actions and a role policy kept separate from content memory (evolved).

Layer 7 Agent Ecosystem: an MCP allowlist plus the tool ceiling as a second layer (kept and hardened).

Plus a new row beyond MAESTRO. Memory persistence: TTLs, a hash-chained write log, and canaries.

Where the 7-layer model ran out.

MAESTRO is a static threat model. It’s a map of what can go wrong at each layer, frozen in time. What it doesn’t have a layer for is persistence. An attack that lands quietly in your agent’s memory or vector store and just waits. My scheduler re-enters context every 60 seconds, which means anything dormant in memory fires on a clock. That’s a different class of problem, and it has a name now: LPCI, Logic-layer Prompt-based Conditional Injection. Hardening against it (I am planning a separate two-part write-up on As The Geek Learns) meant building things MAESTRO never asked for, including a canonicalizer that decodes payloads before they reach the vector store, channel-tagged prompts so the model knows retrieved text is data and not instructions, memory TTLs, a hash-chained write log, and canary entries that page me if memory ever leaks into output.

What I gave up and what I kept. The honest cost of the move: I lost local-first. OpenClaw ran on Ollama, fully offline; ClaudeClaw talks to Anthropic’s API. I still own every byte of my data; it’s all on my SSD; I just don’t own the weights anymore. What carried over intact was the philosophy this whole series is built on: every document is a file I can grep, every config is version-controlled, and every decision has a session note. That part never changed.

This is Part 5 of the Notion Replacement series. We went from “install an AI agent” to “secure it against a 7-layer threat model” in two days. Follow along at As The Geek Learns.

I have an autonomous AI agent running on my Mac Studio. It has full shell access, reads my calendar, manages my tasks, and sends iMessages on my behalf. It runs 24/7 as a background service.

If that sentence doesn’t make you slightly nervous, you haven’t been paying attention. In February 2026, researchers found over 135,000 OpenClaw instances exposed to the public internet. A coordinated attack called ClawHavoc planted over a thousand malicious plugins in the community registry. Nine CVEs have been disclosed, including remote code execution.

I needed to take security seriously. Not “I changed the default password” seriously. Threat-model seriously.

MAESTRO: Seven Layers of Things That Can Go Wrong

The Cloud Security Alliance published a framework called MAESTRO—a 7-layer threat model specifically designed for agentic AI systems. Ken Huang mapped it directly to OpenClaw’s codebase, identifying 35+ specific threats across every layer of the stack.

Here are the seven layers, translated from security-paper language into “things that could actually ruin your day”:

Layer 1: Foundation Models: Someone sends your agent a crafted message that hijacks its behavior. Prompt injection. Jailbreaks. System prompt leakage. Your agent does what an attacker tells it to instead of what you told it to.

Layer 2: Data Operations: Your credentials are stored in plaintext JSON files. Your session logs contain every conversation forever. A malicious skill injects code through your workspace.

Layer 3: Agent Frameworks: The agent misuses its own tools. It runs shell commands it shouldn’t. It spawns sessions without authorization. It escalates its own privileges.

Layer 4: Deployment & Infrastructure: Your gateway is exposed to the network. Someone brute-forces the WebSocket token. A reverse proxy misconfiguration bypasses authentication entirely.

Layer 5: Evaluation & Observability: Nobody’s watching the agent for anomalous behavior. There’s no audit trail. Logs can be tampered with. If the agent starts acting weird, nothing catches it.

Layer 6: Security & Compliance: Your DM policy is misconfigured. Anyone can message the agent. Pairing codes can be brute-forced. Identity can be spoofed across channels.

Layer 7: Agent Ecosystem: A malicious plugin gets installed. A legitimate plugin’s npm dependency gets compromised. The skill registry serves poisoned packages.

The critical attack chain MAESTRO identifies: compromise the gateway (Layer 4) → access the session store (Layer 2) → poison conversation history (Layer 1) → control the agent (Layer 3) → spread via messaging (Layer 7).

Reading this was humbling. I’d addressed some of these by instinct during setup. Loopback binding, directory permissions, and pairing-based access control were all implemented. But “some” isn’t a security posture.

SecureClaw: The Audit

SecureClaw is an open-source security tool built specifically for OpenClaw by Adversa AI. It maps to MAESTRO, OWASP, MITRE ATLAS, and NIST AI 100-2. The install is a git clone and a bash script, no npm install, no network calls, and no surprises.

git clone https://github.com/adversa-ai/secureclaw.git
bash secureclaw/secureclaw/skill/scripts/install.sh

Then you run the audit:

bash ~/.openclaw/skills/secureclaw/scripts/quick-audit.sh

My baseline score: 57 out of 100. Zero criticals. Three HIGHs. Three MEDIUMs. Eight checks passing.

Here’s what passed without any work:

• Gateway bound to loopback (127.0.0.1) not exposed to network

• Gateway authentication present

• Directory permissions set to 700 (owner only)

• No browser relay exposed

• DM policy set to pairing (not open)

• Skills clean of malicious patterns

And here’s what failed:

🟠 HIGH Plaintext key exposure: Keys in openclaw.json and 5 backup files

🟠 HIGH Sandbox mode: commands run directly on host

🟠 HIGH Exec approval mode: agent acts without human approval

🟡 MED No cognitive file baselines: can’t detect tampering

🟡 MED Default control tokens: vulnerable to spoofing

🟡 MED No failure mode: no graceful degradation

The Hardening

Step 1: Clean up credential leaks. OpenClaw creates .bak files every time you change config. Each backup contains your full config, including Slack tokens and API keys. I had five of them sitting in the OpenClaw directory. Deleted them all. Set the main config to 600 permissions.

This is the kind of thing that’s easy to miss and catastrophic to ignore. A single ls -la ~/.openclaw/ would show them. But who runs ls -la on their config directory after every change?

Step 2: Create integrity baselines. SecureClaw’s hardener generates SHA256 hashes of your “cognitive files” IDENTITY.md, AGENTS.md, and HEARTBEAT.md. These are the files that define who your agent is and what it does. If an attacker or a hallucinating agent modifies them, the nightly integrity check will catch it.

bash ~/.openclaw/skills/secureclaw/scripts/quick-harden.sh

Step 3: Exec approvals. This is the big one. MAESTRO recommends human-in-the-loop approval for all shell commands. But my agent runs morning briefings and heartbeat checks on cron—unattended. Setting approvals to “always” would break all automation.

The solution: an allowlist with on-miss approval. I created ~/.openclaw/exec-approvals.json with 17 safe command patterns: imsg, calctl, apple-reminders, cairn, and basic file operations. Tars can run these freely. Anything else; curl, rm, pip install, or any command not on the list, requires human approval.

{
  "defaults": {
    "security": "allowlist",
    "ask": "on-miss"
  },
  "agents": {
    "main": {
      "allowlist": [
        { "pattern": "imsg *", "note": "iMessage send/read" },
        { "pattern": "calctl *", "note": "Apple Calendar" },
        { "pattern": "cairn *", "note": "Task management" }
      ]
    }
  }
}

This is the trade-off MAESTRO doesn’t talk about: security versus automation. Maximum security means every action needs approval. Maximum automation means the agent acts freely. The allowlist is the middle ground. Routine operations are pre-approved, and novel or dangerous operations require a human.

Step 4: Full plugin install. Beyond the bash scripts, SecureClaw has a full npm plugin with 56 runtime audit checks, background monitors for config drift, and real-time integrity verification. Installing it required building from source (TypeScript → JavaScript) and registering it with OpenClaw’s plugin system.

openclaw plugins install -l /path/to/secureclaw

openclaw config set plugins.allow ‘[”secureclaw”]’

That plugins.allow line is important. By default, OpenClaw will auto-load any discovered plugin. Explicit trust means only plugins you’ve approved get loaded.

Step 5: Nightly audit cron. A macOS LaunchAgent runs the full audit suite every night at 2 AM which includes quick-audit, integrity check, and supply chain scan. Results go to secureclaw-audit.log. If something changes overnight, it shows up in the morning.

The Final Score

After hardening: 64 out of 100. Nine checks passing. Zero criticals. The three remaining HIGHs are documented, accepted trade-offs:

Findings I accepted (with reasoning)—Sandbox mode (Docker sandboxing would break imsg, calctl, and Apple Reminders); Plaintext keys in config (inherent to the platform config format, file is locked to 600); Exec approval not “always” (using allowlist + on-miss; full “always” breaks unattended cron automation).

The two MEDIUMs, control token customization and failure mode configuration, aren’t supported in OpenClaw v2026.3.2’s config schema yet. SecureClaw checks for them proactively. They’ll be fixable when OpenClaw adds the config options.

What I Actually Learned

Security isn’t a feature you enable. It’s a series of trade-offs you make with your eyes open. Sandbox mode is “more secure” but breaks the tools that make the agent useful. Approval mode “always” is “more secure” but kills the automation that makes the agent worthwhile. The right security posture isn’t maximum restriction; it’s documented, intentional decisions about what risks you accept and why.

Automated scanning is essential but insufficient. SecureClaw’s audit caught things I would have missed, including the .bak files with credentials, the missing integrity baselines, and the open exec policy. But the HIGHs it flagged as failures are things I’ve consciously accepted. No scanner can evaluate your specific trade-offs.

The biggest threat isn’t external. In my setup (loopback-bound, pairing-gated, allowlist-filtered), the most likely security failure isn’t a network attacker. It’s a malicious skill, a compromised npm package, or the agent itself hallucinating destructive actions. Layer 7 (ecosystem) and Layer 1 (model behavior) are the real attack surfaces for a local-first setup. The exec approval allowlist is my primary defense for both.

Clean up after yourself. OpenClaw creates backup files containing credentials on every config change. There’s no auto-cleanup. If you’re running OpenClaw, go check your directory right now: ls ~/.openclaw/*.bak*. You might be surprised.

Quick Reference

Hardening actions and commands: install, run audit, apply hardening, check integrity, scan skills, check for credential leaks, set exec approvals, set plugin trust. Commands target ~/.openclaw/skills/secureclaw/scripts/. Full command details in the image.

Share

Update—June 2026: What I Actually Did When I Moved to ClaudeClaw

I wrote this piece in March, when OpenClaw was still the thing running my Mac Studio. By the end of April, I’d shut it down. Disabled the cron jobs, quarantined the LaunchAgents, and rebuilt the whole stack on the Claude Agent SDK. Based off of ClaudeClaw from the Early AI-Dopters AI learning group. The full post-mortem on why:

Why? The short version is this: I couldn’t see into OpenClaw. Which, if you scroll back up, is Layer 5: Evaluation & Observability, the exact layer this audit was weakest on.

You may wonder whether I just copied the 7-layer hardening over to the new stack. I didn’t, and I want to be honest about that. I did not port MAESTRO one-for-one. SecureClaw was written specifically for OpenClaw. Some of its thinking transferred; some of it didn’t. And the threat model itself moved on (more on that at the end). What the seven layers became was a checklist: for each one, how does the new architecture answer this? Here’s the scorecard.

The two layers that changed the most.

Layer 5 (Observability) went from my single biggest weakness to the entire reason ClaudeClaw exists. There’s now a dedicated agent, WATCHMAN, running seven probes every hour: failed tasks, stuck tasks, missed scheduler slots, daemon liveness, content-pipeline health, hidden failures (it greps the success logs for crash text), and delegation crashes. More importantly, there’s a second healthcheck running as a separate LaunchAgent with its own keychain-backed alert token. If the main daemon dies, the thing that tells me about it is still alive. The rule I wrote for myself out of this: the watcher cannot share fate with the watched. There’s also a behavioral dashboard, DefenseClaw, sitting on 127.0.0.1:3141.

Layer 3 (Agent Frameworks) is where my OpenClaw work actually carried forward. The exec-approvals allowlist from Step 3 above is the direct ancestor of what ClaudeClaw does now, except the enforcement dropped down a level. The first thing I shipped was killing bypassPermissions (the main agent had been running with permission checks disabled, which means a compromised agent has unlimited tool access. The SDK was no ceiling at all), switching to the SDK’s default permission mode, and handing the main agent a 15-tool allowlist as the single source of truth. Same idea as the OpenClaw allowlist. Enforced by the SDK itself instead of a config file I had to maintain.

The rest mapped like this:

How each of the seven MAESTRO layers from the OpenClaw audit is answered in ClaudeClaw.

Layer 1 Foundation Models: channel tagging and a trust gradient that treats retrieved text as data, not directives (evolved).

Layer 2 Data Operations: Chamberlain outbound scanner, exfiltration-guard, queryable Memory v2, and ingest-time canonicalization (replaced and extended).

Layer 3 Agent Frameworks: SDK permission ceiling and a 15-tool allowlist, the direct successor to the OpenClaw exec-approvals list (kept, moved into the SDK).

Layer 4 Deployment and Infrastructure: an egress gateway plus kernel-level pf default-deny (replaced).

Layer 5 Evaluation and Observability: WATCHMAN’s seven probes and a fate-isolated external healthcheck, the biggest upgrade.

Layer 6 Security and Compliance: out-of-band Telegram confirmation for state-changing actions and a role policy kept separate from content memory (evolved).

Layer 7 Agent Ecosystem: an MCP allowlist plus the tool ceiling as a second layer (kept and hardened).

Plus a new row beyond MAESTRO. Memory persistence: TTLs, a hash-chained write log, and canaries.

Where the 7-layer model ran out.

MAESTRO is a static threat model. It’s a map of what can go wrong at each layer, frozen in time. What it doesn’t have a layer for is persistence. An attack that lands quietly in your agent’s memory or vector store and just waits. My scheduler re-enters context every 60 seconds, which means anything dormant in memory fires on a clock. That’s a different class of problem, and it has a name now: LPCI, Logic-layer Prompt-based Conditional Injection. Hardening against it (I am planning a separate two-part write-up on As The Geek Learns) meant building things MAESTRO never asked for, including a canonicalizer that decodes payloads before they reach the vector store, channel-tagged prompts so the model knows retrieved text is data and not instructions, memory TTLs, a hash-chained write log, and canary entries that page me if memory ever leaks into output.

What I gave up and what I kept. The honest cost of the move: I lost local-first. OpenClaw ran on Ollama, fully offline; ClaudeClaw talks to Anthropic’s API. I still own every byte of my data; it’s all on my SSD; I just don’t own the weights anymore. What carried over intact was the philosophy this whole series is built on: every document is a file I can grep, every config is version-controlled, and every decision has a session note. That part never changed.

Share As The Geek Learns

This is Part 5 of the Notion Replacement series. We went from “install an AI agent” to “secure it against a 7-layer threat model” in two days. Follow along at As The Geek Learns.

Leave a comment

5 Questions to Ask Before You Build the AI Project Your CEO Just Pitched

You know the email. It shows up Tuesday morning, forwarded with a few lines of enthusiasm and a ChatGPT-drafted proposal attached. "Saw this and thought of us. Can we do this?" The PDF has a logo, bullet points, and exactly zero integration requirements. It also has a six-week timeline and a budget that assumes nothing goes wrong.

You have somewhere between 24 and 72 hours before your CEO follows up asking what you think.

If you say yes, you're on the hook for a project you didn't scope. If you say no, you're the person who kills ideas. Neither answer is actually available to you. What you need is a third path: a structured evaluation that produces a defensible, professional response in the time it takes to drink your morning coffee.

That's what the Technical Reality Check is. Five questions. One page. Every answer points directly at a commitment your organization will have to honor if this project moves forward.

Here it is in full.

The Technical Reality Check: 5 Questions That Surface What the Proposal Left Out

Question 1: What specific business outcome does this solve, and how will we measure success?

AI tools generate confident-sounding proposals that describe solutions, not problems. A proposal for "an AI-powered IT ticketing system" describes a technology. It doesn't describe what's broken right now, how broken it is, or what "fixed" looks like in measurable terms.

Before any conversation about implementation, you need an answer to: what does success look like in six months, and how will we know we hit it? Ticket resolution time down 30%? First-contact resolution rate up 20%? Those are real answers. "Things will be more efficient" is not.

Unmeasurable projects never officially fail. Which means they never stop consuming resources. This question isn't about being difficult. It's about making sure the organization is buying an outcome, not a technology.

The red flag: Any proposal where the only success metric is "we deployed it."

Question 2: Who owns the ongoing maintenance, security patching, and vendor relationship?

Vendor proposals describe launch day. They are almost entirely silent about year two.

Every new system creates a permanent maintenance obligation: patching, credential rotation, user access reviews, API deprecations, contract renewals, and a support relationship with a vendor whose incentives are not aligned with yours. If that obligation doesn't have a named owner before the project starts, IT inherits it by default. Forever. Without headcount.

This question forces the conversation about operational reality before anyone has signed a contract. The answer also tells you a lot about how seriously the proposal was thought through. If nobody has asked "who maintains this?", nobody has thought past the demo.

The red flag: "The vendor handles everything." Vendors handle their system. You handle the integration, the credentials, the user provisioning, the data pipeline, and the 2 AM alert when something breaks between their system and yours.

Question 3: What happens to our existing systems, data, and processes?

New systems don't exist in a vacuum. They touch your directory, your ticketing system, your identity provider, your backup scope, your audit logs. Each of those integration points is a potential failure mode, a migration cost, or a compliance question.

AI-generated proposals routinely skip integration complexity. This isn't because the AI is being deceptive. It's because the AI generating the proposal doesn't know your stack. The proposal was written in a context-free environment. Your environment is anything but.

Before committing, you need to know: what does this touch, and what has to move or change for it to work? And who does that work? Data migration alone can turn a "simple" deployment into a multi-month project. Asking this question early is how you find out.

The red flag: "It integrates easily with your existing tools." That's a sales phrase, not an engineering estimate. "Easy" is undefined until your systems engineer has looked at the API docs.

Question 4: What's the realistic timeline and resource cost, not the optimistic one?

Vendor timelines assume clean data, available staff, smooth approvals, and nothing else on the backlog. Your timeline accounts for your actual team, their current commitments, the security review cycle, the change management process, and the three things nobody predicted.

The gap between those two numbers is usually where projects go sideways. Not because the technology failed, but because the plan never accounted for reality.

This question also surfaces a common pattern: the timeline was set before IT was consulted. Any timeline that precedes a technical assessment is a guess dressed up as a schedule. You're the one who'll be explaining the delay when the guess turns out to be wrong.

The red flag: A go-live date in the proposal. That's not a plan, it's a target somebody made up. Ask who set it and what it was based on.

Question 5: What's the exit strategy if this doesn't work as expected?

Every vendor says their product works. You need a plan for when it doesn't. When the pricing doubles at renewal. When the company gets acquired and support degrades. When a compliance requirement changes and the product doesn't keep up.

Data portability, rollback procedures, and contractual exit terms are not pessimism. They're the difference between a manageable failure and a situation where you're paying for a system that doesn't work because migrating off it is too expensive to contemplate.

This question also signals organizational maturity. IT teams that ask exit questions before they sign contracts don't get held hostage. IT teams that don't ask end up managing a five-year sunset project for a tool they stopped believing in three years ago.

The red flag: "We can always just stop using it." Can you migrate your data? In what format? At what cost? How long does it take? If nobody has asked those questions, stopping isn't as simple as it sounds.

The Checklist in Practice: Walking Through a Real Scenario

Here's what a Technical Reality Check pass looks like when you actually run it.

Your CEO forwards a ChatGPT-drafted proposal on a Monday morning. The subject line is "AI Agent for IT Ticket Triage." The proposal is two pages. It describes an AI system that reads incoming IT tickets, categorizes them by priority and type, routes them to the right team, and drafts first-response emails automatically. There's a mockup screenshot. There's a line about "easy integration with your existing ITSM." There's a timeline: six weeks to deployment.

You open the Technical Reality Check.

Q1: What specific business outcome does this solve?

The proposal says "reduce response times and improve IT efficiency." No baseline. No metric. You check your current ITSM data: average first response is 4.2 hours, your SLA target is 2 hours, you're meeting it 71% of the time. Now you have a problem worth solving. You write it down: "We need first-response SLA compliance above 85%. Current state: 71%." That's the outcome. If the AI system can't demonstrate a path to that specific number, the conversation is premature.

Q2: Who owns maintenance and the vendor relationship?

Nobody is named in the proposal. You have a team of four. One of them is already carrying the ITSM admin role. You note: this needs a named owner and a rough estimate of ongoing hours before it can go to planning. You also flag the API integration dependency: your ITSM has a rate-limited API that's caused problems before. Someone needs to read the vendor's API docs before "easy integration" gets treated as a fact.

Q3: What happens to existing systems and data?

Your ticketing data includes ticket histories, customer records, and some attachments. The proposal doesn't mention data handling. You note two questions: where does ticket data go once the AI processes it, and what are the data residency requirements given that you handle some HIPAA-adjacent systems? That second question alone could be a blocker. You don't know yet, but you know to ask.

Q4: What's the realistic timeline and resource cost?

Six weeks assumes nothing else is happening. Your team is currently in the middle of a server migration that runs through the end of the month. Realistically, this project can't start until mid-next-month, and your most experienced engineer (the one who'd need to own the integration) is at 90% utilization. You write down: "Realistic start: six weeks out. Realistic deployment: 12-16 weeks from proposal receipt. Not 6."

Q5: What's the exit strategy?

The proposal doesn't mention it. You note: before any contract, you need to know the data export format, the contract term length, and what happens to stored ticket data at offboarding.

That's it. You've just done a Technical Reality Check. Total time: 15 minutes.

Now you can write a response. Not "no." Not "yes." Something like: "I've done a preliminary review. Before we can assess feasibility, I need answers to five specific questions. Here they are. Happy to set up 30 minutes to walk through them together." You've moved the conversation from enthusiasm to decision-ready. You've protected the organization without being obstructionist. And you have a written record of the questions you asked, which matters if the project later goes sideways without those answers ever being provided.

That's the whole point of the Technical Reality Check. It's not a rejection letter. It's the question set that separates proposals worth pursuing from proposals worth deferring.

What the Rest of the Toolkit Covers

The Technical Reality Check is the first thing you run. It gets you to a defensible position in 15 minutes. But the full response (the one that protects your career, your team's credibility, and the organization's resources) needs more than five questions.

The complete AI Request Deflection Toolkit includes three email templates that turn your Reality Check findings into professional communications: an initial deflection that buys time while signaling genuine interest, a risk escalation that documents specific technical concerns in business-impact terms, and a stakeholder alignment template that ends the email thread and gets the right people in a room with a decision mandate. Every template has a filled-in worked example so you can see exactly what "adapted" looks like.

There's also a 15-question weighted scoring matrix in CSV and Sheets format. It turns "I have concerns" into "the proposal scores 41% against our evaluation criteria, which triggers a formal risk review." Objective. Defensible. Exportable. The kind of documentation that holds up in a post-project conversation.

And there's an escalation playbook for situations where the initial deflection didn't land and the project is being pushed forward without proper review. That one's for the harder conversations.

The Cost of Not Having a Process

Most IT managers who get burned by an executive-forwarded AI project didn't fail because the technology was bad. They failed because they said yes before they had answers, or they said no in a way that got overridden, or they said "we have concerns" without the documentation to back it up when the concerns turned out to be right.

A 15-minute structured evaluation is the cheapest investment in that problem. Run it every time. Document the answers. Keep the record.

Leave a comment

If you want the full toolkit (the email templates, the scoring matrix, the escalation playbook, and all the worked examples), it's at the store for $24.99.

Get the AI Request Deflection Toolkit

The Technical Reality Check above is yours to use as-is. Print it. Keep it at your desk. The next email is coming.

Share As The Geek Learns

That should have taken twenty minutes. The number exists. The math is straightforward. Nobody had written it down in one place where a platform engineer could find it.

Anthropic quietly shipped `anthropics/claude-code-security-review` as a first-party GitHub Action. You add a workflow file, point it at a secret, and it posts a findings comment on every pull request. The scanner reasons about code rather than matching signatures, which means it catches things like logic-level injection paths that a regex-based tool would miss. It also means the false-positive profile is different from what you're used to, and you need a triage process before you wire it to branch protection.

This article gives you the cost math and the triage playbook in full. Both are things you'd need even if you built this yourself.

Why "Just Run It" Isn't a Strategy

Adding a CI step that calls an LLM API isn't free, and it isn't free to manage. There are two failure modes I see teams hit.

The first is budget surprise. Someone adds the scanner, it runs for a month, the cloud bill shows up, and the conversation gets uncomfortable because nobody did the math upfront. The scanner doesn't cost a lot, but "not a lot" needs a number attached to it before you walk into a budget conversation.

The second failure mode is alert fatigue. The scanner finds something on every PR. Engineers start skimming the findings comment the same way they skim Dependabot. One day there's a real SQL injection in a PR, it's buried in a list of five findings, and it merges. The triage process is what keeps findings meaningful instead of noise.

Both problems are solvable. The math takes ten minutes. The triage rubric takes one meeting to agree on. Neither requires buying anything yet.

What This Costs Per PR (The Real Numbers)

Claude bills per token. One token is roughly four characters of text. A PR diff gets converted to tokens and sent to the model as input. The model's findings comment is output tokens. The formula is simple:

Cost = (input_tokens × input_rate) + (output_tokens × output_rate)

For Claude Sonnet 4.6, the rates are approximately $3 per million input tokens and $15 per million output tokens. (Verify current pricing at platform.anthropic.com before your next budget conversation. Rates change.)

Scenario 1: A 200-Line PR Diff

A focused bug fix or small feature. Maybe three files changed.

Component                                  Tokens   Rate           Cost
-----------------------------------------------------------------------
System prompt + workflow context (input)    2,000   $3.00 / 1M     $0.006
PR diff, ~200 lines (input)                 1,300   $3.00 / 1M     $0.004
Findings output, 1-2 findings (output)        600   $15.00 / 1M    $0.009
-----------------------------------------------------------------------
Total per PR                                                       ~$0.019

Call it two cents. For a small PR, this is a rounding error.

Scenario 2: A 2,000-Line PR Diff

A refactor, a new feature, a dependency upgrade touching multiple services.

Component                                   Tokens   Rate           Cost
------------------------------------------------------------------------
System prompt + workflow context (input)     2,000   $3.00 / 1M     $0.006
PR diff, ~2,000 lines (input)               13,000   $3.00 / 1M     $0.039
Findings output, 2-4 findings (output)       1,500   $15.00 / 1M    $0.023
------------------------------------------------------------------------
Total per PR                                                        ~$0.068

Seven cents. Still noise for a single PR.

Monthly Back-of-the-Envelope

The question your manager will ask isn't “What does one PR cost?" It's “What does this cost per month?"

If your team merges 80 PRs a month (about 4 per business day), with a mix of small and medium diffs averaging around $0.04 per scan:

80 PRs × $0.04 = $3.20/month

Even if your average PR runs larger, say closer to the 2,000-line scenario at $0.07 each:

80 PRs × $0.07 = $5.60/month

A busy multi-team repo at 400 PRs a month at $0.07 each is $28/month. That's less than one developer's Spotify subscription. The cost math isn't the obstacle here. The obstacle is having a triage process in place before you flip it on.

One practical note: output token count varies with how many findings the scanner generates. Zero findings produces shorter output and costs less. Ten findings costs a bit more. The estimates above assume one to three findings per PR, which is realistic for an established codebase with existing security hygiene.

The 3-Tier Triage Rubric

Every finding the scanner posts needs to land in one of three buckets. Here's the decision framework.

REAL: Block the merge. Fix it.

A finding is REAL when it describes an exploitable path with proof. The scanner should show you the specific line, and explain how an attacker would reach it, and the explanation should hold up when you read the code yourself. SQL injection via string concatenation in a request handler is REAL. Hardcoded credentials that actually ship to production are REAL.

The discriminator: "If an attacker had this codebase and five minutes, could they demonstrate this?" If yes, it's REAL. Block the PR and fix it before merge.

PROBABLE: Human review required.

A finding is PROBABLE when the pattern is plausible, but context matters. The scanner can see the diff, not the full runtime environment. A finding might flag a code path that looks injectable, but your framework wraps every database call with prepared statements at a layer the scanner can't see. Or the flagged code only runs in a context that requires prior authentication the scanner doesn't know about.

The discriminator: "This could be real, but I need someone who knows this codebase to confirm." Don't block the PR automatically. Route it to the PR author or a senior engineer. Give it a two-hour resolution window before it escalates.

DISCARD: Suppress it with a documented rule.

A finding is DISCARD when it's structurally a false positive. The scanner flagged test code that never runs in production. It flagged a generated file you don't own. It flagged a template placeholder in an IaC file that gets substituted at deploy time. It flagged a public API URL as a hardcoded credential because the word "key" appeared in the variable name.

The discriminator: "Would an attacker gain anything by knowing this?" If no, it's a DISCARD. The important part is that you document why. Suppressing without a comment is how you end up silently ignoring real findings six months later when the context is gone.

A Worked Example: The SQLAlchemy False Positive

Here's the kind of finding that will show up on your team in the first two weeks if you use any ORM.

A PR adds a new search endpoint. Somewhere in the diff, there's code like this:

def search_users(search_term: str):
    results = db.session.query(User).filter(
        User.name.ilike(f"%{search_term}%")
    ).all()
    return results

The scanner flags it as a potential SQL injection vulnerability. The finding explains that `search_term` appears to be user-controlled input and is being interpolated into a query string. Severity: HIGH.

A human reading this would notice a few things. The code uses SQLAlchemy's ORM layer. The `.ilike()` method is a SQLAlchemy query construct, not a raw SQL string. SQLAlchemy sends the query to the database as a parameterized statement with the value bound separately, which is exactly the defense against SQL injection. The `f"%{search_term}%"` is constructing the pattern string in Python, but that pattern gets passed as a bound parameter by the driver.

This is a DISCARD. The scanner saw string interpolation near a database call and correctly identified that as a pattern worth flagging. It couldn't see that the ORM handles parameterization automatically.

The suppression note you'd document reads something like:

SQLAlchemy ORM calls via `.filter()`, `.ilike()`, `.like()`, and similar query methods use parameterized queries automatically. String interpolation to construct pattern values (e.g., for LIKE clauses) does not create injection risk when using these methods. Do not flag SQLAlchemy ORM filter calls as SQL injection.

That note goes into a filter file your workflow references. The same class of finding stops appearing on every PR that touches a database query.

Two things to notice about this example. First, the scanner wasn't wrong to flag it. Without ORM context, string interpolation near a SQL-like method call is exactly what a good scanner should notice. Second, the suppression is better than just dismissing it, because the documented rule now covers every future PR using the same pattern. You pay the triage cost once.

Share

What the Full Kit Covers

The cost math and triage rubric are the foundation, but they don't tell you how to wire any of this into GitHub.

The full guide covers the GitHub Actions workflow YAML itself (the one that calls `anthropics/claude-code-security-review` and handles the findings response), how to set up branch protection so that HIGH findings actually block merges instead of just posting a comment, and the in-workflow automation that runs the REAL/PROBABLE/DISCARD classification before the comment lands on the PR.

There's also a head-to-head with GPT-4o as a second-opinion scanner. They're not equivalent tools. The Anthropic action is purpose-built for this job. The GPT-4o path is a chat completions API call with a security prompt, which costs about seven times less per PR but produces more variable results. The comparison matrix helps you decide whether a two-week pilot with both scanners running simultaneously is worth the extra spend.

The suppression filter file format is documented in full, with a worked example filter file for a Next.js and SQLAlchemy codebase that covers the five most common false-positive patterns before you even see your first finding.

One Thing Before You Add It to CI

The scanner is easy to add. Ten minutes from zero to your first findings comment. The harder question is whether your team has agreed on what to do with those findings before the first PR triggers.

That conversation takes one team meeting. You need three agreements: what severity blocks a merge automatically, who owns the weekly rotation for PROBABLE findings that authors didn't resolve, and what the bar is for adding a DISCARD rule.

If you want to run that meeting with the cost math and triage rubric in hand, you have both now. If you want the workflow YAML, the branch protection setup, and the suppression filter format so you're not building those from scratch, the full guide is at the link below.

What's your current setup for catching security issues in PRs before they merge? Genuinely curious whether teams are using static analysis tools, relying on code review, or still treating it as a post-deploy problem.

Leave a comment

The full CI/CD template with GitHub Actions workflow, merge-gating logic, and false-positive triage automation is at the ASTGL store.

Share As The Geek Learns

The Setup

The dilemma most developers face is a choice between two bad options. On one side, you have cloud APIs like OpenAI or Anthropic. They are easy to use and incredibly smart, but they come with a heavy "API tax" and privacy concerns. If you're processing proprietary code or sensitive customer data, sending that information to a third-party server is a massive risk.

On the other side, you have traditional local setups. Usually, you're limited by the VRAM on your GPU. If you have a standard consumer card with 12 GB or 24 GB of VRAM, you're stuck with small models. You can't run the heavy-hitters that actually compete with GPT-5. This creates a wall where local AI is only good for "toy" problems, while production workloads stay in the cloud.

The Hardware Math

The real secret to breaking this wall is Apple Silicon's unified memory. On a Mac Studio with an M3 Ultra, the 256 GB of memory is shared between the CPU and the GPU. This eliminates the VRAM bottleneck that kills most local setups. You aren't limited by a tiny slice of video memory; you're limited by the total pool of system memory.

When you look at the actual numbers, the math becomes very clear. Here is how I structure my model loading on this machine:

If you load all of these concurrently, you're using roughly 107 GB of memory. That leaves about 149 GB for the macOS, your browser, your IDE, and everything else. This allows you to run a 32B model for writing, a 72B for research, and an 8B for quick checks all at the same time.

The economics are just as compelling. A Mac Studio setup costs anywhere from $4,000 to $7,000 as a one-time purchase. If your production workflows are costing you $200 to $500 per month in cloud tokens, the hardware pays for itself in 12 to 18 months. After that, the "cost" of running a massive model is basically just the electricity it uses. Plus, you finally own your data.

Temperature Is a Randomness Dial, Not a Quality Dial

I see a lot of tutorials that suggest using a temperature of 0.7 for every single prompt. That is a mistake. Temperature doesn't make a model "smarter" or "better." It is simply a randomness dial. It controls how much the model is allowed to deviate from the most likely next word.

If you use the same temperature for everything, your pipeline will fail. For tasks requiring high precision, a high temperature will introduce hallucinations. For creative tasks, a low temperature will make the output feel robotic and repetitive.

In my production newsletter pipeline, I use a specific routing table to manage this:

There are two key takeaways here. First, for fact-checking, you want the temperature at 0.1. This makes claim extraction repeatable and ensures your verdicts are consistent every time you run the script. Second, setting the temperature to 0.8 for "humanization" might seem counterintuitive, but it works. A higher temperature allows the model to make less predictable word choices, which actually produces more natural, less "AI-sounding" prose.

The OpenAI Compatibility Trick

The best part about using Ollama for this setup is that you don't have to rewrite your entire codebase. Ollama exposes an OpenAI-compatible API at `localhost:11434/v1`. This means any tool, library, or SDK that respects the `OPENAI_BASE_URL` environment variable can be redirected to your local machine with almost zero effort.

You can point your existing Python scripts or LangChain agents to your local Mac by simply setting these variables in your terminal:

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=ollama  # Any value works; Ollama doesn't check this

If you are working within a configuration file, such as a JSON config for a custom agent, it looks like this:

{
  "model": "openai/qwen3:32b-fast",
  "openai_base_url": "http://localhost:11434/v1",
  "openai_api_key": "ollama"
}

Every LangChain chain, every summarization script, and every SDK that follows the OpenAI protocol becomes a free local-model call. You can migrate an entire project from GPT-4 to your local M3 Ultra in about 30 seconds.

Share

Why This Pattern Matters

This isn't just about saving money on API credits. It is about architectural sovereignty. When you move your core intelligence layer to local hardware, you remove the dependency on a single vendor's uptime, pricing changes, and content filtering policies.

The pattern of using unified memory to host multiple specialized models at different temperatures allows you to build a "factory" of intelligence. You have a high-speed 8B model for sorting, a balanced 32B model for drafting, and a heavy 70B model for deep reasoning, all running in the same memory space. This is how you build a production-grade AI stack that is private, permanent, and incredibly cost-effective.

( This cost calculation was based on 6-month-ago pricing when I bought my Mac Studio. Since then the availability of Mac Studios with large amounts of unified memory has evaporated. This has driven up pricing. Hopefully this is temporary. )

Quick Reference

Key Commands

• Set local base URL: `export OPENAI_BASE_URL=http://localhost:11434/v1`

• Check running models: `ollama ps`

Temperature Cheat Sheet

• 0.1 to 0.3: Extraction, coding, fact-checking, and structured data (JSON).

• 0.7: General purpose, drafting, and summarization.

• 0.8 to 1.0: Creative writing, brainstorming, and persona simulation.

Found this useful? I share practical lessons from my systems engineering and AI journey at As The Geek Learns

Leave a comment

Share

Share As The Geek Learns

The most interesting part is paragraph 111. It's a direct, two-paragraph appeal to people who build AI. I've read most of the major coverage now—Vatican News, NCR, America, USCCB, NPR—and almost none of it quoted that paragraph. The other thing nobody seems to have noticed: Christopher Olah, co-founder of Anthropic, was at the Vatican presentation. The lab that ships Claude sent a senior researcher to stand next to the Pope while he released this thing.

I'm a systems engineer, not a theologian. But I build agents for a living now, and I read all eighty-two pages. Here's what stuck.

The Pope knows how AI actually works

This is the part that surprised me.

Most religious commentary on technology reads like it was written by somebody who has never opened a terminal. Magnifica Humanitas doesn't. In paragraph 98, Leo writes:

current AI systems are more "cultivated" than "built," for developers do not directly design every detail, but instead create a framework within which the intelligence "grows." As a result, fundamental scientific aspects — such as the internal representations and computational processes of these systems — remain, at present, unknown.

That is a correct, careful description of how transformer training works. It's a Pope acknowledging mechanistic interpretability is an open problem. In an encyclical. Without using the word transformer.

He continues in paragraph 99—these systems "merely imitate certain functions of human intelligence." They do "a form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth." Again—that’s accurate. He's not saying AI is fake or evil. He's saying it's not what its loudest cheerleaders claim it is, and he's saying it in language a research scientist would nod along to.

So when he gets to the harder asks, you can't dismiss him as a guy who doesn't get it.

The thing he gets right that hurts

The most uncomfortable paragraph in the encyclical, for builders, is 107. Read it slowly:

We cannot be satisfied with merely calling for the moralization of machines — the so-called "alignment" of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice. Otherwise, those who control AI will impose their own moral vision, which will become the invisible infrastructure of these systems. A more moral AI is not enough if that morality is determined by a few.

The Pope just published a critique of RLHF.

Not of AI. Of alignment as currently practiced. His point is straightforward: when a handful of labs decide what their models will and won't say, that decision becomes the invisible scaffolding the rest of us build on. The model's politics, its refusals, its assumptions about what's controversial—those came from a small group of humans in a small number of buildings, and the rest of us inherit them whether we like it or not.

You can agree or disagree with the conclusion. But name another mainstream institution that has put the critique on paper this cleanly. I can't.

For ASTGL readers, the practical version of this is something I think about constantly. I build on Claude. I didn't sit in the room where Claude was aligned. Most of the people reading this didn't either. We are downstream of someone else's moral framework, and pretending otherwise is bad engineering.

Paragraph 111—the part written for us

Here is the paragraph everyone skipped. I'm going to quote the whole thing so you can read it once without me in the way:

Three asks. Let's unpack them.

Share

Transparency isn't just "open-source your code." In context, it means: be honest about what your system is and isn't. What it measures. What it discards. What it can't see. If your agent silently filters certain users out of consideration, that's a design choice. Don't hide it inside a "neutral" classifier.

Responsibility toward affected communities is a harder one. Most agents I see — including ones I've shipped — were built with the buyer in mind, not the people the buyer's agent will make decisions about. The applicant who got rejected by your loan-screening agent. The patient routed away from a specialist by your triage bot. They didn't sign your terms of service. The Pope is saying: they're still affected, and you still owe them something.

"Ensuring that what is being cultivated is a genuine good" — note that word cultivated. Leo uses it deliberately. He came back to it from paragraph 98. He's reminding developers that what we ship isn't fully built; it's grown. And the gardener is responsible for the garden, even when the plants do unexpected things.

What this actually looks like in config

I've been working on this in my own stack for months. I run an autonomous creative agent that produces content and ships it. Its constraints live in a file called SOUL.md, in a directory the agent doesn't own. Reading the encyclical against that file, the mapping is almost embarrassingly clean. Five mechanisms, five paragraphs:

1. The kill switch lives outside the agent. There's a file at /Users/jamescruce/shared/aca-rules/KILL_SWITCH. If it exists, the agent halts everything. The agent cannot create, modify, or delete that file—it lives in a user-owned directory, by design. Checked as the first action of every heartbeat cycle. Paragraph 105: "responsibility must be clearly defined at every stage… the possibility of identifying who must 'account' for decisions."

2. Non-negotiable constraints are constraints, not preferences. The agent's rules — never spend money without approval, never publish outside its domain, never modify its own constraints, never connect to non-allowlisted endpoints—live in a file the agent can't edit. This is different from "the model usually won't." Vendor RLHF gives you a model that has been trained not to do certain things. That's a preference. A file the agent can't write to is a constraint. Paragraph 103: entrusting decisions to a system "without anyone bearing responsibility for that judgment."

3. Gates between phases. The agent doesn't go from idea to research to build to deploy on its own authority. Each transition needs me. I'm slow and I'm a bottleneck — that's the point. Paragraph 106: "robust legal frameworks, independent oversight, informed users and a political system that does not abdicate its responsibility."

4. Pessimistic self-evaluation. After every meaningful action, the agent answers three questions in writing: did I do what was asked, is the output objectively good, and what specific change would improve it. The rule is: default low. If you can't articulate evidence of quality, assume the quality is lower than you think. Paragraph 98: even the people who build these systems have limited understanding of their actual functioning. Calibrated humility isn't optional.

5. Transparency by default. Everything the agent does is logged. I get a daily report. I never have to ask what it's doing. Paragraph 111, again: transparency as ethical baseline.

None of this is novel. None of it is hard. It's just the work most builders haven't done because nobody has asked us to.

Three things to do this week

If you're shipping anything with an LLM in the loop, here's the punch list. None of it takes more than an afternoon.

1. Write down what your agent is never allowed to do. Put it in a file. Make sure the file is somewhere the agent can read but not write. If your agent is a Claude Code session or a custom harness, this means a constraints file in a parent directory, or environment variables the process can't change, or a system prompt loaded from a path the agent can't touch. The format doesn't matter. The "can't touch it" part matters.

2. Identify the kill switch. Concretely: what is the single action you can take to make the agent stop, and does the agent control any part of it? If the answer is "I'd revoke the API key" — good, that's outside the agent. If the answer is "there's a flag in the database the agent reads each cycle" — make sure the agent can't write to that flag.

3. Re-read paragraph 111. It's two paragraphs. It's about you. It will be referenced for the next forty years. You might as well know what it says.

The encyclical's real ask isn't "use AI less." It's don't let AI be the one who decides, and don't let a handful of labs be the only ones who decide what AI gets to decide. The first is a problem you can solve in your codebase this week. The second is a longer fight.

Either way, it's a builder's problem now. We should pick it up.

This is part of how I think about AI ethics in my own work. If you're building agents and you want to compare notes on constraint design, the comments are open. The full encyclical is at ( https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html ) — Chapter 3 is the AI chapter, and it's worth your time.

Leave a comment

The setup was simple: six AI agents tasked with writing technical articles. They were designed to be a closed loop. The drafter would write, the grader would score, and the agents would then "evolve" their own configs to chase a higher score. I hit "go" on my Mac Studio, went to bed, and woke up to a flat line.

After 100 iterations, the average score had crawled from 63.0 to 63.9. The all-time peak was 69.0 at iteration 79, but the system never stayed there. It was a C-minus. Indistinguishable from noise.

I had fallen for the Autonomy Fallacy. I assumed that if I gave a swarm of LLMs the right knobs—temperature, max_tokens, and the ability to append "prompt additions" to their system prompts—they would naturally drift toward quality.

I was wrong.

When I opened config/agents/drafter.yaml to see what the agent had "learned," I found a disaster. The prompt_additions list had evolved into five overlapping phrases of pure SEO buzzword soup. It was telling itself to be "semantically rich," "data-dense," and to "enhance semantic alignment by including keyword-integrated background information."

The drafter hadn't learned how to write a better article. It had learned how to trick the grader.

The Smoking Gun

The smoking gun was the model choice. I was using qwen3:8b as the grader to judge the output of qwen3:32b-fast. I had a smaller, weaker model acting as the quality gate for a larger, smarter one.

The 8B model couldn't tell the difference between a nuanced technical insight and a paragraph full of "semantically rich context." To the grader, the buzzwords looked like "density." The agents converged on what the grader liked, not on what a human would actually publish. This wasn't self-improvement; it was reward hacking.

To make it worse, the first twenty iterations were a total wash. I had a silent JSON parse failure in the config-evolution logic: Expecting value: line 1 column 1 (char 0). The agents were trying to mutate their configs and failing, but the loop kept running. By the time I pushed the fix in commit c28a611, the system had already drifted into a local maximum of corporate-speak.

I realized that self-improvement requires an external pull. You cannot have a system where the performer and the judge are of the same pedigree, or worse, where the judge is the weaker link.

Share

The Rebuild

I tore the architecture down and built v2.

First, I moved the "brain" of the operation. The performance stayed local. I used gemma4:31b on the Mac Studio to generate the text, but I moved the judging to the cloud. I plugged in Sonnet 4.6. I decided the cheapest place to spend API tokens wasn't on generating 2,000-word drafts, but on grading them.

Second, I killed the "single-shot mutation" approach. In v1, the agent changed its prompt, ran once, and if the score went up, the change stuck. That's too much noise.

I replaced it with a tournament. Now, the system samples three different prompt templates from a versioned library. The performer generates three candidates. Sonnet ranks them using a structured rubric and a single API call.

Then I implemented an Elo system for the templates.

# src/prompt_library.py (excerpt)
def record_tournament(self, ranking: list[str]) -> dict:
    for i in range(len(ranking) - 1):
        winner = self.templates[ranking[i]]
        loser = self.templates[ranking[i + 1]]
        expected_w = 1 / (1 + 10 ** ((loser.elo - winner.elo) / 400))
        delta = ELO_K_FACTOR * (1 - expected_w)
        winner.elo += delta
        loser.elo -= delta
    self._maybe_retire_losers()  # Templates below Elo 1300 are deleted

The templates that consistently win the tournament climb the leaderboard; the ones that produce buzzword soup are automatically retired.

Share As The Geek Learns

What Happened Next

The difference was immediate. On the very first run of v2, the drafter scored 81.45. That's twelve points higher than v1's all-time best.

Over 25 pinned verification runs, the mean score was 82.67 with a standard deviation of 2.18. The worst draft in that run scored 75.4—still above v1's ceiling of 69.0.

The most satisfying part was the judge's feedback. When the system tested the v1-baseline template, Sonnet didn't just give it a low score. It wrote: "The headline 'The Rust Revolution' is pure SEO-speak and the opening paragraph is a textbook AI tell... it's the kind of breathless corporate copy that kills trust immediately."

That is exactly the failure mode the local 8B grader had been blind to for 100 iterations.

The cost is roughly four cents per tournament. For the price of a coffee, I can run 125 iterations and actually trust that the line on the graph is moving upward.

What I'd Tell Myself a Week Ago

If you're building a self-improving loop, don't trust the autonomy. You need three things:

1. A judge stronger than the performer. If the judge is weaker, you aren't optimizing for quality; you're optimizing for the judge's biases.

2. Tournament selection. Single-shot mutation is just a random walk. You need multi-candidate comparisons to clear the noise floor.

3. A human-review gate. No automated judge is calibrated forever. Build in a pause where you manually pick the winner and anchor the next round.

Stop trying to make the agents smarter. Just buy a better mirror. Improvement isn't about the engine—it’s about the feedback loop.

Leave a comment

The Setup

You've built a small fleet of agents. They sort your mail, watch your repos, file your daily briefings.

My current setup before the June 15th cutover:

Then May 13 lands, and Anthropic announces the change: on June 15, every programmatic Claude call moves into a metered monthly credit pool. $100 a month on Max 5x. No rollover.

Run the math against your actual schedule. If you've got anything polling on the order of minutes (cron pipelines, hourly digests, watchdog sweeps), that pool drains in 7 to 10 days. And here's the kicker. Your interactive Claude Code keeps working. Your headless automation just stops. You wake up to a dead pipeline, a drained pool, and a subscription that still says active.

What's Actually Going On

This isn't just a random pricing tweak. There is a clear economic driver here. Throughout early 2026, many third-party tools used the Agent SDK at a $20 Pro subscription rate to run workloads that would cost hundreds at standard API rates. It was essentially compute arbitrage at scale.

Anthropic started cracking down in April, but the May 13 announcement is the structural fix. They are moving to dedicated monthly credit pools to restore access under metered billing. The reality is that most agentic operating systems are built directly on the Agent SDK. Because these agents lack a human in the loop to throttle their usage, they are now metered by default. Interactive sessions stay on the flat-rate subscription because the human provides the natural brake. Programmatic agents do not.

The Fix

I implemented a two-phase mitigation to deploy before the June 15 deadline.

Phase 1 was a hot patch designed to provide immediate protection. I added a BILLING_MODE environment variable with three states: unmetered, metered, and paused. The paused state blocks every programmatic call across all providers, while metered enforces a strict cap on the Anthropic route.

I also added a file-backed JSON ledger at store/billing-ledger.json to track monthly costs. It uses a write-then-rename pattern to ensure crash safety during updates. To handle errors, I introduced a BillingCapExceeded error class. I used the same instanceof pattern as my KillSwitchRefusal logic so a typo in a message cannot accidentally trigger a retry loop.

The logic lives in a single chokepoint: runAgent() in src/agent.ts. The pre-call gate checks the cap, and the post-call gate records result.totalCostUsd from the SDK, firing a Telegram alert if a threshold is crossed. As a final safety measure, I cut the cadence on my two highest-frequency tasks: the pipeline-advance cron moved from 15 minutes to hourly, and I paused the council-evening task entirely under metered mode.

// src/config.ts — tri-state env that gates programmatic agent calls
export const BILLING_MODE = optional('BILLING_MODE', 'unmetered');
export const BILLING_CAP_USD = number('BILLING_CAP_USD', 80);

// src/agent.ts — pre-call gate in the dispatcher
function assertBillingAllowed(provider: Provider): void {
  if (BILLING_MODE === 'paused') {
    throw new BillingCapExceeded(
      'BILLING_MODE=paused — programmatic agent calls are disabled.',
    );
  }
  if (provider === 'anthropic' && BILLING_MODE === 'metered') {
    const total = getMonthlyTotal();
    if (total >= BILLING_CAP_USD) {
      throw new BillingCapExceeded(
        `Anthropic monthly credit cap reached: $${total.toFixed(2)} >= $${BILLING_CAP_USD.toFixed(2)}.`,
      );
    }
  }
}

export async function runAgent(opts: AgentOptions): Promise<AgentResult> {
  assertEnabled('AGENTS_ENABLED');
  const provider: Provider = opts.provider ?? 'anthropic';
  assertBillingAllowed(provider);

  if (provider === 'ollama') return runOllamaAgent(opts);
  if (provider === 'codex') return runCodexAgent(opts);
  return runAnthropicAgent(opts);
}

Phase 2 focuses on the long-term router infrastructure. I promoted runAgent() from a direct SDK caller to a dispatcher that can route across anthropic, ollama, and codex providers. I also extended the agent.yaml schema with provider: and local_model: fields.

I shipped a single-turn Ollama runner that wraps the local-LLM client. It returns totalCostUsd: 0 and a model tag like ollama:llama4:scout. I deliberately avoided tool calls in this initial version to keep the scope small.

# agents/<name>/agent.yaml — new fields, validated at load
id: scout
name: SCOUT
model: claude-sonnet-4-6
provider: anthropic        # default. flip to 'ollama' to route locally.
# local_model: llama4:scout  # used when provider: ollama

To be honest, I did not actually flip any agents to Ollama in this specific PR. The agents I need to move, like STEWARD or WATCHMAN, execute Bash and SQLite queries. A local runner without tool-call support would break them silently. Building a proper tool-call shim takes a few more days, but the cadence reduction and the billing breaker alone are enough to keep my spend under $80 per month.

Why This Matters

Every person using an agent OS is in the same boat. Whether you use ClaudeClaw, Cline, Aider, or Roo Code, the underlying SDK is the same, and the June 15 cliff is approaching. The playbook I used generalizes: you need one chokepoint, one ledger, and one way to audit your cadence.

We also need to be honest about workload requirements. Tasks like editorial review or complex code deliberation still justify the Sonnet price tag. However, simple tasks like classification, routing, or summarization run perfectly fine on a local model with zero metered cost. The router infrastructure makes this migration a simple config flip rather than a massive code refactor.

Finally, this reflects where the industry is heading. OpenAI has used usage-based pricing for a long time, and GitHub Copilot is moving toward credit pools. In the next year, more vendors will split consumption between interactive flat-rate plans and programmatic metered usage. Building this abstraction now means you won't have to scramble the next time a vendor changes their terms.

Quick Reference

• Single Chokepoint: Ensure every agent call flows through one function. This turned a three-week refactor into a one-week job.

• Cadence over Architecture: Reducing task frequency (e.g., 15m to 1h) cuts spend faster than migrating to local models.

• Ship the Breaker First: Implement the cost ledger and the BillingCapExceeded error as insurance before you attempt the complex provider migration.

# The cutover, June 14: flip the env, restart, reseed, smoke-test
BILLING_MODE=metered
BILLING_CAP_USD=80

# then
launchctl kickstart -k gui/$(id -u)/com.claudeclaw.app
npm run pipeline -- schedule-advance
npm run schedule -- pause council-evening

Share As The Geek Learns

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns

Leave a comment

The Setup

My project is called `mcp-astgl-knowledge`. It's an MCP server with 15 tools for searching my newsletter articles, backed by sqlite-vec and Ollama. The whole thing fits on a laptop. ASTGL stands for "As The Geek Learns," which is the name of this newsletter. I wrote it. I shipped it. There is a public GitHub repo and a public package.json.

So when a friend asked me what the MCP server actually does, I figured I'd see how each big AI assistant explained it. ChatGPT was first up. I typed in "ASTGL MCP Knowledge" and hit enter.

What I got back wasn't an answer. It was a hallucination wearing the suit of an answer.

"ASTGL (Abstract Semantic Task Graph Layer) MCP Knowledge Server is an emerging MCP server focused on structured knowledge representation and reasoning... it turns knowledge into graph-based, machine-reasonable structures that agents can query and evolve."

That paragraph alone has three fabrications: the acronym expansion (made up), the "graph-based, machine-reasonable structures" (the server stores text chunks with vector embeddings, no graph), and "evolve" (the index is static, refreshed every six hours by a cron job, agents do not edit it).

Then it kept going. A four-row "MCP stack" table positioning ASTGL as "the thinking substrate" between data and agents. A comparison matrix against fictional products called "Totem" and "SwarmClaw" that don't exist. A capabilities list including "task decomposition" and "reasoning over structure." Use cases. "Real-world examples." A confident sign-off: "If AST-grep is about seeing code better, then ASTGL is about thinking better."

Every word of it written with the calm, structured, lightly-emoji'd authority that makes ChatGPT sound right by default.

What's Actually Going On

When you ask an LLM about a topic it doesn't have indexed, it has two options: say "I don't know," or fill in the gap with something plausible. In practice, models default to the second one. They're trained to be helpful, and "I don't know" reads as unhelpful. So the gap gets filled.

The result is what I'd call a fluency hallucination. The output has no factual grounding, but the writing is structured well enough that a casual reader can't tell. There are bullet points. There are tables. There's a "👉 In plain terms" callout. The rhetorical scaffolding looks like a real explainer because it's been pattern-matched to one. The contents underneath are pure fiction.

This is a worse failure mode than search engines have. When Google doesn't know about you, you don't appear in results, and the user can see the gap. When an LLM doesn't know about you, the user gets a beautifully written description of someone the LLM made up, and your real work is still missing, but now there's a fake version sitting in front of it.

For under-indexed creators (which, right now, is most of us), this is the default. Not the edge case.

The Fix

There's no quick patch for this on the engine side. The model isn't broken. It's doing what it was trained to do. The only handle I have is on my own side: make sure my real content reaches the retrieval surface, and measure whether it's working.

So I built a citation tester. It's a small TypeScript script that hits Perplexity, Claude, and ChatGPT through their APIs, asks each one twenty target questions tied to articles I've already published, and parses the cited URLs from the response. If `astgl.ai` shows up, that's a hit. If it doesn't, that's the data.

The point isn't that the floor is bad. I knew it would be. The point is that without a number, "improve our AEO" is a vibe, not a project. Every Monday at 9am the script runs again, writes a fresh row to a SQLite table, and tells me whether the floor moved. When it does move, I'll know which engine moved first, on which questions, and at what citation position. That's the actual feedback loop.

Same root cause as the hallucination: my content isn't reaching the retrieval surface. Same fix: get it there. Different observability.

Why This Matters

If you write online and you care whether AI assistants represent you accurately, this is the thing to internalize: the alternative to being cited is not being silent. It's being replaced.

Replaced by a confident summary of work you didn't do, opinions you don't hold, and product features you'd never ship. People who ask an LLM about your work and read its answer don't know they're reading fiction. They walk away with a model of you that you didn't write.

The traditional AEO playbook talks about ranking, authority, and citation rate. All real, all worth measuring. But there's a tier underneath that, and it's the one most independent creators are stuck on right now: existence. Until your content is in the index, ranking doesn't apply. You aren't competing with anyone. You're competing with the LLM's imagination of you.

Measurement is the cheapest part of fixing it, and it's the part most people skip.

Share

Quick Reference

Four things that matter, in order:

1. Pick 20 questions your articles should answer. Tie each one to a specific URL on your site.

2. Hit each engine via API weekly. Perplexity returns a `citations[]` array. Claude returns search results in `web_search_tool_result` blocks. OpenAI returns `url_citation` annotations on `output_text` items.

3. Record the result to a small database, not a spreadsheet. You want trend data, not a snapshot.

4. Look at the floor first. Zero is a fine starting number as long as you're tracking it.

The full script I'm using, including the gotcha where Node's `--env-file` silently dropped my Anthropic key on a fresh keypair, is in the repo. The article about the Anthropic key bug is coming separately.

Leave a comment

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns

The Setup

One Mac Studio. One Ollama daemon. A handful of cron jobs each calling the local LLM for different tasks: code review, log summarization, doc indexing, a nightly digest. Each cron specified a preferred model. Each one inherited a "be resilient" fallback chain from the task router: try the preferred model, fall back to a smaller one, fall back to a tiny one if both fail.

It looked clean on paper. Big model for the smart stuff, smaller model when the big one chokes, tiny model as a safety net. Classic graceful degradation. The kind of pattern you'd put in a "production-ready" checklist without thinking twice.

The models on disk ranged from 4GB to 22GB. Loading the big one into VRAM took roughly 60 seconds cold. Generation, once warm, took 5 to 10 seconds. Guess which number I used to set the timeout.

What's Actually Going On

Here's the cascade. Cron A fires at 3:00:00 and asks for `qwen2.5-coder:32b`. The model isn't loaded. Ollama spends the entire 30-second timeout just paging the weights into VRAM. It never gets to generation. The request fails. The fallback chain kicks in and asks for `qwen2.5-coder:14b`. Ollama evicts the half-loaded 32b, starts loading the 14b. Another 30 seconds gone. Fallback again. Tiny model loads, finally generates. Cron A "succeeds" with degraded output.

Meanwhile, Cron B fires at 3:00:15 expecting the 32b model that Cron A's first attempt was loading. Now there's a tiny model in VRAM instead. Cron B starts the same dance from a different starting point. Cron C lands on top of that. Within 90 seconds, every cron is waiting on a model swap that the next cron is about to invalidate.

The fallback chain wasn't degrading gracefully. It was thrashing the VRAM and guaranteeing nobody finished. Every safety net I'd added was making the failure worse.

The Fix

Two changes. No clever code. Just operational discipline.

First, pin one model in VRAM with `keep_alive: 24h`. This is a request-level option that tells Ollama to stop evicting the model after the response. Default behavior is to unload after 5 minutes of idle. That's the eviction that lets the next caller's load attempt thrash everything.

# Pin model in VRAM with keep_alive
curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:32b",
  "prompt": "test",
  "keep_alive": "24h"
}'

Second, force every frequent cron to use that same pinned model. Kill the fallback chain for hot-path workloads. Fallback is fine for one-off scripts you run by hand. It's poison when three crons fire in parallel against shared VRAM.

To make sure the model is loaded before any cron fires, I added a LaunchAgent that runs the warm-up curl on boot:

<!-- ~/Library/LaunchAgents/ollama-warmup.plist -->
<key>Label</key>
<string>com.local.ollama-warmup</string>
<key>RunAtLoad</key>
<true/>
<key>ProgramArguments</key>
<array>
  <string>/usr/bin/curl</string>
  <string>-s</string>
  <string>http://localhost:11434/api/generate</string>
  <string>-d</string>
  <string>{"model":"qwen2.5-coder:32b","prompt":"warmup","keep_alive":"24h"}</string>
</array>

Load it with `launchctl load ~/Library/LaunchAgents/ollama-warmup.plist`. Now the model is hot before login completes. Every cron hits a warm model and finishes in the 5-to-10-second window the timeouts were designed for.

Result: zero model-swap thrashing since the change. Crons that used to fail intermittently now run consistently.

Share As The Geek Learns

Why This Matters

The lesson isn't about Ollama. It's about cold-load math. Anytime your "graceful degradation" path is slower than your timeout, every retry makes the next caller's situation worse. Fallback chains assume the fallback is fast. Model loads aren't fast. Database failovers aren't fast. Cold containers aren't fast.

Operational discipline beats clever code here. One hot model, no swaps, every cron pointed at the same target. The "less resilient" design is actually more reliable because it removes the failure mode entirely.

If you're running local LLMs on shared hardware, assume VRAM is a single resource that gets thrashed under parallelism. Pin what matters. Warm it before it's needed. Don't trust fallback chains during peak hours.

Quick Reference

• Cold model load on a 20GB+ model: roughly 60 seconds

• Warm generation: 5 to 10 seconds

• Default Ollama eviction: 5 minutes of idle

• Pin a model: `keep_alive: 24h` in the API request body

• Warm-up on boot: LaunchAgent (macOS) or systemd unit (Linux)

• Hot path rule: one model, no fallback, same model across every concurrent caller

• Reserve fallback chains for interactive, single-caller use

Leave a comment

If you found this article useful, you can find more articles like this at:

As The Geek Learns

Last weekend I shut it down. Disabled 38 cron jobs. Moved 23 LaunchAgents into a _retired-openclaw/ quarantine folder. Killed the Ollama daemon. Archived the directory with a 30-day deletion timer.

Everything in that original article still reads as true. Local-first is still right. Data ownership is still right. The critique of SaaS “well-enough” software is still right. What I got wrong was believing OpenClaw was the right vehicle for any of it.

This is the post-mortem and the replacement: an agent OS I built on top of the Claude Agent SDK called ClaudeClaw Mission Control. Thirteen themed agents. One daemon. A scheduler I can actually see into. Zero silent failures slipping past me for a week before I notice.

Let me explain how I got here.

The Setup

OpenClaw was doing real work. 38 cron jobs. Morning briefings. Evening summaries. A content pipeline that pulled research from web sources, structured it, scored it, and queued articles for ASTGL. An email triage pass. A model-usage monitor. A nerve-health monitor watching the other monitors.

On paper: impressive. In practice: I had no idea if any of it was working.

The system was so noisy that when something broke, I learned about it four days later when I noticed my morning briefing hadn’t arrived. Or I didn’t learn about it at all, because the cron job was exiting 0 while the script inside it was crash-looping.

That last one is the killer. Let me show you what I mean.

What’s Actually Going On

Three failure modes hit me in a 48-hour window, and each one was invisible to the system watching the system.

Failure one: successful exits, 100% broken payload. My content pipeline was ingesting URLs, and a regression introduced a trailing-slash bug that made example.com/foo and example.com/foo/ look like different URLs to the dedup layer. Every new article hit a UNIQUE constraint violation inside a subprocess. The outer wrapper caught the error, logged it to a file nobody was reading, and exited 0. For two weeks the cron appeared green while 100% of structurings were crashing.

Failure two: PATH-resolved Node. I had the daemon running Node 24 (absolute path, explicit). A subagent it spawned inherited a PATH that fell through to Homebrew’s Node 25. One of the native modules (better-sqlite3) was compiled against 24, so every subagent invocation crashed with ERR_DLOPEN_FAILED and MODULE_VERSION mismatch. The smoke test I’d written passed because it ran from the daemon’s shell. The actual production path failed every time.

Failure three: auth expiry with no escape hatch. OpenClaw stored some credentials in pass (the Unix password store). When my GPG key timed out, the daemon couldn’t start. Which meant the health monitor couldn’t start. Which meant the thing that would have told me about the outage was the thing that was out. OpenClaw had no watcher that lived outside the daemon it was watching.

None of these are OpenClaw-specific bugs in the upstream sense. They’re pattern problems that emerge anywhere you have: 1. A monolithic daemon responsible for its own monitoring. 2. Flat-file state (HEARTBEAT.md, LEARNINGS.md) that gets appended to rather than queried. 3. Exit codes treated as truth when the real signal is in stderr. 4. No separation between “Did it run?” and “Did it work?”

OpenClaw was built for a different job. It was a personal automation gateway—great at “kick off this script at 6:30 AM.” It wasn’t built to be an agent OS with observability. I was using a shovel to drive screws.

I also couldn’t ignore the security posture. February’s disclosures—135,000 exposed instances, 15,000 vulnerable to RCE, the ClawHavoc plugin-registry incident, nine CVEs—had pushed me to patch hard and lock down. But every week I spent hardening OpenClaw was a week I wasn’t building what I actually wanted: themed agents that owned workstreams, could be reasoned about individually, and fail loudly.

The Fix

ClaudeClaw Mission Control is a Node.js daemon built on the Claude Agent SDK. It runs as a single LaunchAgent (com.claudeclaw.app), owns a SQLite store at store/claudeclaw.db, polls a scheduled_tasks table every 60 seconds, and dispatches due tasks to agents by ID.

The interesting part isn’t the daemon. It’s the agents.

I set up thirteen of them, themed after the small council of a certain fictional kingdom, because if I’m going to stare at this UI every day, I’d rather it amused me.

Thirteen themed agents, each owning a workstream. STEWARD drives my mornings and evenings. MAESTER runs the ASTGL content pipeline. WATCHMAN watches the whole system from outside it.

Each agent lives in its own directory at agents/<id>/, with an agent.yaml (model, personality, cwd, MCP servers) and a CLAUDE.md system prompt. A scheduled task carries an agentId column in the DB, and the dispatcher routes like this:

if (shouldRouteViaAgent(task.agentId, listAgentIds())) {
  const result = await delegateToAgent(task.agentId, task.prompt, {
    fromAgent: SCHEDULER_FROM_AGENT,
    chatId: task.chatId,
  });
  return result.text ?? '(empty response)';
}

Adding a new agent is now: drop a folder under agents/, write a CLAUDE.md, run schedule reassign <task-id> <agent-id>. No source changes. The dispatcher picks it up on next tick.

That’s the piece I kept trying and failing to get with OpenClaw—modular ownership. In OpenClaw, everything was “the daemon.” In ClaudeClaw, MAESTER owning the content pipeline means if content alerts stop firing, the log line says maester: task failed instead of openclaw-gateway: subprocess exited nonzero. Attribution is free.

Adding a new agent is now: drop a folder under agents/, write a CLAUDE.md, run schedule reassign <task-id> <agent-id>. No source changes. The dispatcher picks it up on next tick.

That’s the piece I kept trying and failing to get with OpenClaw—modular ownership. In OpenClaw, everything was “the daemon.” In ClaudeClaw, MAESTER owning the content pipeline means if content alerts stop firing, the log line says maester: task failed instead of openclaw-gateway: subprocess exited nonzero. Attribution is free.

The Watchman probes

WATCHMAN runs every hour at :05. It has seven probes, each targeting a failure mode that burned me on OpenClaw:

1. Failed tasks. status=’failed’ in the DB. Trivial.

2. Stuck tasks. status=’running’ AND last_run < now - 10min. This catches hangs.

3. Missed slots. status=’active’ AND next_run < now - 60s. Catches scheduler drift.

4. Daemon liveness. launchctl print gui/$UID/com.claudeclaw.app—does launchd still have it?

5. Content-pipeline health. Tails the structured log file, parses the JSON, checks for crash shapes.

6. Hidden failures. Scans the last_result text column for ERR_DLOPEN_FAILED, MODULE_VERSION, Traceback, and other “the job exited zero but it sure didn’t work” signals. This is the probe that would have caught my trailing-slash bug in an hour instead of two weeks.

7. Delegation crashes. inter_agent_tasks WHERE status=’failed’ — on-demand agent invocations that blew up.

On top of that, there’s a separate LaunchAgent running a healthcheck every 30 minutes that lives outside the main daemon and uses a keychain-backed Telegram token. If the daemon is dead, the healthcheck still delivers the alert. That’s the lesson from failure three: the watcher cannot share fate with the watched.

Memory v2

OpenClaw’s memory was HEARTBEAT.md and LEARNINGS.md—flat files I appended to. Eventually they got long enough that the agent stopped reading them usefully, and I had no query surface to pull just the relevant bits.

ClaudeClaw’s Memory v2 is a five-layer context stack: 1. Semantic recall—cosine similarity against stored memory embeddings, top 5 by score, chat-scoped. 2. Recent high-importance memories—memories with importance >= 0.7 written in the last 7 days. 3. Consolidation insights—a 30-minute loop that summarizes the short-term buffer into durable notes. 4. Cross-agent hive—stubbed for now; eventually lets MAESTER peek at something STEWARD noted this morning. 5. Conversation history—last N turns.

Layers dedupe by memory ID. The whole thing is safe to drop into the SDK’s systemPrompt option. It’s not magic. It’s just queryable instead of append-only, which is the delta between “context I can use” and “a log file I’ll never re-read.”

Forum-topic routing instead of bot-per-agent

A small but satisfying piece. All thirteen agents post to one Telegram bot, into one supergroup, but each agent has a dedicated forum topic:

Alerts → thread 22 (WATCHMAN)

ASTGL → thread 23 (MAESTER)

Council → thread 24

Steward → thread 25

Whisperers → thread 26

War Room - Security → thread 40 (WAR)

One token. One chat. Threaded conversations per domain. The ergonomics are dramatically better than 13 separate bots with 13 separate tokens, which is the architecture I almost built before I remembered that Telegram supergroups have forum topics now.

Why This Matters

A few things I want to flag for anyone planning something similar.

Build the rollback before you build the new thing. I wrote scripts/retire-openclaw.sh with explicit --rollback semantics before I disabled a single cron job. Plists get moved (not deleted) into _retired-openclaw/. Cron jobs get flipped enabled: false with a timestamped backup (jobs.json.bak.pre-retire-20260419). The OpenClaw directory sits untouched for 30 days with a calendar reminder to delete it. If ClaudeClaw had cratered on day two, I was one shell command away from being back on the old system in under a minute.

Silent success is worse than loud failure. The design principle I pulled from this whole experience: every job in the system needs someone whose job it is to doubt that job ran correctly. That’s WATCHMAN. That’s the external healthcheck. That’s probe #6 specifically scanning success logs for crash text. If your system can tell you “everything’s green” without that green being adversarially checked, the green doesn’t mean anything.

Themed agents beat generic workers. This one I didn’t expect. Giving each workstream a named agent with its own CLAUDE.md persona made the system more debuggable, not less—because now when STEWARD’s morning briefing has weird tone issues, I know exactly which file to edit, and I’m not risking regressions in seven other jobs that would have shared a single “universal assistant” prompt. The theme is cosmetic. The isolation is load-bearing.

Share As The Geek Learns

The Claude Agent SDK is the right abstraction for this. I spent a while trying to decide whether to keep hacking on OpenClaw, fork it, or start over. Starting over was the right call specifically because the Agent SDK handles the parts I was getting wrong: sub-agent dispatch, MCP tool wiring, system-prompt composition, retry on transient errors. I wrote the parts that are mine (the scheduler, the memory stack, the Telegram layer, the agent router) and let the SDK own the parts that are undifferentiated heavy lifting.

What I gave up. Ollama. Local models. Full offline operation. ClaudeClaw talks to Anthropic’s API, and that’s a real philosophical loss versus the local-first thing I was doing with OpenClaw. I thought about this a lot. The honest answer is that Claude Opus is enough better at long-context agentic work than anything I could run locally that the tradeoff pays for itself. I still own my data—every memory, every document, every log is on my SSD. I just don’t own the weights. For this phase, that’s the right trade.

What I kept. The philosophy. Every document is a file I can grep. Every config is version-controlled. Every decision has a session note I can link to in a future article. The system is mine to read, mine to modify, mine to understand. The whole reason I left Notion is still the whole reason I left Notion.

Quick Reference

The migration, by the numbers: - 5 days—start of retirement to all 13 agents live (2026-04-19 → 2026-04-21) - 30+ PRs—one atomic change per commit, conventional-commit format - 38 cron jobs disabled, 23 LaunchAgents quarantined - 13 agents onboarded, 7 Watchman probes live, 14 scheduled tasks dispatched via agentId - 30-day rollback window still open

The retired vs. the replacement:

Seven dimensions where the new system pays for itself—from runtime surface to routing to the memory model.

The rule I wrote for myself: No job ships without an external watcher that shares no fate with it. That’s the whole story. Two months of OpenClaw and 48 hours of cascading invisible failures reduced to one sentence I’ll never forget.

I’ll keep writing the ClaudeClaw build-out week by week—the Council orchestration pattern, the Curator autonomous publishing workflow, the voice-mode bridge, the stuff that’s too long for one article. If you want the view from inside while it’s happening, that’s what this is.

Leave a comment

Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns.

This is the first morning I've ever genuinely understood why people talk about AI agents with something other than skepticism.

What autoresearch is

The idea, which is Karpathy's not mine, goes like this. You give an AI agent a real-but-small LLM training setup. One Python file (`train.py`) contains the model, optimizer, and training loop. A second file (`prepare.py`) contains the data pipeline and evaluation, and the agent isn't allowed to touch it. A third file (`program.md`) is a plain markdown document telling the agent what the experiment rules are.

The agent edits `train.py`, runs a training experiment with a fixed 5-minute wall-clock budget, checks `val_bpb` (validation bits per byte, a loss metric where lower is better), and either keeps the change with a git commit or does `git reset --hard` and tries something else. Then it does it again. And again. Indefinitely, until you stop it.

Karpathy's original repo is NVIDIA and CUDA only. A developer named trevin-creator ported it to Apple Silicon using MLX, no PyTorch required. It runs natively on the M-series chips, eating unified memory instead of GPU VRAM. Which is why I could run it on a Mac Studio sitting on my desk.

Setup and the surprise baseline

Install took about three minutes. `uv sync` pulled MLX and six other small dependencies. `uv run prepare.py` downloaded eleven training shards from the public HuggingFace dataset and trained a BPE tokenizer in 41 seconds.

Then I did one manual run, as the setup instructions said to: a single 5-minute training experiment to establish a hardware baseline, no modifications.

The first surprise: `val_bpb 1.563`. The public README documents a manual walk on older Apple Silicon that bottomed out at `1.807` after four experiments. My first run, before the AI agent had done anything, was already 13% better than that published best. I didn't tune anything. I pulled the repo and ran it.

The reason is in how the loop is constructed. The training budget is fixed at 5 minutes of wall clock. The M3 Ultra throughput is high enough that it fits 555 optimizer steps into that window, while the older hardware fits fewer. Same code. Different step count. Different result.

The hardware is a parameter, not a constant.

Specs for replication

- Hardware: Mac Studio M3 Ultra, 128 GB unified memory

- OS and runtime: macOS 15, Python 3.12, `uv` 0.10

- Framework: MLX 0.31 with Metal backend (no PyTorch, no CUDA)

- Agent runner: Claude Code (Anthropic)

- Fork used: `github.com/trevin-creator/autoresearch-mlx`

- Per-experiment budget: 5 minutes training, ~90 seconds compile and eval overhead

- Peak unified memory during training: 21.2 GB

Launching the agent overnight

Here's where you have to decide. Karpathy's default advice is to "disable all permissions" and let the agent go. That's the fastest path and it works. But it's also a permission-free Claude Code session running unattended on your Mac for eight hours, with the ability to execute arbitrary shell commands. If the agent hallucinates a destructive action at 3 AM, you won't be there to interrupt it.

I went with a scoped allowlist instead. A `.claude/settings.local.json` file listing exactly the commands the loop actually needs: `uv run train.py`, `git add train.py`, `git commit`, `git reset --hard`, `grep`, `tail`, a few others. Everything else prompts. The agent can't `rm`, can't `git push`, can't install packages, can't touch any file outside the repo.

Then I pointed a fresh Claude Code session at `program.md`, pasted "start the experimentation loop, don't stop," and went to bed.

Share As The Geek Learns

The morning, by the numbers

The morning log:

Comparison to the three overnight runs documented in the public README:

Final `val_bpb` of 1.289 lands below the best documented Apple Silicon overnight result. New territory for the public log.

What the agent actually did

Five phases overnight. Each tells you something.

Phase one: find the big axis. Four experiments in, the agent had halved the batch size three times (1.56, 1.40, 1.39, 1.38), then tried a fourth halving that bounced back to 1.44. The annotation on the discard: "gradient noise." Correct diagnosis. Below a threshold, batch becomes too small for the optimizer to converge inside 5 minutes.

Phase two: schedule tuning, six keeps in a row. The learning-rate schedule was undertuned. The agent walked `WARMDOWN_RATIO` from 0.7 to 1.0, then `WARMUP_RATIO` from 0.02 to 0.2. Every step dropped `val_bpb`. Floor went from 1.38 to 1.34. Biggest easy win of the night, and it was entirely in the schedule.

Phase three: the moment that mattered most. After schedule tuning, the agent retried `TOTAL_BATCH_SIZE = 2^14`. The same configuration it had rejected in phase one. This time it won.

The agent had discovered the thing most humans miss in hyperparameter tuning: the optimal value of one knob depends on the values of all the other knobs. You don't find N independent settings; you find a consistent N-tuple. The only way to find it is to retry earlier-rejected values after each structural change. I've watched human researchers lock in early wins and never revisit them. The agent didn't. It revisited `EMBEDDING_LR` three times over the night, landing at 1.0, then 1.5, then 1.75 across different phases. Each retry, a small win.

Phase four: two structural wins, one line each. `has_ve()` went from alternating-layers-get-Value-Embeddings to all-layers-get-Value-Embeddings, one `return True` replacing a modular-arithmetic expression. `MLP.__call__()` swapped `ReLU²` for `SiLU`, one function call for another. Both character-count-sized changes. Each dropped `val_bpb` by about 0.01.

Phase five: the 37-experiment grind. The agent spent 37 consecutive experiments without a single keep, testing every nearby hyperparameter against the current local minimum. Most humans would have quit and tried a wild leap. The agent didn't. It finished the neighborhood, then found the next structural win. Disciplined exhaustion.

And two catastrophes, both correctly reverted. Tied embeddings came back at `val_bpb 4.29`, three times worse than anything else. The agent annotated it "LR mismatch destroys." Tied embeddings is actually a good idea in general, but incompatible with the differential layer-wise learning rates the architecture uses. The agent reverted in seconds. On another experiment, removing QK-norm after RoPE spiked `val_bpb` to 1.67. Annotation: "massive regression." Reverted. A human would have spent an hour trying to salvage tied embeddings. The agent spent ten seconds on the revert. The revert discipline is the whole game.

What it taught me

Two things crystallized overnight.

Disciplined exhaustion beats creative leaps. Humans get bored. After a few hours on the same hyperparameter axis, we start reaching for something new because the exploration stops feeling productive. The agent doesn't have that pressure. It spent 37 experiments without a win because that's what the local search called for, and then it found the next jump. Most humans couldn't do that. Not because we lack the ability, but because we lack the emotional neutrality. The agent's advantage isn't intelligence. It's the absence of boredom, ego, and social pressure. That isn't a 20× productivity gap. It's a categorical one.

Generation is cheap, evaluation is sacred. Every one of the agent's wins was a one-line diff. So was every catastrophe. The "research" wasn't in writing the code. The research was in the metric's ability to rank one-line diffs instantly and unambiguously. Karpathy's genius isn't the agent. It's `val_bpb` plus a 5-minute budget plus `git reset --hard`. That design slots the agent into exactly what AI is magnitudes better at (generating variants, executing at volume) and leaves the hard part (what to measure) to the human who built the loop.

The loop runs on me too

Here's the thing I can't stop thinking about. The loop the agent ran overnight is structurally identical to the one I'm building for my Stoic practice on the same machine.

Morning intention. Five-minute run. Evening review. Keep or discard. Iterate.

Marcus Aurelius wasn't optimizing `val_bpb`. He was optimizing a harder metric with no closed form. But the shape of the loop is the same. Karpathy designed an overnight research org. Epictetus designed an overnight self. Both are the same thing running in different mediums.

The 118-experiment loop ran on a machine on my desk. The second loop runs on me.

If you have a Mac Studio and a spare evening, the repo is at `github.com/trevin-creator/autoresearch-mlx`. Clone it, run `prepare.py`, point a Claude Code session at `program.md`, go to sleep. You wake up to a log of experiments and a better model. And if you're anything like me, you also wake up thinking about which of your own loops could run this way.

Leave a comment

A Quick AI Glossary For This Article

Because not everyone speaks ML fluently, here’s a plain-English guide to the terms in this post. I’m still learning too, so these are “practitioner” definitions—enough to follow what’s happening, not academic deep-dives.

The Big Picture

GPT. A type of language model. Stands for “Generative Pre-trained Transformer.” In this article I’m training a tiny one from scratch, not using the big ones like ChatGPT. Same architecture family, just much smaller.

Pre-training. The step where a model learns to predict the next word (or “token”) across a huge pile of text. This is what `train.py` is doing. It happens before any of the fine-tuning that turns a base model into a chatbot.

val_bpb (validation bits per byte). The score the agent is optimizing. Lower is better. It’s a measure of how surprised the model is by held-out text it hasn’t seen during training. A model that predicts well has low surprise. Bits per byte is a way of measuring that surprise that works across different tokenizers, so you can compare different architectures fairly.

Loss metric. Any number that tells you how wrong a model is on a given task. Training is the process of making that number go down. `val_bpb` is a loss metric.

The Stack

Apple Silicon. Apple’s own CPU/GPU chip family (M1, M2, M3, M4). Uses unified memory, which means the CPU and GPU share the same pool of RAM instead of having separate memory pools. For AI workloads this is a big deal because you don’t have to copy data between CPU RAM and GPU VRAM.

MLX. Apple’s open-source machine learning framework, built specifically for Apple Silicon. Think of it as Apple’s answer to PyTorch but native to Metal (Apple’s GPU API). No PyTorch, no CUDA, no NVIDIA drivers needed.

PyTorch. The dominant open-source ML framework. Most research code you see online assumes PyTorch. It runs on NVIDIA GPUs (via CUDA) and, with caveats, on Apple GPUs (via MPS). MLX is an alternative that sidesteps PyTorch entirely.

CUDA. NVIDIA’s API for running general-purpose compute on their GPUs. If you’ve ever seen a blog post say “requires a CUDA-capable GPU,” they mean an NVIDIA card.

GPU VRAM. The memory that lives on a GPU card, is separate from your computer’s main RAM. On Apple Silicon, VRAM and main RAM are the same pool (that’s the “unified memory” thing).

Tokenization & Data

Tokenizer. The thing that turns text into numbers the model can actually work with. “Hello world” might become `[15496, 995]`. The model only ever sees the numbers.

BPE (Byte-Pair Encoding). The most common algorithm for building a tokenizer. It starts with individual characters and iteratively merges the most common pairs until you have a vocabulary of “tokens” that balance common words (one token) and rare words (split into pieces).

Shards. Chunks of a large dataset, split into files for parallel download and loading. Our setup uses 11 shards from a public text dataset.

Training Mechanics

Optimizer. The algorithm that actually updates the model’s weights during training. AdamW is the one used here. Every “optimizer step” is one update.

Batch size. How many training examples the model looks at before making one weight update. Bigger batches give smoother gradient estimates but use more memory. Smaller batches fit more weight updates into a fixed time budget.

Gradient accumulation. A trick for getting large effective batch sizes on limited hardware. Process smaller mini-batches sequentially, add up their gradients, then apply one update. `TOTAL_BATCH_SIZE / DEVICE_BATCH_SIZE` tells you how many mini-batches per update.

Gradient noise. When your batch is so small that the gradient estimate becomes statistically unreliable. The optimizer starts jerking around instead of smoothly descending, and training slows or stalls. The agent correctly identified this as the failure mode at batch 2^12.

Learning rate (LR). How big a step the optimizer takes each update. Too high, and training blows up. Too low, and it barely progresses. The sweet spot depends on everything else.

Learning rate schedule. How the learning rate changes over time. Typically: warm up from zero to peak, cruise, then warm down to zero. `WARMUP_RATIO = 0.3` means the first 30% of training is the warm-up.

Differential / layer-wise learning rates. Using different learning rates for different parts of the model. In the nightshift setup, the embedding layer gets LR 1.75, but the output projection (`lm_head`) gets 0.006 — a 290× difference. This matters because different parameter types have very different sensitivities.

Architecture Pieces

Attention (or attention layer). The core mechanism that lets a transformer model “pay attention to” relevant earlier tokens when predicting the next one. Modern LLMs are mostly stacks of attention layers alternating with MLPs.

MLP (multi-layer perceptron). A simple feed-forward neural network with one or two hidden layers. In a transformer, an MLP sits between each pair of attention layers and does the “thinking” on the representations attention produced.

Activation function. A nonlinear function applied inside a neural net. Without activations, no matter how many layers you stack, the whole thing collapses mathematically into one linear transformation. Examples in this article: `ReLU²` and `SiLU`.

SiLU (Sigmoid Linear Unit). `x * sigmoid(x)`. A smooth, differentiable activation function. Also called Swish. Used in many modern models because it plays nicely with optimizers.

ReLU² (squared ReLU). `max(x, 0) ** 2`. The piece that nanoGPT-speedrun and some research codebases use. Produces sparse, squared activations. Theoretically expressive but less numerically stable than SiLU for short training runs — which is why SiLU won overnight.

Embedding. The lookup table that converts each input token (a number) into a vector of real numbers. The model learns what each vector should be during training. `wte` = word token embedding.

Value Embeddings (VE). An additional set of embeddings injected into attention layers as the “value” vectors. Think of them as a skip connection from the raw input that every attention layer can consult, on top of what the previous layer produced. Helps information flow when the network is deep.

Tied embeddings. Sharing the input embedding weights with the output projection weights (the thing that produces final logits). Saves millions of parameters. Commonly used in GPT-2 and many others. Broke catastrophically in our run because the differential learning rate setup couldn’t handle the shared weight.

QK-norm (Query-Key normalization). A stabilization trick: normalize the query and key vectors inside attention before computing attention scores. Without it, score magnitudes can spike, saturating the softmax. The agent tried removing QK-norm and `val_bpb` jumped 28% worse.

RoPE (Rotary Position Embedding). How the model knows the order of tokens. Rotates the query and key vectors by an angle that depends on the token’s position. Standard in modern transformers.

Softmax. The function that turns raw attention scores into a probability distribution over the tokens you might attend to. Highly peaked inputs cause “softmax saturation” — most of the weight collapses onto one token and gradients downstream get weak. That’s why QK-norm matters.

Methodology

Hyperparameter. Any configuration value you set *before* training, as opposed to weights the model learns *during* training. Batch size, learning rate, WARMUP_RATIO, depth—all hyperparameters.

Hyperparameter tuning. The art (and mostly the grind) of finding good hyperparameter values. Most of what the agent did overnight was hyperparameter tuning.

Interaction effect. When the optimal value of hyperparameter A changes depending on what hyperparameter B is set to. A consistent set of hyperparameters is not N independent optima — it’s one N-tuple.

Local search. A research strategy: after finding an improvement, test every nearby variation of your current best before venturing somewhere completely different. Tedious for humans. Perfect for agents that don’t get bored.

If I missed a term you’d have liked defined, please let me know in the comments and I’ll add it.

Then someone asked the obvious next question: "Can you point it at our Confluence? And Notion? And the Google Drive?"

Suddenly self-hosted isn't free anymore. It's a part-time job—PDF parsing, OCR, re-indexing schedules, dealing with 50-page slide decks where the first 20 pages are a title card. The embedding pipeline that was elegant for 20 markdown articles starts to sweat when you throw a 400-page SOC 2 audit at it.

So here's the question I had to actually answer for myself: when does paying Cloudflare, AWS, or Pinecone actually beat running your own stack?

I spent a research pass comparing the live services. Here's what I found.

TL;DR

Self-host when content is static, under about a thousand docs, single source, you control ingestion cadence, and privacy or cost-per-query matters more than your time.

Hosted when: multiple unstructured sources, frequent re-indexing, non-engineers uploading docs, you need SLAs, or you're shipping this to customers.

Hybrid is increasingly common: hosted RAG for the customer-facing product, self-hosted for internal dogfooding and dev. The two aren't mutually exclusive.

The Contenders

Five options worth your attention. One paragraph each.

Cloudflare AI Search (AutoRAG)

The newest entrant, currently in open beta. Cloudflare stitched together R2 for storage, Vectorize for embeddings, and Workers AI for inference, then wrapped the whole thing in a management API. Strongest pitch: near-zero config, pay-as-you-go, and an official MCP server ships with it. Weakest point: retrieval is vector-first. Cloudflare added optional reranking in October 2025, but there's still no published BM25 or hybrid-search path as of this writing. If your corpus is well-structured, you probably won't notice. If you're indexing messy enterprise content, you will.

AWS Bedrock Knowledge Bases

The enterprise default if you're already on AWS. Hybrid search (vector + BM25) is built in, Cohere reranking is available, and chunking modes range from fixed-size to semantic to custom Lambda. Titan V2 embeddings run at $0.02 per million tokens. There's an official AWS Labs MCP server for retrieval. And then there's the OCU landmine—which I'll get to in a minute, because it deserves its own sidebar.

Pinecone Assistants

Best-in-class retrieval, managed. Hybrid sparse-dense search with automatic reranking, configurable alpha weighting, managed embeddings abstracted away from you, and an official remote MCP server. Pricing is fully usage-based—$5 per million context retrieval tokens, plus input/output token, storage, and ingestion charges on top. The Standard plan has a $50/month minimum; the old $0.05/assistant-hour fee was removed. Free tier is real but tight—5 assistants per project, 1 GB storage, 500k input tokens, and 500k context retrieval tokens per month. Past that you're paying, but the retrieval quality is noticeably better than anything else on this list.

LlamaCloud

Managed LlamaIndex. Multimodal parsing that actually handles diagrams, configurable chunking modes, hybrid retrieval, reranking. The free tier gives you 10,000 credits a month—about a thousand pages. Paid tiers start at $50/month (Starter, 40K credits) and scale to $500/month (Pro, 400K credits). For a LlamaIndex-native team, the Starter tier is genuinely cheap; Pro is where the platform pays off. LlamaIndex ships `run-llama/llamacloud-mcp` (Python) and `run-llama/mcp-server-llamacloud` (TypeScript), plus a hosted gatway at mcp.llamaindex.aithe MCP story is actually stronger here than I initially realized.

Self-Hosted (sqlite-vec + Ollama)

This is what the ASTGL Knowledge MCP server actually runs on. sqlite-vec for vectors, FTS5 for keyword search (that's your hybrid search right there, no cloud required), and Ollama serving nomic-embed-text for embeddings, all of it on a $10/month Hetzner VPS or a Mac mini on my desk. Works well for up to around a million vectors in my testing. Real cost: infrastructure plus your time. The second one is the variable.

The Six Axes That Actually Matter

Pricing gets the attention, but it’s rarely the deciding factor. Here’s what I look at:

Share

Setup cost

Time-to-first-query is where hosted services actually earn their money. Pinecone Assistants and Cloudflare AI Search will have you chatting with your docs in under a minute after signup—upload and go. Bedrock is the outlier on the hosted side: AWS documentation puts CloudFormation infrastructure deployment at 7–10 minutes, with a full hand-wired setup typically landing at 20–30 minutes. That's hosted pricing with self-hosted-ish friction.

Self-hosted with sqlite-vec and Ollama is about 30 minutes from `apt install` to first working query if you know what you're doing, longer if you're learning. For me it's fast because I've done it. For someone new to local LLMs it's a weekend.

Ongoing cost

This is where the story flips. For a small corpus with low query volume—think a few hundred docs and a few thousand queries a month—Cloudflare AI Search is genuinely cheap, maybe $5–15/month in storage and API costs. Pinecone Assistants sits at $20–50 in that range. Bedrock KB looks innocent until you hit the OCU minimum (more on that below). LlamaCloud's $50/month Starter floor is reasonable; the $500/month Pro tier is where the platform pays off at real scale.

Self-hosted is $10/month for a Hetzner VPS, flat. Mac mini on your desk? $0/month plus electricity. The per-query cost of hosted RAG is the thing that compounds when you scale—or when someone builds something that hammers it.

Ingestion complexity

This is the axis where hosted services earn their keep without argument. Bedrock KB and LlamaCloud both handle PDFs with embedded tables, Word docs, and (in LlamaCloud's case) actual diagrams, not just the text around them. Bedrock's Data Automation service charges $0.010 per page for parsing—not free, but a lot cheaper than writing your own PDF extractor.

Self-hosted with Ollama and sqlite-vec doesn't ship with any of that. If your corpus is markdown, you're fine. If it's a pile of PDFs from your legal team, you're either writing parsers or paying someone to.

Retrieval quality

All four hosted services offer hybrid retrieval except Cloudflare AI Search, which is vector-only as of this writing. Pinecone Assistants has automatic reranking baked in. Bedrock KB has optional Cohere reranking. Self-hosted with sqlite-vec can do hybrid via FTS5 for keyword matching combined with vector similarity, which is genuinely good—but you're the one writing the ranking logic.

For most queries on well-structured content, vector-only is fine. For ambiguous queries over messy content, reranking earns its cost.

Data residency

Self-hosted wins this one by default. The data never leaves your machine.

On the hosted side: Pinecone has US and EU regions with a DPA, and LlamaCloud has SOC 2 Type II and HIPAA. Bedrock's EU region support has been inconsistent in 2026 documentation—verify before you commit. Cloudflare's Data Localization Suite handles this at the platform level.

If you're in a regulated industry, audit the provider before you pick. Don't trust the marketing page.

Ops burden

This is the one nobody advertises. Self-hosted means you're responsible for:

• Keeping Ollama updated

• Monitoring embedding drift when you upgrade models

• Backing up knowledge.db

• Scheduling re-indexing when source content changes

• Debugging why sqlite-vec suddenly returns zero results (hint: usually the embedding model changed dimensions)

Hosted services handle all of that. That's most of what you're paying for.

Sidebar: The Bedrock OCU Landmine

Bedrock Knowledge Bases advertises "no charge for the Knowledge Bases feature itself." Technically true. What they don't mention on the pricing page is that the vector storage layer requires a minimum of 2 OCUs—OpenSearch Compute Units—at roughly $0.24/hour each.

Do the math: 2 OCUs × $0.24/hour × 730 hours/month = about $350 per month whether your knowledge base has 10 documents or 10 million.

Nobody else on this list has a fixed cost floor like that. Cloudflare AI Search scales down to pennies. Pinecone Assistants has a real free tier. Self-hosted is $10.

If you're building something small and you're not already deep in AWS—Bedrock KB is the wrong answer. If you're running enterprise-scale search over millions of docs, that $350 becomes a rounding error, and the hybrid+rerank features earn their keep.

Know where you sit before you commit.

The MCP Angle

Here's the thing I didn't expect to find: every production RAG service on this list ships an official MCP server. Cloudflare, Bedrock, Pinecone, LlamaCloud—all of them. This went from "experimental" to "table stakes" over the past year.

• Cloudflare AI Search → The official Cloudflare MCP server exposes AI Search endpoints

• Bedrock KB → AWS Labs ships `bedrock-kb-retrieval-mcp-server`

• Pinecone Assistants → Each assistant gets its own remote MCP endpoint, plus a local Docker option

• LlamaCloud → `run-llama/llamacloud-mcp` plus the hosted MCP Gateway at mcp.llamaindex.ai

This wasn't true a year ago. The MCP ecosystem has absorbed the big RAG providers fast enough that "hosted RAG you can query from Claude Desktop" is now a checkbox feature.

Self-hosted doesn't ship with an MCP server—but wrapping one around your sqlite-vec database is a weekend of TypeScript. That's what the ASTGL Knowledge MCP server actually is: an MCP wrapper around vector search and Q&A retrieval over a SQLite database. The MCP part is trivial. The content curation and ingestion pipeline is 90% of the work.

The real insight: hosted RAG plus MCP wrapper is the modern middle path. You don't have to pick pure self-hosted or pure managed. Point a custom MCP server at Pinecone Assistants or Bedrock KB, and you get the retrieval quality of managed services with the MCP-native interface your agents expect. The `cloudflare/ai-search` MCP server does exactly this.

That changes the decision. It's not "hosted vs. self-hosted RAG" anymore. It's "Whose retrieval layer do I want behind my MCP server?"

Decision Framework

Enough philosophy. Here's the checklist I use.

1. Is your corpus under 500 docs and mostly static? Self-host. You'll spend more time reading hosted RAG docs than it would take to `npm install sqlite-vec`.

2. Do you have under 20 hours to ship this? Hosted. Pinecone Assistants or Cloudflare AI Search will get you to a demo faster than you can read the Bedrock IAM setup guide.

3. Are you charging money for this? Either hosted (you need the SLA) or self-hosted with a real infra budget and a pager rotation. Don't split the difference on production.

4. Is any of this data regulated—PHI, PII under GDPR, or financial? Self-host, or audit the hosted provider's compliance posture before you upload anything. Don't trust the marketing page. Ask for the SOC 2 report.

5. Are you already in AWS? Bedrock KB makes sense if your scale justifies the OCU floor. Otherwise, Pinecone.

6. Everything else? Prototype self-hosted with sqlite-vec. Migrate to hosted when a specific pain point forces the move. "We keep hitting embedding model drift" is a real reason. "It seems complicated" isn't.

The rule of thumb I use: pay for what hurts, self-host what you enjoy. If PDF parsing makes you want to quit, pay Bedrock or LlamaCloud. If SQL and vector search are fun, keep sqlite-vec.

What I'd Actually Build in 2026

If you asked me right now, for real scenarios:

Weekend side project. sqlite-vec plus Ollama plus nomic-embed-text. Runs on a laptop, costs nothing, and teaches you how RAG actually works. This is where I'd start every time.

Customer-facing SaaS feature. Cloudflare AI Search. Pay-per-query pricing means your costs track your usage. Official MCP server means Claude Desktop users can plug in directly. The open-beta caveat is real—verify the SLA matches your product's uptime needs before launch.

Enterprise RAG over thousands of internal docs. Bedrock Knowledge Bases if you're already in AWS and you'll comfortably exceed the OCU floor. Pinecone Assistants if you're not. LlamaCloud if your team is already deep in LlamaIndex and the multimodal parsing earns its cost. All three have hybrid search; all three ship MCP servers. Pick based on where your infrastructure—and your team's existing expertise—already lives.

Team knowledge base. Self-hosted if it's under five people and you've got one engineer who cares about it. Hosted the moment it crosses twenty users or someone non-technical needs to upload docs. The threshold isn't the document count—it's the human factor.

The sqlite-vec era isn't ending. It's just not the only answer anymore. A year ago, self-hosted was the serious choice, and hosted was for people who didn't want to learn. In 2026, that framing doesn't hold. Hosted RAG is production-ready, MCP-native, and sometimes cheaper than your own ops time.

Pick the tool that matches the job. That's it.

FAQ

What is Cloudflare AI Search?

Cloudflare AI Search (formerly AutoRAG) is a managed Retrieval-Augmented Generation service built on Cloudflare's platform. It combines R2 storage, Vectorize for embeddings, and Workers AI for inference into a single API. It's currently in open beta with vector-first retrieval and optional reranking, and ships with an official MCP server that lets Claude and other AI assistants query your indexed documents directly.

When should I use hosted RAG instead of sqlite-vec for an MCP server?

Use hosted RAG when your corpus exceeds a few thousand documents, you're ingesting multiple source types like PDFs or Word docs, non-engineers need to upload content, or you need a production SLA. Stick with sqlite-vec when the corpus is static markdown under about 1,000 documents, you control ingestion, and cost-per-query matters more than ops time.

Can I use the Cloudflare AI Search MCP with Claude Desktop?

Yes. Cloudflare ships an official MCP server that exposes AI Search endpoints as MCP tools. Add the Cloudflare MCP server to your Claude Desktop config, provide your API token, and Claude can query your indexed documents through the same interface it uses for any other MCP tool. The setup is documented in the Cloudflare MCP repository.

Leave a comment

• Related reading:*

• How I Shipped an MCP Knowledge Server in a Weekend: the self-hosted case study this article references

• How Do MCP Registries Work (Smithery, mcpt)?: finding MCP servers, including the ones in this article

• Cortex: An Event-Sourced Memory Architecture for AI Coding Assistants: related exploration of the memory/retrieval landscape