Claude Code Agent Runner
A harness that hands an AI coding agent real build tasks and measures what it can ship end-to-end — with the feedback loops that keep it honest.
Stack
Problem
AI coding demos are cherry-picked. I wanted a repeatable way to measure what an agent can actually complete on real tasks, and where it fails.
Idea
A runner that gives the agent a task, a sandbox, and real checks (typecheck, lint, build, tests), then logs every attempt and failure mode.
Why I'm building it
This grew directly out of I Gave Claude Code an App to Build. The one-off experiment was interesting; a repeatable harness that scores it is useful.
Current state
- ✅ Task spec format + sandboxed workspace per run
- ✅ Feedback loop: typecheck / lint / build wired as tools
- 🔬 Failure-mode logging (what broke, how it "fixed" it)
- ⏳ Eval suite across a fixed task set
- ⏳ Cost + wall-clock accounting per task
Development log
2026-08-06 — Added a dry-run mode after an early version made real network calls it shouldn't have. Guardrails first.
2026-07-29 — First fully-autonomous task completed end to end: a validated API endpoint with tests, no human edits.
Lessons so far
The agent optimizes to whatever check you give it. A weak test suite produces confident, wrong code that passes CI — so the eval design is the product.