Exercises

Pick a task from your own work. Ask for it once in a sentence. Then ask again, in a new directory, with a request that names the files you want, their format, and the checks the answer has to pass.

What each one left behind. Whether the bare version checked anything you had not asked for. Then look at the list of things you had to spell out the second time, and decide which of them you would rather say once, in a file, than repeat every time you start.

Nothing. Your own project.

One task from your own work, run four times: two models, each at a low and a high effort setting. Same request every time, written down before you start so it does not drift between runs.

Not only whether the answers agree. Whether the CHECKING agrees - what each run verified on its own initiative, and what it quietly stopped doing when the dial went down.

Nothing. Your own project.

On day 2's grid, turning the effort down did two different things to two models. One returned a different number and kept verifying it. The other returned the identical number to four figures and stopped verifying altogether, which is the more dangerous of the two failures because nothing about the output says so.

One question, four directories. a/ empty. b/ the notes and nothing else. c/ the notes and a CLAUDE.md that mentions them. d/ the notes and a skill that says to read them. The same question in all four.

Whether b opens the notes at all - that is the one to watch. Whether the answer changes. Whether a run can tell you which page it used.

day5/exercise-notes/ - the question, the CLAUDE.md, the skill in both layouts, and a real project wiki: 105 pages behind a published paper, with an index, a notation contract, a formula list, fifteen concept pages, a page per source read and an append-only log.

The question that ships with it. Better, once you have done it once: build notes for a project of your own with the build-project-wiki skill in day5/skills/, and ask something whose answer you already know.

a and b gave the same wrong answer and neither opened anything. b began its reply by saying it would work the problem out directly rather than reach for a skill, with the notes sitting in the directory beside it. c and d read the notes and returned a result that is written down there and in no publication. The skill in d fired unprompted, on a description that names no subject matter and no filename.

Take a task from your own work and run it twice, in two directories, with the same model, the same settings and the same words. Then a third time if the first two disagree in a way you did not expect.

The two answers. Then the harder question: if somebody had told you these two runs differed in their setup, which difference would you have believed was responsible?

Nothing. Your own project.

While testing another exercise on this page we ran two directories whose only difference was a folder that neither agent opened. They gave different answers - one right and one wrong. That is what an unmeasured noise floor looks like when it is mistaken for a result.

A task from your own work, run once as a conversation and once as a single command with no interaction. Give both the same request.

The files each one left. Not the summary it wrote about itself - the files. A run that describes what it did is not evidence that it did it.

Nothing. Your own project.

One run in day 2's grid reported success, exited cleanly, took 109 turns, and produced no output file at all. Nothing in its own summary said so.

Take something your agent produced. First write down yourself what you think is wrong or weak in it. Then open a new session, give it the output and nothing else - no transcript, no reasoning, no context - and ask what is wrong with it.

Your list against its list. What it found that you had not. What it objected to that turns out to be fine, and whether you can tell which is which without going and checking.

Nothing. Your own project.

One task, three directories, the same request and the same model in each. a/ is empty. b/ has a skill stating the rule and a hook that checks it. c/ has the hook and no skill. Then read the three directories side by side - the directories, not the replies.

Where the result ends up: in a file, or only in a message on screen that disappears when you close the terminal. Whether the hook ever has to fire. What the difference cost in turns.

day5/exercise/ - the request, the skill, the hook, the rule it enforces, and the wiring for both command-line tools.

A short calculation ships with it, which needs no data and takes seconds. Any task that produces a figure and a number will do, including one of your own.

a left its result in the reply and nowhere else. b wrote the record without being asked, and the hook never had to fire. c decided it had finished, was stopped by the hook, and complied. All three got the same answer, and the record cost nothing in turns.

Take something of your own - a result, a paper, a talk you owe somebody - and have an agent build it as a slidev deck, and then as a self-contained page. Same content, twice.

Which one you would be willing to maintain, and why. What each format makes easy and what it makes impossible. What the agent did differently for each without being asked.

Both sources ship. Every lecture-*/ carries the slides.md it was built from, its figures and its build script; how-a-page-is-built/ carries the page builder, its template, and one session page's source.

Yours. This is the one where using somebody else's subject defeats the point.

Write one audit prompt and tell it, in the prompt, to change nothing. Seven measurements:

(1) every CLAUDE.md or AGENTS.md in scope, and the size of each;

(2) every MCP server, how many tools each contributes, whether tool deferral is on, and whether a proxy or gateway is configured - routing through one can turn deferral off without saying so;

(3) which model is set and at what effort, and the file each is set in;

(4) any hook that runs before a tool and trims the output of commands that print thousands of lines;

(5) every agent definition you have, and whether each pins a model or inherits one;

(6) every scheduled or recurring job, with its interval beside how long a cached prefix survives - one that fires after the cache has expired pays full price every time;

(7) a parse of the logs. Not the newest session: EVERY session in the log directory, one row each - requests, the context on its first turn, the context on its last, tokens written, total - and then the totals across all of them.

Ask for one table, sorted by cost, with five columns: the finding, its severity as RED, AMBER or GREEN, the evidence, what it is costing you, and what to do about it. Evidence has to be a number or a path and never an adjective; UNKNOWN is a legal answer where a guess is not. Then one line naming the single change with the most behind it.

The sessions against each other. Which of them cost several times what the others did, and what was different about how you were working. How much of each total was a cache read rather than something new. Whether the ones where you started fresh came out cheaper than the ones where you carried on. And, separately from the logs: how much of what is loaded before you type anything was used by the work you actually did.

One session tells you almost nothing. Twenty side by side tell you which way of working costs what, which is the whole point of doing it directory-wide.

Nothing. Your own machine, and your own logs.

Yours, and it has to be - the input is the week you have just had. By now you have run something headless, left a run alone against a written objective, and put a team of agents on a job, so your own logs carry real differences rather than ours.