A team on an open question

First thing, before anything else

Install the tool

Everything today runs through one command-line tool called agent-team. Clone it, and run its installer:

git clone git@github.com:matiaszaldarriaga/agent-team.git
cd agent-team && ./install.sh

Cloning it does nothing on its own. The installer names three consumers and wires one thing for each, and only the first is what you would guess. It puts the job command on your path, for you. Then it links the same SKILL.md into ~/.claude/skills/ and into ~/.codex/skills/ — one file, two places, because both tools read the same format. Without it, a fresh session of either has no idea this one exists: it will not find the tool, because nothing told it to look.

That is what a skill is and where it lives: a short written description of a tool, dropped where a session reads it at start-up, so that the tool becomes reachable by name. Both command-line tools read the same format, so one file serves both. It is the first thing you install today because nothing after it works otherwise.

Before I say anything about mine

Start it now.

Put this in a file called objective.txt, in a directory of its own, and launch it:

job new derive "@objective.txt" --rounds 3 --run

It will not be finished when the hour ends. Our rounds of this same job ran 32.4, 39.7 and 47.5 minutes on one laptop [src:team_v2_student_cl_r1_span_min, team_v2_student_cl_r2_span_min, team_v2_student_cl_r3_span_min], so three rounds is about two hours of work [src:team_v2_student_cl_active_s] and more than that in wall clock. Launch it, leave it, and come back to it at the end of the hour. The weekend is long enough for the rest.

Two things in the file are load-bearing. It asks for the same result in three formats — a LaTeX spine, a self-contained page and a marimo notebook with the parameters left free — because what an agent does differently for each of the three is a result. And it says which C_l is meant, every time, because there are two objects with that name in this week and the morning lecture derives the other one.

We ran it twice, to check the task was reasonable and produced something. Here is the shape of what came back, and nothing else:

The task names five files, and the last two are the same result in two formats: a self-contained report.html, and report.py, a marimo notebook with the number of sources and the power fractions left free so a reader can move them and watch the answer respond. One of the two runs left the notebook unwritten. Nothing about what either of them found is on this page, and that is deliberate — your own run is the experiment, and every team's sky is different.

[src:team_student_cl_discrete_sky_cost_usd, team_student_cl_discrete_sky_rounds_asked, team_student_cl_discrete_sky_rounds_done, team_student_cl_discrete_sky_tokens, team_v2_student_cl_cost_usd, team_v2_student_cl_rounds_asked, team_v2_student_cl_rounds_done, team_v2_student_cl_tokens]

What either of ours found is not on this page and will not be said out loud. Every team's sky is a different draw, the answer is short enough to check by hand, and the whole point of the exercise is the checking. A page that prints our outcome turns it into an answer key.

Run 1 · the ask

The puzzle, as it stood

This is a question we had actually been stuck on, written down the way it sat in my head rather than tidied up. It names no quantity to compute. The last line of the goal is decide for yourself what has to be computed, and that is the whole of the instruction.

Three things in it are doing work besides the question. It lists the papers the job may read and fences it out of the two directories that hold worked answers, so what comes back is the team's and not ours. It says every number must come from code in the job or from a named paper with its equation, and that an order of magnitude is usually a bug — stop on the second kind. And it says, in as many words, do not assume the framing is correct because we wrote it.

Who ran it

Every role ran at the same reasoning effort. The lead and the verifier are on one command-line tool and the workers on the other, on both of the jobs this page reports — a verifier on a different model family is the part of the check that is independent. The two student jobs took the default and ran everything on one tool, which is why they cost the most per token. The clock is the sum of the rounds' own spans and does not include the stretches where a job sat waiting for a human.

[src:team_pta_run1_puzzle_cost_usd, team_pta_run1_puzzle_rounds_asked, team_pta_run1_puzzle_rounds_done, team_pta_run1_puzzle_tokens_m, team_pta_run2_questions_cost_usd, team_pta_run2_questions_rounds_asked, team_pta_run2_questions_rounds_done, team_pta_run2_questions_tokens_m, team_student_cl_discrete_sky_cost_usd, team_student_cl_discrete_sky_rounds_asked, team_student_cl_discrete_sky_rounds_done, team_student_cl_discrete_sky_tokens_m, team_v2_run1_puzzle_cost_usd, team_v2_run1_puzzle_rounds_asked, team_v2_run1_puzzle_rounds_done, team_v2_run1_puzzle_tokens_m, team_v2_run2_questions_cost_usd, team_v2_run2_questions_rounds_asked, team_v2_run2_questions_rounds_done, team_v2_run2_questions_tokens_m, team_v2_student_cl_cost_usd, team_v2_student_cl_rounds_asked, team_v2_student_cl_rounds_done, team_v2_student_cl_tokens_m]

The staffing is the part worth copying. The lead and the verifier run on one command-line tool and the workers on the other, so the thing that checks a claim is a different model family from the thing that made it. Same prompt, same papers, same machine; what varies down this page is the question, and then the version of the engine.

Run 1 · what came back

Two documents and a monitor

Whole, in the state the job left them. Nothing is edited, reformatted or excerpted, and open puts it in its own tab. report.html is what a physicist reads; agent.html is the same content with every number resolving to the code that computed it. view.html is neither: it is the job's own monitor, and it is the file that makes a team legible while it is still running.

Both prompts name five files: notes.tex, the spine, whose every quantitative claim carries a key; provenance.json, the registry those keys resolve in; checks.py, which prints coverage numbers rather than verdicts; and two self-contained HTML files, one a physicist reads and one carrying the provenance behind every number in it. All four jobs left all five. The last column is what they added unasked — figures on disk, a compiled PDF, the scripts that built the pages.

[src:team_pta_run1_puzzle_out_bytes, team_pta_run2_questions_out_bytes, team_v2_run1_puzzle_out_bytes, team_v2_run2_questions_out_bytes]

The greyed-out button is the honest form of an absence. The first run of this prompt has no round reports at all, because the directory they go in did not exist yet: its rounds survive only as raw transcripts. That gap is one of the results, and the last section of this page is about it.

Run 1 · what we make of it

One new result, then the packaging

The exact part is free. A sky made of point sources has an angular power spectrum you can write down in closed form, and all three of the prompts on this page had it in round one — each from its own derivation, and each confirmed by a verifier working from an implementation it wrote rather than the worker's.

Both are that job's own words, re-read out of its own files every time this page is built [src:team_q_identity, team_q_identity_check]. That the derivable part comes back correct in one round, three times independently, is why it is the exercise you launched and not the thing I am showing.

The one thing it found that we did not have

An anisotropy limit and a limit on the loudest single source are not the same statement, and the conversion between them is not the one you would write down without thinking. The team derived the exact chain and then said what follows:

A fourth root, not a square root. It is a genuinely new result and it was not in the prompt, in the papers, or in the lead's own plan for that round — the plan had written the square root, and the verifier corrected it: “The ε^1/4 correction to the plan is right, and it is close to tight in the interesting regime.” It survived into the deliverable: “The fourth root is the robust strain ceiling; using √ε as the strain ceiling confuses a strain-power fraction with a strain ratio.” [src:team_q_fourth_root, team_q_fourth_root_correction, team_q_fourth_root_shipped]

And then the rounds ran out

That is round one. The last round of the same job was spent entirely on the two files it had not written:

Run the same prompt again and it does not land in the same place. The second time, the tension the question was built on came apart: “The proposed contradiction was a comparison error, not new astrophysics.” [src:team_q_run1_v2_conclusion] One question, two teams, two different headlines — which is what you would expect of a question this loose, and is the reason the next section exists.

Run 1 · what we make of it

What the verifier filed, and what shipped

In round two the verifier went at the run's own headline and refuted it. In round three the run shipped that sentence anyway, in both deliverables.

The verifier named the lines. The next round's lead assigned exactly one edit to that file and it was a different one. The refutation was not overruled; it was never delivered. The engine of the day pasted a filtered slice of the ledger into every prompt, and the filter kept verified claims: “refuted and unclear claims were stored and never shown to anyone, so a verifier's finding could not reach the round that had to act on it.” [src:team_q_run1_refutation, team_q_run1_shipped_refuted, team_q_engine_filter]

That is fixed, and the same prompt run afterwards shows what the fix looks like from the outside. Its lead's report carries a section headed “Unresolved claims — how each is being handled this round” and rules on every one, including the one it disagrees with: “Not a detectability crossover. The verifier showed both statistics were normalized by their null std”. [src:team_q_unresolved_section, team_q_unresolved_ruling]

Minutes are that round's own clock: from the lead's plan landing to the round closing. Tokens and dollars are cumulative for the job, which is how the log records them.

[src:team_pta_run1_puzzle_r1_span_min, team_pta_run1_puzzle_r2_span_min, team_pta_run1_puzzle_r3_span_min, team_v2_run1_puzzle_r1_span_min, team_v2_run1_puzzle_r2_span_min]

Read the clock column against the round number rather than against the other job. What a round costs is set by how much its lead put in it, and both of those numbers are decisions somebody made.

Run 2 · the ask

The same subject, as five questions

Same physics, same papers, same fence, same team. What changes is that somebody sat down and worked out what would actually have to be computed, and wrote it as five numbered questions in the order they have to be answered.

The fifth question is the one that is hard to write and easy to leave out: make everything you have found in 1–4 compatible with each other and with the published results, and say plainly where it leaves the question. Without it a team hands back four answers and no position.

It also keeps the escape hatch open. Under what you cannot get from the sources it says whether the five questions are the right five — if answering them turns up something that makes one of them the wrong question, say so. That instruction is answered, at the end, and the answer is on this page.

Run 2 · what came back

Six rounds, and every round written down

The same two documents and the same monitor, and this time the rounds themselves. Each round page carries every role's report for that round, in the order the round ran them, as markdown they wrote and nothing has rendered.

Round 3 is the one to open if you only open one. It is the round the team believed was its last, and the round after it is the one that was given three more.

Run 2 · what we make of it

What six rounds bought

Where the loose prompt got one new result and then spent its last round on packaging, this one kept going, and what it produced is the shape of a small paper.

Every sentence in those two cards is that job's own, re-read out of its notes.tex at every build [src:team_q_grid, team_q_driver, team_q_inversion, team_q_no_winner]. Which of the two searches wins is the question the whole hour started from, and the answer it arrives at is it depends, and here is the number nobody has measured.

It also went back and computed a thing it had left at one. Round one had set the geometry factor relating a dipole to a monopole to unity, which is the sort of placeholder that quietly becomes an assumption; round two's verifier did the integral: “I also computed g₁ = 0.661 (0.618–0.721), the dipole/monopole Fisher-norm ratio they set to 1.” [src:team_q_geometry_at_one]

Run 2 · what we make of it

How the rounds behaved

All of this is in reports/, transcript/ and state.json, which is to say all of it is in the directory your own job is writing right now.

A round budget is an input the lead plans against

With two rounds left, the lead reorders the work around the deadline rather than around the physics: “if I spend this round on science-only, round 3 must produce Q5 plus notes.tex plus the merged registry plus checks.py plus two offline-clean HTML files from nothing. That fails.” A round later it closes enquiry outright: “No new science beyond what is listed; this is the last round.” [src:team_q_science_only, team_q_last_round]

That round's clock ran 8.5 minutes, against 37 for round one. [src:team_pta_run2_questions_r3_span_min, team_pta_run2_questions_r1_span_min] The wind-down is visible in the timings before it is visible in the text.

Which is a problem, because a stopped job can always be resumed, so no round is ever the last one. The other job was told exactly that, in the middle of the run: “A run can always be resumed, so ‘the last round’ is never final” — and was asked instead to “end every round's report with a short section headed WHAT MORE COULD BE DONE”. That is now in the lead's own role file: “the PI now produces the best deliverable the remaining rounds allow AND reports what further rounds would buy, since that is the input to the human's decision to resume.” [src:team_q_steer_rounds, team_q_steer_rounds_ask, team_q_pi_rule, team_q_pi_rule_why]

They price their own wasted effort, late

“no browser backend exists in this environment; three roles have now tried. Record as a known unverified property and stop re-attempting.” [src:team_q_stop_reattempting] Three roles, over three rounds, each independently deciding to have another go at the same impossible check before a lead finally wrote it down as impossible.

They are blind to their own coverage

At round six the lead found “two figures in 2306.16221 that no role has opened in six rounds” — in a paper the job was handed on day one. They were not incidental: “That is the external anchor for ρ_total that r05 said the job did not have”, which is to say they were the outside number for the quantity the team had been arguing about since round two. Opening one of them settled it against the team: “The data themselves reject ρ_total = 1.44 as the measured background-S/N scale.” [src:team_q_never_opened, team_q_never_opened_why, team_q_never_opened_payoff]

Under a tight round it is the format that goes

The first run of this prompt shipped two documents whose mathematics does not render. Its own verifier found it in round six: “out/report.html and out/agent.html emit LaTeX math as raw source inside <span class="math"> with no renderer present (no KaTeX, MathJax, or MathML anywhere in either file).” The round-seven lead read the state and ruled: “Structurally the deliverable is complete; bookkeeping is not what's blocking. Both tasks this round are physics.” Both tasks were physics, and the defect shipped. [src:team_q_defect_found_inside, team_q_defect_lead_reply]

A human then sent a direction into the running job saying the same thing — “out/report.html and out/agent.html do not render mathematics: the LaTeX delimiters were stripped and 477 raw commands are sitting in the visible text” [src:team_q_steer_maths] — and it was never read. Three roles ran after it landed on disk and the job's own cursor over that file never moved. Both halves of this are in the last section.

Same clock. Round 3 of the first job never closed: its two workers reported and no verifier ran, so the round has no verdict and the job's own summary skips it. It is on this page because the record is the record.

[src:team_pta_run2_questions_r1_span_min, team_pta_run2_questions_r2_span_min, team_pta_run2_questions_r3_span_min, team_pta_run2_questions_r4_span_min, team_pta_run2_questions_r5_span_min, team_pta_run2_questions_r6_span_min, team_pta_run2_questions_r7_span_min, team_v2_run2_questions_r1_span_min, team_v2_run2_questions_r2_span_min, team_v2_run2_questions_r3_span_min, team_v2_run2_questions_r4_span_min, team_v2_run2_questions_r5_span_min, team_v2_run2_questions_r6_span_min]

Both runs · measured against the paper

The result nobody reached

There is an answer key. The work these jobs were pointed at is a finished paper with a released repository, and every quantitative claim in it is stamped in a registry — 106 keys, each with the code that computed it. Three of its four results are on this page in one form or another. The fourth is not, and it is the one the paper is for.

They came within one step of it, twice, and both times it was filed and dropped. A round-one verifier quoted a supplied paper — “the current NANOGrav bound is also sensitive to the analysis priors adopted” — and then wrote down exactly what would follow: “Q3's inversion to p_1 rests on this and must not be presented as settled until it is checked.” It was logged unclear and never checked. The inversion shipped. [src:team_q_prior_raised, team_q_prior_raised_consequence]

The same half-step reached both deliverables, correct and one clause too narrow: “the paper reports that those posteriors return their priors under its Hellinger-distance diagnostic. They are therefore not data-informed measurements of the differential anisotropy”, and “The 0.2 number is a basis-dependent broadband upper-limit shorthand, and the frequency-resolved limits are largely prior quantiles”. Both sentences are about the frequency-resolved limits. Neither job turned the same question on the broadband number, and the broadband number is the one both of them then invert. [src:team_q_prior_landed, team_q_prior_shorthand]

What the six extra rounds went after instead was a factor of six: “NANOGrav is 3.68 times the grid maximum and 6.40 times the unrounded central fit”. Three of three added rounds carried a task aimed at closing it, and it ended where it started: “It is a measured signature, not an identified cause; isolating the cause still requires matched population products and conditioning”. [src:team_q_residual, team_q_residual_verdict]

The number on the other side of that factor is itself a simulation, and the job says so in its own description of how it was made: “the paper constructs a separate map in each bin, puts the 2000 loudest sources at random sky positions, treats the remainder as isotropic, and runs anafast through ℓ=6.” [src:team_q_residual_is_a_prediction] Nothing in the machinery asks what a thing you are measured against is made of. The question does, and neither prompt asked it.

The concrete prompt did answer the one question it was told it could not get from the sources. Its closing section says the five questions are “not quite the right five” [src:team_q_run2_conclusion], and says which two it would replace. That is worth more than the factor of six, and it arrived in the last round.

Since the tool is the subject

Three defects the six jobs found in the tool

Running these jobs put three items in the tool's own backlog. They are on this page because a defect you find by using something is the most useful thing you can say about it, and because the second one was introduced by the fix to the first.

One. A verifier's verdicts are the record; the file the engine reads back is a summary of them, capped. “On a six-round job the verifier emitted 5 refuted and 30 unclear; state.json ended holding 0 refuted and 20 unclear.” [src:team_q_backlog20] Five refutations and ten open items fell out of the window, and the index that tells a lead “n claims are unresolved — each one is your responsibility” counts from the truncated list. So it reported none open in a round where five were.

Verified / refuted / unclear. The left column is counted from every transcript/r*-verifier.txt in the job; the right is what state.json ends up holding, which is the file the engine itself reads back. They differ on 2 of the 6 jobs, and always in the same direction. Every refuted and unclear count anywhere on this page is the left column.

[src:team_pta_run1_puzzle_filed_refuted, team_pta_run1_puzzle_filed_unclear, team_pta_run1_puzzle_filed_verified, team_pta_run1_puzzle_kept_refuted, team_pta_run1_puzzle_kept_unclear, team_pta_run1_puzzle_kept_verified, team_pta_run2_questions_filed_refuted, team_pta_run2_questions_filed_unclear, team_pta_run2_questions_filed_verified, team_pta_run2_questions_kept_refuted, team_pta_run2_questions_kept_unclear, team_pta_run2_questions_kept_verified, team_student_cl_discrete_sky_filed_refuted, team_student_cl_discrete_sky_filed_unclear, team_student_cl_discrete_sky_filed_verified, team_student_cl_discrete_sky_kept_refuted, team_student_cl_discrete_sky_kept_unclear, team_student_cl_discrete_sky_kept_verified, team_v2_run1_puzzle_filed_refuted, team_v2_run1_puzzle_filed_unclear, team_v2_run1_puzzle_filed_verified, team_v2_run1_puzzle_kept_refuted, team_v2_run1_puzzle_kept_unclear, team_v2_run1_puzzle_kept_verified, team_v2_run2_questions_filed_refuted, team_v2_run2_questions_filed_unclear, team_v2_run2_questions_filed_verified, team_v2_run2_questions_kept_refuted, team_v2_run2_questions_kept_unclear, team_v2_run2_questions_kept_verified, team_v2_student_cl_filed_refuted, team_v2_student_cl_filed_unclear, team_v2_student_cl_filed_verified, team_v2_student_cl_kept_refuted, team_v2_student_cl_kept_unclear, team_v2_student_cl_kept_verified]

That index is itself the fix to the earlier bug, the one two sections up: the engine used to paste a filtered slice of the ledger into every prompt, and the fix was to stop pasting and start pointing. The pointer counts from the same truncated list. “The fix that closed the original bug reintroduced it one layer up.” [src:team_q_backlog20_lesson, team_q_engine_found]

Two. Two words are carrying four situations. refuted and unclear is all a verifier can file, so “a factor-of-three contradiction in a shipped number sits in the same bin as a missing browser” [src:team_q_backlog21] — one needs somebody to adjudicate it and the other needs everybody to stop trying, and both of those are on this page as things that actually happened.

Three. The reflex when a role cannot see what it needs is to engineer what each role gets passed. “A filter in the harness is a decision taken away from the PI without telling it.” [src:team_q_backlog22] The job directory is small and every role can read all of it; what the harness owes is an index and a statement of what is outstanding, and which of it matters this round is the lead's call.

All three are entries in agent-team's own backlog, at the commits named beside them, so every sentence quoted here is re-read out of that file when this page is built.

The rest of the hour

Back to your run

It is still going, and it will still be going when we stop. That is the normal case and it is worth getting used to: open the directory and read it while it works.

The one to do if you do only one is the last. Every job on this page that was given more rounds used them on something it had already decided to leave undone, and it said so in its own words when asked. That is cheap to test and you will have a stopped job in front of you.

The weekend activity is the same objective as many times as you like, with whatever you want to vary — the number of workers, the rounds, which tool the verifier runs on. Come back with the run you would defend.

Reference

The six jobs, and what they cost

Three prompts, each run twice, before and after the engine was fixed. Nothing here was run live and nothing is being run now: they finished on the two days in the last column, and this page reads their files.

Tokens and wall time are durable, prices are not: a cost in dollars is a fact about a price list on the day it ran. Wall time is as measured on the machine named, and a laptop is contended.

Each directory holds what the job was given (spec.json), what it ended up believing (state.json), every call it made (log.jsonl), every role's whole answer every round (transcript/), the reports it wrote (reports/), what it delivered (out/) and its own monitor (view.html). What is NOT here is work/, the agents' own scratch directory, and tmp/, where a paper gets rasterised so a figure can be read off it: between them they are most of what the six jobs occupied on disk, all of it regenerated arrays and image caches, and none of it read by anything on this page. The papers are not here either. They are already in papers/, and the copy inside each job existed only to fence it.

Tokens are the total across every role, which is the one figure the harness records: neither command-line tool hands back a split between what went in and what came out through it, so the table's column carries the total and the other half is a dash [src:team_pta_run1_puzzle_tokens, team_v2_run1_puzzle_tokens, team_pta_run2_questions_tokens, team_v2_run2_questions_tokens, team_student_cl_discrete_sky_tokens, team_v2_student_cl_tokens]. The clock is the sum of the rounds' own spans and excludes the stretches where a job sat waiting for somebody to resume it. The prices were the prices on the day.