Day 3 — the practical session
Matias Zaldarriaga
Ondas Gravitacionales e Investigación Asistida por IA
Use ← and →, or the dots at the top. p switches between reading size and presenting size.
So you know which is which
They are both somebody else's code, both written for a school, and neither of them runs as shipped. One of them you are about to fight yourself. The other has already been fought, five times over, and the rest of this page is what came back.
Yours — you launch it in a minute
Parameter estimation with cogwheel. A binary whose parameters you are not told is injected into Gaussian noise coloured by example LIGO–Virgo curves. You find it in a spectrogram, read a chirp mass off the whitened strain by fitting the slope of f−8/3 against time, hand that guess to cogwheel to maximise the likelihood into a reference waveform, then sample the posterior and put a corner plot next to the injected truth.
github.com/jroulet/pitp-pe-exercises — the folder 2-parameter_estimation.
By Javier Roulet, at Caltech, for the IAS's Prospects in Theoretical Physics school. cogwheel is his own code — and he is Argentine, which is worth saying in this room.
Mine — already run, nobody runs it today
A search for a signal in real strain. A two-part problem set that goes from raw Hanford data to a matched-filter detection: whiten it, build one template, calibrate a veto against a glitch, then build a whole template bank and search the segment with it. The notebooks say which event is in the data and ship no timestamps with it, so it has to be searched for rather than looked up.
github.com/JonathanMushkin/pitp-search-tutorial.
By Jonathan Mushkin and Barak Zackay at the Weizmann Institute, out of their course Statistics, Algorithms and Experiment Design — Zackay lecturing, Mushkin among the teaching assistants.
The pairing is deliberate. Yours is inference on a signal somebody has already found; mine is the finding. Between them they are most of what a search-and-parameter-estimation group actually does, and neither of them was written by us.
Before I say anything
This is objective.txt. Put it in a directory of its own and hand it over:
/goal read objective.txt and follow its instructions. Do not stop until you are done.
It is meant to outlast the session, and that is deliberate. You leave with the pieces we build while it runs, plus whatever your own run got to.
Launch it twice, in two directories, changing nothing. Same model, same effort, same words. At the end, put the two answers side by side. Nothing else in this course runs the same thing twice, so nobody yet knows how far an answer moves when nothing moves — and the room can find that out in one afternoon.
You have to decide the permissions before you walk away. An agent that stops to ask is an agent that is not running while you are in the corridor; an agent that never asks is an agent you have to be able to let loose in that directory. Which mode you pick is yours — but pick it now, on purpose, rather than discovering it at the first prompt.
Two things are worth noticing about the file. It asks for the report FIRST and the tool last, so that the hour produces something whatever happens. And it says what a good report is: one that says the download failed, with the traceback, is a good report. One that implies things ran when they did not is the failure.
Link 1 · before the prompt
Worth having in mind before you look at what came back, because the interesting question is not whether they got an answer — they all did — but which of these decisions each one made, and whether it said so.
Notebook 1 — the machinery.
Load the strain and transform it. Whiten it with a median-averaged Welch estimate, everything below twenty hertz removed by force. Then, in the notebook's own words, find a very obvious “glitch” in the time-domain whitened data, and histogram a stretch without one.
Build a single template at arbitrarily chosen masses, apply the linear-free phase transformation, and normalise it so that the auto-correlation returns one at zero lag. Form the complex overlap and its SNR-squared, and check both against the normal and the chi-squared predictions — on the real data, and on mock data drawn from the estimated noise curve.
Then the veto: split the template at the frequency that halves the accumulated SNR-squared, form the statistic out of the difference of the two halves, and set its threshold so that it will remove 1 in a 100 good signals.
Notebook 2 — the search.
It opens with the budget: assume you have an error budget of 5% in the SNR-squared, and two derivations on paper — the log-likelihood in terms of SNR-squared, and the SNR-squared loss written through the overlap between two waveforms. Those two are what turn a budget into a bank spacing.
Then build an orthonormal phase basis by singular-value decomposition, on a deliberately coarse frequency grid, and decide from the eigenvalue plot how many components to keep. Draw mass samples, project them into those coordinates, and select a subset such that the distance between any 2 samples is not smaller than 0.1. That subset is the bank, and its size is the reader's to report.
Run every template over the segment with the veto, keeping the best template and the best SNR in each tenth of a second. Then report the time, the masses, and a false-alarm rate built out of the bank size, the duration of the data, and the autocorrelation length of a waveform.
Twenty-two lines across the two notebooks are placeholders marked wrong — not blanks, values. The notebooks run with them in and produce plausible-looking garbage. Among the things a reader has to derive to replace them: the whitening filter actually applied to the data, the normalisation that makes the auto-correlation come out at one, the split into unit-norm halves, the threshold that removes one good signal in a hundred, the common phase evolution and the weights inside the decomposition, which factors are the coordinates and which are the basis, how many components to keep, and the distance scale that sets how dense the bank is. No solutions ship.
Keep the last of those in mind. The distance scale is the line that turns the error budget into a number of templates, it is one of the deliberately wrong values, and every one of the five had to pick it. It is what a reviewer measures them against, three sections from here. [src:chain_problem_set_wrong_lines, chain_problem_set_1_wrong_lines, chain_problem_set_2_wrong_lines, chain_problem_set_error_budget_pct, chain_problem_set_min_distance, chain_problem_set_veto_false_positive]
Nothing on this stage is ours. The counts are a grep over the two notebooks; the numbers and every phrase in italics are theirs, re-read out of the notebooks each time this page is built.
Link 1 · the ask
The same file went to five sessions that could not see each other. It asks for four things, in this order: a runnable search.py, a results.json carrying the detection and every choice behind it, a reference.npz of three arrays with their digests, and a self-contained report.html.
Three constraints in it are doing work. Nothing may be installed, so the environment is fixed and what is being compared is the program. No published solution may be looked up. And nothing may be optimised for speed — because the straightforward version is what the last two links get measured against, and that comparison is void if it has already been tuned.
Link 1 · what came back
Whole, in the state the run wrote it. Nothing is edited, reformatted or excerpted, and open puts it in its own tab.
The letters are how the next link saw them. The greyed-out one is the fifth session: it ended with its own search still running and wrote no report at all [src:chain_B_delivered]. Its numbers exist because its own program was re-run afterwards, unmodified, and the folder that went to review says so in plain words. Keep that one in mind. Every one of the four opens with no network and no server, which the prompt asked for and which both reviewers checked rather than assumed.
Link 1 · what came back
Their search.py, whole, as it was written. Pick one and scroll. Nothing is highlighted or reformatted; the header carries the line count and the digest of the file the lines came from.
| program | search.py | its own .py files | __main__ guard | qmc.Halton | its report |
|---|---|---|---|---|---|
| A · claude-opus-5 · xhigh · /goal | 1,006 | 2 | yes | rng= | 2.4 MB |
| B · claude-opus-5 · xhigh · inline | 1,736 | 1 | no | seed= | — |
| C · gpt-5.6-sol · xhigh · inline | 700 | 1 | yes | seed= | 1.4 MB |
| D · claude-opus-5 · max · /goal | 1,158 | 3 | yes | seed= | 1.6 MB |
| E · gpt-5.6-sol · xhigh · /goal | 1,057 | 1 | yes | rng= | 1.8 MB |
A program with no __main__ guard does its work on import, so it cannot be read into another file, tested in pieces or asked to run half of itself. The Halton column is one word of source: seed= and rng= both work and give different sequences, so two of these printing the same seed searched different banks.
[src:chain_A_py_files, chain_A_report_bytes, chain_A_search_py_lines, chain_B_py_files, chain_B_search_py_lines, chain_C_py_files, chain_C_report_bytes, chain_C_search_py_lines, chain_D_py_files, chain_D_report_bytes, chain_D_search_py_lines, chain_E_py_files, chain_E_report_bytes, chain_E_search_py_lines]
Open B and C next to each other. They answer the same question, they get the same answer, and one of them is two and a half times the length of the other with no entry point in it — import that file and the search starts. The reviewer that ranked it first still called it, in its own words, a 1,736-line module that starts a 17-minute run on import [src:chain_B_search_py_lines].
Link 1 · some comments
| program | via /goal | its session delivered | templates | spacing | winning template | chirp mass | SNR² | search.py |
|---|---|---|---|---|---|---|---|---|
| A · claude-opus-5 · xhigh · /goal | yes | 4 of 4 | 1,097 | 0.1 | 860 | 1.19773 | 333.79 | 1,006 |
| B · claude-opus-5 · xhigh · inline | no | 1 of 4 | 1,139 | 0.1 | 896 | 1.19769 | 334.50 | 1,736 |
| C · gpt-5.6-sol · xhigh · inline | no | 4 of 4 | 1,151 | 0.1 | 191 | 1.19761 | 329.54 | 700 |
| D · claude-opus-5 · max · /goal | yes | 4 of 4 | 1,097 | 0.1 | 1,062 | 1.19775 | 333.14 | 1,158 |
| E · gpt-5.6-sol · xhigh · /goal | yes | 4 of 4 | 369 | 0.2 | 281 | 1.19761 | 327.00 | 1,057 |
Five independent programs, five different banks, five different winning templates — and the chirp masses agree to 0.00014 solar masses. The bank column is what each one chose to build, not what it was told to build. The prompt names neither a size nor a spacing; the notebook prescribes 0.1, and one of the five went to 0.2 and halved its bank. Three of the five were handed the objective through /goal and a file rather than inline, and all three left all four deliverables; of the two given it inline, one did. Five runs is not a measurement of that, but it is the reason the invocation on the first stage is the one written there.
[src:chain_A_bank_n, chain_A_delivered, chain_A_distance_scale, chain_A_mchirp, chain_A_search_py_lines, chain_A_snr2, chain_A_winner, chain_B_bank_n, chain_B_delivered, chain_B_distance_scale, chain_B_mchirp, chain_B_search_py_lines, chain_B_snr2, chain_B_winner, chain_C_bank_n, chain_C_delivered, chain_C_distance_scale, chain_C_mchirp, chain_C_search_py_lines, chain_C_snr2, chain_C_winner, chain_D_bank_n, chain_D_delivered, chain_D_distance_scale, chain_D_mchirp, chain_D_search_py_lines, chain_D_snr2, chain_D_winner, chain_E_bank_n, chain_E_delivered, chain_E_distance_scale, chain_E_mchirp, chain_E_search_py_lines, chain_E_snr2, chain_E_winner, chain_mchirp_spread, chain_n_delivered_all, chain_n_programs, chain_n_via_goal]
They are the same event, found five times, by five programs that share no line of code. That is the thing worth stopping on. Writing the analysis a second independent way used to be the check you saved for a result you were going to defend for a decade; here it is one more run of the same prompt, and the fifth one costs what the first one cost.
Link 2 · the ask
Five lettered folders, no run ids, no models, no token counts. Two fresh sessions got the same brief and could not see each other's answer. What “best” means was left to them on purpose — the criteria they chose are as much the result as the ranking they produced.
Link 2 · what came back
Each is the reviewer's whole answer, self-contained, as it was written. Read the two verdicts first, then their criteria.
They worked differently and said so. One re-implemented the conditioning chain and re-derived every number it quotes; the other read all the source, checked every recorded digest against the arrays, and ran one targeted experiment. What separates the two answers is not what they found — they found nearly the same things — but what they decided those things were worth.
Link 2 · some comments
| place | Claude Opus 5 — built its own implementation and re-derived the numbers | Codex gpt-5.6-sol — read all of it, checked every hash, and ran one targeted experiment |
|---|---|---|
| 1 | B · claude-opus-5 · xhigh · inline | B · claude-opus-5 · xhigh · inline |
| 2 | A · claude-opus-5 · xhigh · /goal | C · gpt-5.6-sol · xhigh · inline |
| 3 | D · claude-opus-5 · max · /goal | D · claude-opus-5 · max · /goal |
| 4 | C · gpt-5.6-sol · xhigh · inline | E · gpt-5.6-sol · xhigh · /goal |
| 5 | E · gpt-5.6-sol · xhigh · /goal | A · claude-opus-5 · xhigh · /goal |
Same five folders, same brief, no communication, and they agree on 2 of 5 places — the first and the middle one. The winner is unanimous. Everything else is inverted, and the largest single disagreement is A, which one of them puts second and the other puts last.
[src:chain_A_delivered, chain_B_delivered, chain_C_delivered, chain_D_delivered, chain_E_delivered, chain_n_rank_agreements]
The winner is unanimous, and it is B — the session that delivered least. Its own hour produced one file of the four [src:chain_B_delivered]; the numbers it was ranked on came from re-running that one file afterwards. What both reviewers rewarded was the program, not the session that wrote it, and those turn out not to be the same thing.
The cheaper review is less than half the wall clock [src:chain_link2_opus5_wall_s, chain_link2_sol56_wall_s]. It is also the one that read C's phase handling and credited it — “correctly restores the removed linear phase for physical coalescence time” — where the reviewer that had built its own copy of the conditioning chain came back with a coalescence time more than a millisecond late. That is what the extra hour bought, and it is the only place on this page where one of them is simply wrong about the other's subject.
The same evidence, twice
Both found the same defect in A: its search.py writes the numbers it was asked for and then dies, because the last thing it does is import a module that is not in the folder. One reviewer ranked it second anyway. The other ranked it last.
Ranked it second.
“So this is a packaging failure, not a program that was never run; but the folder I was handed does not build, and that is what a reviewer has to score.”
It weighed the two files the brief actually names as written before the crash, and paid A back on the analysis, which it rated the best in the set.
Ranked it last.
“A normal full run writes new results.json and reference.npz, then fails instead of regenerating the report. The existing HTML can silently become stale.”
Same sequence, opposite sign: writing new numbers and then dying next to an old report is worse than dying early, because what it leaves behind disagrees with itself and nothing says so.
Neither is wrong. They are answering different questions — is the analysis good, and is the folder safe to hand to somebody else — and the brief did not say which one mattered, on purpose. You only find out that the criterion was doing the work when there are two of them.
Link 2, the part worth reading
Every one of the five had to choose how tightly to pack its template bank, and every one of the five justified its choice in one sentence. Those sentences were checked against a fresh draw of signals none of the banks was built from.
| program | templates | spacing | loss it claims | loss a reviewer measured |
|---|---|---|---|---|
| A · claude-opus-5 · xhigh · /goal | 1,097 | 0.1 | 1.0 % | 2.20 % |
| B · claude-opus-5 · xhigh · inline | 1,139 | 0.1 | 2.24 % | 1.87 % |
| C · gpt-5.6-sol · xhigh · inline | 1,151 | 0.1 | about 1 % | 1.97 % |
| D · claude-opus-5 · max · /goal | 1,097 | 0.1 | 1.0 % | 2.30 % |
| E · gpt-5.6-sol · xhigh · /goal | 369 | 0.2 | 4 % | 6.07 % |
4 of 5 claim a tighter bound than they have. The error is the same one every time: the greedy placement rule does guarantee that every one of the 8192 candidates it drew is within d of a kept template, and that is not a statement about a signal from the continuum, which can sit further out. Measured against a fresh draw the covering radii are larger, and the loss goes as the square of them.
[src:chain_A_bank_n, chain_A_loss_claimed_pct, chain_A_loss_measured_pct, chain_B_bank_n, chain_B_loss_claimed_pct, chain_B_loss_measured_pct, chain_C_bank_n, chain_C_loss_claimed_pct, chain_C_loss_measured_pct, chain_D_bank_n, chain_D_loss_claimed_pct, chain_D_loss_measured_pct, chain_E_bank_n, chain_E_loss_claimed_pct, chain_E_loss_measured_pct, chain_n_loss_understated]
Two more findings from the same review, both of the same kind — a thing that reads like a check and is not one.
Two programs, one seed, two different banks.
qmc.Halton(seed=…) and Halton(rng=…) are both accepted, and they scramble differently. Two of these programs print the same seed and searched different banks — “both work in scipy 1.16.2 but produce different sequences, so B's and E's seed 170817 are different banks.”
Every one of the five does reproduce its own reference exactly. The seed is not a name for the bank.
The GPS time is an assumption wearing a measurement's clothes.
The data carries no timestamps. All five get an absolute time by assuming the segment starts on an integer second, then several of them quote the residual against the published time as agreement. “The residual is bounded by ±0.5 s by construction, because it is the rounding error of the integer-second assumption; it cannot be large, so it is not evidence.”
The one that handled it best simply declined to play, and said the time was anchored rather than recovered.
The bank sentence is the same slip in four different programs, and it is worth naming exactly: greedy placement really does guarantee that each of the 8192 drawn candidates sits within d of a kept template [src:chain_n_loss_understated]. It says nothing about a signal drawn from the continuum, which is what a real one is.
Link 2 · after the fact
Two reviewers, two backends, neither able to see the other, the same winner and the same two reasons for it. That is the shape of a result. Both reasons were checked afterwards against the archive. The first one was ours.
“Only B's folder rebuilds itself.”
The five folders were assembled by assemble_review.py, which copies four files per program [src:chain_review_artifacts] — the program, and the three things the brief asks it to produce.
A's own run directory holds two Python files of its own [src:chain_A_py_files]: search.py, and the report_builder whose absence both reviewers scored it on. D's holds three [src:chain_D_py_files] — its search, the make_report.py that assembles its report, and a verifier. B, C and E wrote one each [src:chain_B_py_files, chain_C_py_files, chain_E_py_files], so nothing of theirs was lost.
Three files did not travel [src:chain_n_py_dropped], and all three belong to the two programs that were marked down for not regenerating their own reports. A's folder, as A left it, runs end to end; D's takes two commands rather than one, and the second command was in D's directory.
“Only B measured its own bank.”
This one holds. It is the slip on the stage before: four of the programs justified their spacing with a guarantee about the candidates they drew, and B is the one that went and measured the thing the guarantee does not cover.
It separated nobody. Every bank at the notebook's 0.1 [src:chain_problem_set_min_distance] comes out between 1.87 % and 2.30 % [src:chain_B_loss_measured_pct, chain_D_loss_measured_pct], inside the 5 % the problem set allows [src:chain_problem_set_error_budget_pct], measured or not.
And B did not read its own number. The sentence it was praised for says the placement rule guarantees the loss, and then that the measured worst case is 2.24 % [src:chain_B_loss_claimed_pct], and carries on. The measurement was made and the conclusion was not drawn.
The bottom of the ranking survives, and for reasons that are in the code. C is the only one of the five whose coalescence time disagrees with the other four, because it recomputes the offset on the full-resolution frequency grid when the bank's convention was fixed on the sparse one. E set its spacing to 0.2 [src:chain_E_distance_scale], said that would cost 4 % [src:chain_E_loss_claimed_pct], and it costs 6.07 % [src:chain_E_loss_measured_pct] — the one program that departed from the notebook, over the budget it invoked to justify the departure. Between A, B and D there is nothing.
Both reviewers read the same five folders, and two of those folders were missing a file we did not copy. Their agreement says they were given the same exhibit; it does not say the exhibit was right. The next two links are still built on B, because B is a fine program to build on and the archive is what it is — but the ranking that chose it was separating its top three on one artefact of ours and one measurement that changed no order.
Link 3 · the ask
We took the winner's program and asked for speed, with an acceptance test a machine settles by itself: three arrays back, and the same three digests. Bit-identical means exactly that and nothing softer — the reference is a file, not a number somebody has to agree with, and there is no tolerance in it. Faster and different is a failure, not a trade-off.
It also asks for the thing that was found and could not be used: if there is something much faster that moves the last bits, say what it was and what it would have bought — but do not ship it. That paragraph is what the next link was built out of.
Link 3 · what came back
Each profiles the original, says what it changed, and shows the digests coming back unmoved. Both end with what they rejected.
One is more than ten times the length of the other. That is a difference in how much of the work is written down, not in how much of it was done.
Link 3 · some comments
| program | wall | speed-up | user CPU | cores busy | peak memory | it claimed |
|---|---|---|---|---|---|---|
| the baseline — B's own program, unchanged | 16 m 52 s | — | 828 s | 0.8 | 7.2 GB | — |
| Claude Opus 5 · bit-identical | 2 m 52 s | 5.9× | 969 s | 5.6 | 8.6 GB | 6.7x |
| Codex gpt-5.6-sol · bit-identical | 6 m 16 s | 2.7× | 1013 s | 2.7 | 8.2 GB | 2.48x |
Every wall clock here was measured by us, on an idle machine, with /usr/bin/time -l. The last column is what the run said about itself: the two differ by one to two per cent in both cases, which is what it looks like when a program times itself and somebody else times it.
[src:chain_baseline_cores_busy, chain_baseline_peak_rss_gb, chain_baseline_user_s, chain_baseline_wall_s, chain_link3_opus5_cores_busy, chain_link3_opus5_cpu_reduction_vs_baseline, chain_link3_opus5_peak_rss_gb, chain_link3_opus5_speedup_vs_baseline, chain_link3_opus5_user_s, chain_link3_opus5_wall_s, chain_link3_sol56_cores_busy, chain_link3_sol56_cpu_reduction_vs_baseline, chain_link3_sol56_peak_rss_gb, chain_link3_sol56_speedup_vs_baseline, chain_link3_sol56_user_s, chain_link3_sol56_wall_s]
Read the user-CPU column. Neither program made the computation cheaper. Both made it wider: the arithmetic went up and the wall clock came down because it was spread over cores that were sitting idle [src:chain_baseline_user_s, chain_link3_opus5_user_s, chain_link3_sol56_user_s]. That is a real result and it is worth having, but it is not the same result as making the program better, and only the second column would have told you which one you got.
Both programs reproduce all three digests exactly. The contract holds only for the machine and the library versions it was measured on: the environment is part of the claim, the same way the machine is part of a wall clock.
Link 4 · the ask
Both link-3 reports said, unprompted, that more was available if the last bits were allowed to move — and both listed what they had rejected to keep them still. So we changed the acceptance test and asked again.
What one of them turned down.
“It disagrees with numpy.fft in the last bit, with or without workers, on the very first array I tried it on. Dead on arrival, and it took the whole family of ‘use a better FFT library’ ideas with it (FFTW, Accelerate/vDSP): any of them would move the answer.”
What the other one turned down.
“The clearest larger shortcut—reconstructing the full overlap from low/high overlaps—made its kernel 1.40× faster and could plausibly cut another 20–30% from the whole program when combined with threading, but it moved last bits in almost every sample.”
Both are that run's own words, re-read out of its own report every time this page is built [src:chain_link3_opus5_rejected_quote, chain_link3_sol56_rejected_quote]. The numbers in them are its measurements of a thing it did not ship, and it is the one that would know.
Link 4 · what came back
Same shape as the last pair, and both now carry a section on what the approximation cost. Both also ship a check_extra.py that nobody asked for.
Both were held to the same five checks in check_detection.py, shipped with the task and not editable by the run it judges, and both passed them.
Link 4 · some comments
| program | wall | speed-up | user CPU | cores busy | peak memory | it claimed |
|---|---|---|---|---|---|---|
| the baseline — B's own program, unchanged | 16 m 52 s | — | 828 s | 0.8 | 7.2 GB | — |
| Claude Opus 5 · science gate | 56 s | 18.0× | 341 s | 6.2 | 3.9 GB | 17.9x |
| Codex gpt-5.6-sol · science gate | 2 m 38 s | 6.4× | 225 s | 1.6 | 3.8 GB | 6.34x |
Same measurement, same machine, same column order. This time user CPU falls, and peak memory with it.
[src:chain_baseline_cores_busy, chain_baseline_peak_rss_gb, chain_baseline_user_s, chain_baseline_wall_s, chain_link4_opus5_cores_busy, chain_link4_opus5_cpu_reduction_vs_baseline, chain_link4_opus5_peak_rss_gb, chain_link4_opus5_speedup_vs_baseline, chain_link4_opus5_user_s, chain_link4_opus5_wall_s, chain_link4_sol56_cores_busy, chain_link4_sol56_cpu_reduction_vs_baseline, chain_link4_sol56_peak_rss_gb, chain_link4_sol56_speedup_vs_baseline, chain_link4_sol56_user_s, chain_link4_sol56_wall_s]
Now the user CPU falls, and the memory with it [src:chain_link4_opus5_user_s, chain_link4_sol56_user_s, chain_link4_opus5_peak_rss_gb, chain_link4_sol56_peak_rss_gb]. Under the loose contract the computation genuinely got cheaper, and the two split the win differently. Codex wrote the leaner program — least CPU of anything on this page, and it barely parallelises. Claude wrote the faster one, less efficient per core and winning the wall clock on six of them [src:chain_link4_opus5_cores_busy, chain_link4_sol56_cores_busy, chain_link4_opus5_cpu_reduction_vs_baseline, chain_link4_sol56_cpu_reduction_vs_baseline].
The part no amount of compute buys
The gate that link 4 was held to is five checks in one file: the same moment wins, from the same template, its height moves by less than a tenth of a per cent, every loud interval agrees to one per cent, and nothing quiet comes back loud.
“That is the MINIMUM. Passing it is not the same as the science being unchanged, and I have deliberately not tried to write down every way it could differ. If you make a change whose effect on the result is not covered by those five checks, working out what else has to be true is your job.”
Both of them answered the sentence rather than the number. They wrote extra checks nobody specified — fourteen and eight of them [src:chain_link4_opus5_extra_checks, chain_link4_sol56_extra_checks] — and both reinstated constraints the gate had dropped on purpose: that the bank is the same bank, that the winning template is the same one in every interval and not only at the event, and that the background's whole shape is unchanged, since the false-alarm rate is a property of all of it and no per-interval tolerance implies it.
And then the question the gate could not put a threshold on. Asked what the approximation had cost, in physical terms rather than in last bits:
“SNR moves 7.9×10⁻⁸ (≈3 pc on a 40 Mpc distance whose measurement error is ±8 Mpc); 5 samples in 9.55×10⁹ are vetoed differently.”
That answer is the run's own, re-read out of its own report every time this page is built [src:chain_link4_opus5_cost_quote]. Deciding in advance what a wrong answer could still reproduce is the one check on the list that has not got cheaper. Everything else on this page did.
Since it is the same task five times
| link | claude | codex | faster by | claude tokens out | codex tokens out | leaner by |
|---|---|---|---|---|---|---|
| 1 · inline | 44 m 51 s | 33 m 52 s | 1.3× | 170,256 | — | — |
| 1 · via /goal | 2 h 07 m | 30 m 16 s | 4.2× | 235,493 | 38,870 | 6.1× |
| 2 · review | 27 m 20 s | 12 m 10 s | 2.2× | 104,386 | 31,457 | 3.3× |
| 3 · bit-identical | 1 h 45 m | 1 h 13 m | 1.4× | 217,685 | 46,378 | 4.7× |
| 4 · science gate | 1 h 58 m | 51 m 35 s | 2.3× | 287,114 | 57,817 | 5.0× |
Wall clock is the agent's own session, not the program's. Link 4's claude column is two sessions summed, because that side ran in two. Both backends were given the same prompt file, byte for byte, on the same laptop on the same day.
[src:chain_link1_opus5_tokens_out, chain_link1_opus5_wall_s, chain_link1_sol56_wall_s, chain_link1b_opus5_goal_tokens_out, chain_link1b_opus5_goal_wall_s, chain_link1b_sol56_goal_tokens_out, chain_link1b_sol56_goal_wall_s, chain_link2_opus5_tokens_out, chain_link2_opus5_wall_s, chain_link2_sol56_tokens_out, chain_link2_sol56_wall_s, chain_link3_opus5_tokens_out, chain_link3_opus5_wall_s, chain_link3_sol56_tokens_out, chain_link3_sol56_wall_s, chain_link4_opus5_resumed_tokens_out, chain_link4_opus5_resumed_wall_s, chain_link4_opus5_tokens_out, chain_link4_opus5_wall_s, chain_link4_sol56_tokens_out, chain_link4_sol56_wall_s]
One of them is faster on every link and leaner in output tokens by more than it is faster. It also does a bit less. Rarely on what it was explicitly asked for — usually on the extras, taking whichever reading of the task is the shortest one. The clearest instance is on this page already: it is the program that chose the wide bank, half the templates of anybody else's, and justified the choice with a number that turned out to be the optimistic one [src:chain_E_bank_n, chain_E_loss_claimed_pct, chain_E_loss_measured_pct].
The instance that has to go in beside it: on the thing it was explicitly asked to do, its link-4 program is the leanest of the lot — least CPU, least memory, and it gets there on fewer than two cores [src:chain_link4_sol56_user_s, chain_link4_sol56_cores_busy]. Neither of these is a verdict on a brand. They are two shapes of answer, and knowing which one you are being handed is the whole reason to run both.
The rest of the hour
First, look at what it has already written.
Not the summary in the terminal — the files. Did the report get written before the tool, the way the objective asked? Does it say what failed, or does it only describe what worked?
Then check the permission mode you chose against what it actually did while you were not looking.
Then pick one.
The weekend activity is the same objective, as many times as you like, and you come back with your best attempt. Tomorrow the same idea gets several agents with different jobs instead of one.
Reference
Four prompts, run one after another, each one handed what the last one produced. Nothing here was run live and nothing is being run now: they finished on the two days in the last column, and this page reads their files.
| run | model | effort | format asked for | tokens in / out | wall | when | receipt |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 · xhigh | claude-opus-5 | xhigh | search.py + results.json + reference.npz + self-contained .html | 30,196,936 / 170,256 | 44 m 51 s | 2026-08-13 | — |
| Codex gpt-5.6-sol · xhigh | gpt-5.6-sol | xhigh | search.py + results.json + reference.npz + self-contained .html | — | 33 m 52 s | 2026-08-13 | — |
| Claude Opus 5 · xhigh · /goal | claude-opus-5 | xhigh | search.py + results.json + reference.npz + self-contained .html | 58,906,227 / 235,493 | 2 h 07 m | 2026-08-13 | — |
| Claude Opus 5 · max · /goal | claude-opus-5 | max | search.py + results.json + reference.npz + self-contained .html | 39,510,597 / 216,365 | 1 h 17 m | 2026-08-13 | — |
| Codex gpt-5.6-sol · xhigh · /goal | gpt-5.6-sol | xhigh | search.py + results.json + reference.npz + self-contained .html | 6,796,509 / 38,870 | 30 m 16 s | 2026-08-13 | — |
| Review · Claude Opus 5 · xhigh | claude-opus-5 | xhigh | search.py + results.json + reference.npz + self-contained .html | 19,736,050 / 104,386 | 27 m 20 s | 2026-08-13 | — |
| Review · Codex gpt-5.6-sol · xhigh | gpt-5.6-sol | xhigh | search.py + results.json + reference.npz + self-contained .html | 4,022,550 / 31,457 | 12 m 10 s | 2026-08-13 | — |
| Speed-up · Claude Opus 5 · xhigh | claude-opus-5 | xhigh | search.py + results.json + reference.npz + self-contained .html | 30,464,529 / 217,685 | 1 h 45 m | 2026-08-13 | — |
| Speed-up · Codex gpt-5.6-sol · xhigh | gpt-5.6-sol | xhigh | search.py + results.json + reference.npz + self-contained .html | 15,247,343 / 46,378 | 1 h 13 m | 2026-08-13 | — |
| Science-gate speed-up · Claude Opus 5 · xhigh | claude-opus-5 | xhigh | search.py + results.json + reference.npz + self-contained .html | 27,289,627 / 225,809 | 1 h 33 m | 2026-08-14 | — |
| Science-gate speed-up · Claude Opus 5 · xhigh (resumed) | claude-opus-5 | xhigh | search.py + results.json + reference.npz + self-contained .html | 20,493,337 / 61,305 | 24 m 40 s | 2026-08-14 | — |
| Science-gate speed-up · Codex gpt-5.6-sol · xhigh | gpt-5.6-sol | xhigh | search.py + results.json + reference.npz + self-contained .html | 10,250,114 / 57,817 | 51 m 35 s | 2026-08-14 | — |
Tokens and wall time are durable, prices are not: a cost in dollars is a fact about a price list on the day it ran. Wall time is as measured on the machine named, and a laptop is contended.
[src:chain_link1_opus5_cost_usd, chain_link1_opus5_tokens_in, chain_link1_opus5_tokens_out, chain_link1_opus5_turns, chain_link1_opus5_wall_s, chain_link1_sol56_wall_s, chain_link1b_opus5_goal_cost_usd, chain_link1b_opus5_goal_tokens_in, chain_link1b_opus5_goal_tokens_out, chain_link1b_opus5_goal_turns, chain_link1b_opus5_goal_wall_s, chain_link1b_opus5_max_cost_usd, chain_link1b_opus5_max_tokens_in, chain_link1b_opus5_max_tokens_out, chain_link1b_opus5_max_turns, chain_link1b_opus5_max_wall_s, chain_link1b_sol56_goal_tokens_in, chain_link1b_sol56_goal_tokens_out, chain_link1b_sol56_goal_wall_s, chain_link2_opus5_cost_usd, chain_link2_opus5_tokens_in, chain_link2_opus5_tokens_out, chain_link2_opus5_turns, chain_link2_opus5_wall_s, chain_link2_sol56_tokens_in, chain_link2_sol56_tokens_out, chain_link2_sol56_wall_s, chain_link3_opus5_cost_usd, chain_link3_opus5_tokens_in, chain_link3_opus5_tokens_out, chain_link3_opus5_turns, chain_link3_opus5_wall_s, chain_link3_sol56_tokens_in, chain_link3_sol56_tokens_out, chain_link3_sol56_wall_s, chain_link4_opus5_cost_usd, chain_link4_opus5_resumed_cost_usd, chain_link4_opus5_resumed_tokens_in, chain_link4_opus5_resumed_tokens_out, chain_link4_opus5_resumed_turns, chain_link4_opus5_resumed_wall_s, chain_link4_opus5_tokens_in, chain_link4_opus5_tokens_out, chain_link4_opus5_turns, chain_link4_opus5_wall_s, chain_link4_sol56_tokens_in, chain_link4_sol56_tokens_out, chain_link4_sol56_wall_s]
Tokens and wall time are recorded because both command-line tools hand them over for free. Wall time is the agent's own session on one laptop, not the program's; where two ran at once the machine column says so.