One prompt, eight runs

What I ran, before you got here

One prompt, eight times

The same task you have just started, given to eight combinations of model and reasoning effort. Every cell got this string, byte for byte. Rewording it for one cell would have ended the comparison, so it was settled before the first cell ran and no cell has seen a different one.

Two things in it are worth noticing now, because they are columns further down. The waveform generator is off the shelf and the filter is not, so every cell has the same easy half and the same hard half. And it asks for the histogram of ρ² and its mean without saying the mean should be 2 — whether a run knows that is one of the things being measured.

The eight cells

Tokens and wall time are durable, prices are not: a cost in dollars is a fact about a price list on the day it ran. Wall time is as measured on the machine named, and a laptop is contended.

[src:grid_gpt54_xhigh_tokens_in, grid_gpt54_xhigh_tokens_out, grid_gpt54_xhigh_wall_s, grid_gpt55_xhigh_tokens_in, grid_gpt55_xhigh_tokens_out, grid_gpt55_xhigh_wall_s, grid_haiku45_xhigh_cost_usd, grid_haiku45_xhigh_tokens_in, grid_haiku45_xhigh_tokens_out, grid_haiku45_xhigh_turns, grid_haiku45_xhigh_wall_s, grid_opus5_low_cost_usd, grid_opus5_low_tokens_in, grid_opus5_low_tokens_out, grid_opus5_low_turns, grid_opus5_low_wall_s, grid_opus5_xhigh_cost_usd, grid_opus5_xhigh_tokens_in, grid_opus5_xhigh_tokens_out, grid_opus5_xhigh_turns, grid_opus5_xhigh_wall_s, grid_sol56_low_tokens_in, grid_sol56_low_tokens_out, grid_sol56_low_wall_s, grid_sol56_xhigh_tokens_in, grid_sol56_xhigh_tokens_out, grid_sol56_xhigh_wall_s, grid_sonnet5_xhigh_cost_usd, grid_sonnet5_xhigh_tokens_in, grid_sonnet5_xhigh_tokens_out, grid_sonnet5_xhigh_turns, grid_sonnet5_xhigh_wall_s]

What came back — 1

The noise curve, and the event by eye

Estimate the noise, divide it out, and look. No filtering, no template, no statistics: the chirp is visible in the whitened strain.

Opus 5, xhigh

Opus 5, low

gpt-5.6-sol, xhigh

Three runs, three amplitude spectral densities, the same instrument. Six of the seven cells that produced a page produced all three of the figures the prompt names [src:grid_n_three_figures, grid_n_ran], and their noise curves agree. Estimating the noise is the half of the task that does not separate anybody — the seventh cell is at the end, and what it got wrong was not this.

Opus 5, xhigh

Opus 5, low

gpt-5.6-sol, xhigh

Whitening is dividing each frequency by how noisy the instrument is there. The event is a fifth of a second long and it is the loudest thing in a third of an hour of data. You do not need a matched filter to see it. You need one to measure it, and to find out how often something that loud happens by accident.

What came back — 3

The SNR time series

Slide the template along the whitened data, weight every frequency by how noisy it is, and plot what comes out.

Opus 5, xhigh

Opus 5, low

gpt-5.6-sol, xhigh

Flat noise, and one spike. The spike is the whole of what a search does. Everything after it is deciding whether to believe it.

What came back — 4

The mean of ρ²

Run the same filter where there is no signal. If the normalisation is right, ρ² is χ² with two degrees of freedom and its mean is 2.

Opus 5, xhigh — mean 2.012, measured [src:grid_opus5_xhigh_rho2_mean]

Opus 5, low — mean 2.257, measured [src:grid_opus5_low_rho2_mean]

gpt-5.6-sol, xhigh — mean 1.884, measured [src:grid_sol56_xhigh_rho2_mean]

gpt-5.4, xhigh — mean 1.959, fitted [src:grid_gpt54_xhigh_rho2_mean]

Nobody told any of them the answer was 2 — the prompt asks for the histogram and the mean and stops there. Six of the seven runs that got this far supplied the expected value themselves, before quoting a number [src:grid_n_stated_2, grid_n_rho2_reported]. The seventh is two stages down.

Same check, different weight of evidence

What came back — 5

The constant that was fitted

Codex gpt-5.4, at xhigh effort, reported a mean of ρ² closer to 2 than anybody else, and it is the only cell whose check could not have failed.

def build_scale(...):
    ...
    return np.sqrt(2.0 / np.mean(raw_means))

Held-out validation is a real practice and that number is honest about what it measures: whether two stretches of noise look alike. What it cannot measure is the normalisation, because the normalisation was chosen to make it come out. The one check the prompt insisted on had been turned into a fit.

This is why the scoreboard has a column for how the mean was arrived at and not only for its value. Ranked by value alone, gpt-5.4 wins the column.

What came back — 6

The cheapest cell

Claude Haiku 4.5, at xhigh effort. Twenty-one cents [src:grid_haiku45_xhigh_cost_usd], three minutes [src:grid_haiku45_xhigh_wall_s], seventeen turns [src:grid_haiku45_xhigh_turns]. Against the reference cell's $16.47 [src:grid_opus5_xhigh_cost_usd] and its hour. And a report that reads like all the others.

This is the cell the grid was run to find, and it is why the prompt does not say the mean should be 2. Every other run supplied that number itself. This one did not have it, and so nothing in its own report could tell it that a peak SNR of 5,236 sits where the answer is about 19.5.

The whole grid

The scoreboard

One row per cell. Every column is something you can check without knowing whether the page reads well.

Every number above was read out of the cell's own results.json or measured off the page it left, by a rule fixed per column before the cells were read. scale is the square root of the cell's own mean of rho^2 over 2 — what its own background says its rho was multiplied by. or fitted to 2 is yes where the cell chose a normalisation constant by requiring that mean to come out at 2, which is decided from the source it wrote: such a cell cannot fail the check, whatever number it prints. An asterisk on the wall time means another cell was running on the same laptop.

[src:grid_gpt54_xhigh_chirp_mass, grid_gpt54_xhigh_choices_measured, grid_gpt54_xhigh_choices_n, grid_gpt54_xhigh_gps_peak, grid_gpt54_xhigh_page_external_refs, grid_gpt54_xhigh_page_inline_figures, grid_gpt54_xhigh_rho2_fitted_scale, grid_gpt54_xhigh_rho2_mean, grid_gpt54_xhigh_rho2_scale, grid_gpt54_xhigh_snr_h1, grid_gpt54_xhigh_tokens_in, grid_gpt54_xhigh_tokens_out, grid_gpt54_xhigh_wall_s, grid_gpt55_xhigh_chirp_mass, grid_gpt55_xhigh_choices_measured, grid_gpt55_xhigh_choices_n, grid_gpt55_xhigh_gps_peak, grid_gpt55_xhigh_page_external_refs, grid_gpt55_xhigh_page_inline_figures, grid_gpt55_xhigh_rho2_mean, grid_gpt55_xhigh_rho2_n_samples, grid_gpt55_xhigh_rho2_scale, grid_gpt55_xhigh_snr_h1, grid_gpt55_xhigh_tokens_in, grid_gpt55_xhigh_tokens_out, grid_gpt55_xhigh_wall_s, grid_haiku45_xhigh_chirp_mass, grid_haiku45_xhigh_choices_measured, grid_haiku45_xhigh_choices_n, grid_haiku45_xhigh_cost_usd, grid_haiku45_xhigh_gps_peak, grid_haiku45_xhigh_page_external_refs, grid_haiku45_xhigh_page_inline_figures, grid_haiku45_xhigh_rho2_mean, grid_haiku45_xhigh_rho2_scale, grid_haiku45_xhigh_snr_h1, grid_haiku45_xhigh_tokens_in, grid_haiku45_xhigh_tokens_out, grid_haiku45_xhigh_turns, grid_haiku45_xhigh_wall_s, grid_opus5_low_chirp_mass, grid_opus5_low_choices_measured, grid_opus5_low_choices_n, grid_opus5_low_cost_usd, grid_opus5_low_gps_peak, grid_opus5_low_page_external_refs, grid_opus5_low_page_inline_figures, grid_opus5_low_rho2_mean, grid_opus5_low_rho2_n_samples, grid_opus5_low_rho2_scale, grid_opus5_low_snr_h1, grid_opus5_low_tokens_in, grid_opus5_low_tokens_out, grid_opus5_low_turns, grid_opus5_low_wall_s, grid_opus5_xhigh_chirp_mass, grid_opus5_xhigh_choices_measured, grid_opus5_xhigh_choices_n, grid_opus5_xhigh_cost_usd, grid_opus5_xhigh_gps_peak, grid_opus5_xhigh_page_external_refs, grid_opus5_xhigh_page_inline_figures, grid_opus5_xhigh_rho2_gaussian_calibration, grid_opus5_xhigh_rho2_mean, grid_opus5_xhigh_rho2_n_samples, grid_opus5_xhigh_rho2_scale, grid_opus5_xhigh_snr_h1, grid_opus5_xhigh_tokens_in, grid_opus5_xhigh_tokens_out, grid_opus5_xhigh_turns, grid_opus5_xhigh_wall_s, grid_sol56_low_chirp_mass, grid_sol56_low_choices_measured, grid_sol56_low_choices_n, grid_sol56_low_gps_peak, grid_sol56_low_page_external_refs, grid_sol56_low_page_inline_figures, grid_sol56_low_rho2_mean, grid_sol56_low_rho2_n_samples, grid_sol56_low_rho2_scale, grid_sol56_low_rho2_sem, grid_sol56_low_rho2_sigmas_from_2, grid_sol56_low_snr_h1, grid_sol56_low_tokens_in, grid_sol56_low_tokens_out, grid_sol56_low_wall_s, grid_sol56_xhigh_chirp_mass, grid_sol56_xhigh_choices_measured, grid_sol56_xhigh_choices_n, grid_sol56_xhigh_gps_peak, grid_sol56_xhigh_page_external_refs, grid_sol56_xhigh_page_inline_figures, grid_sol56_xhigh_rho2_mean, grid_sol56_xhigh_rho2_n_samples, grid_sol56_xhigh_rho2_scale, grid_sol56_xhigh_rho2_sem, grid_sol56_xhigh_rho2_sigmas_from_2, grid_sol56_xhigh_snr_h1, grid_sol56_xhigh_tokens_in, grid_sol56_xhigh_tokens_out, grid_sol56_xhigh_wall_s, grid_sonnet5_xhigh_cost_usd, grid_sonnet5_xhigh_tokens_in, grid_sonnet5_xhigh_tokens_out, grid_sonnet5_xhigh_turns, grid_sonnet5_xhigh_wall_s]

The columns are the day's four subjects, and none of them needed a slide. Models and effort are the two left-hand columns. Benchmarking is what the rest of the row is. Headless is how every one of them ran: one command, no conversation, whatever it decided to do in between.

The same model, twice, with the dial turned down

[src:grid_opus5_low_cost_usd, grid_opus5_low_page_inline_figures, grid_opus5_low_rho2_mean, grid_opus5_low_snr_h1, grid_opus5_low_tokens_in, grid_opus5_low_wall_s, grid_opus5_xhigh_cost_usd, grid_opus5_xhigh_page_inline_figures, grid_opus5_xhigh_rho2_mean, grid_opus5_xhigh_snr_h1, grid_opus5_xhigh_tokens_in, grid_opus5_xhigh_wall_s, grid_sol56_low_page_inline_figures, grid_sol56_low_rho2_mean, grid_sol56_low_snr_h1, grid_sol56_low_tokens_in, grid_sol56_low_wall_s, grid_sol56_xhigh_page_inline_figures, grid_sol56_xhigh_rho2_mean, grid_sol56_xhigh_snr_h1, grid_sol56_xhigh_tokens_in, grid_sol56_xhigh_wall_s]

Turning the effort down does not degrade both lines the same way. On one of them the number moves and the checking survives. On the other the number does not move at all — it agrees to four figures — and what goes is the checking: the low cell never searches L1 on its own, so the loudest thing in that detector is never looked at. Both cost about a fifth as much and finish four to five times sooner.

The column that is this afternoon's exercise

The loudest thing in L1 in these 512 seconds is not the event. It is a glitch at GPS 1126259245.23, and in one cell's filter it reaches ρ = 17.2 where the real signal in that detector reaches 13.6. Four of the seven runs found it [src:grid_n_glitch_found, grid_n_ran]. All four found it by searching L1 on its own [src:grid_n_glitch_blind] — and only two threw it away without using H1 [src:grid_n_glitch_vetoed_alone], with a χ² that asks whether each band of the template contributed its share of the SNR. The other two simply noted that H1 saw nothing at that time.

Both answers are correct here, and they are not equally useful: a single-detector event has no second detector to appeal to. That is why it is on the list at the end of the hour.

Every cell left the same things — a page, a results.json, the source it wrote, and a record of what it consumed. You are not given ours. Your own run leaves them too, and that is the one worth opening.

All of it, as they left it

Their reports

Every page the grid produced, whole, in the state its cell wrote it. Pick one. Nothing here is edited, reformatted or excerpted — the file below is the file, and open puts it in its own tab.

Seven pages, one per cell that produced one [src:grid_n_ran]. The eighth is the greyed-out button: it ran for 31 minutes and wrote no page at all. Reading two of these side by side is the fastest way to see what the scoreboard is measuring — the same event, the same strain, the same prompt, and seven quite different accounts of it.

The rest of the hour

Back to your run

The weekend activity is the same task, as many times as you like, and you come back with your best attempt. Tomorrow is populations, and the thing being turned up is what you hand an agent when you are going to walk away from it.