granite — calibration report

2026-09-05 12:24 · pipeline 2026.09.05 · seed ? · 60 probe generations · 24 repair attempts

Scope: this file. granite-4.1-3b-Q5_K_M.gguf

Everything below was measured on this model, at this size, at this quantization. Quantization changes the logit distribution and every signal here is a property of that distribution — a different quant of the same weights is a different organism and needs its own run. None of it is claimed to transfer to another model.

Entropy signal

none

p=0.151

Effect seen

d=-0.55

detectable: d≥1.06

Failures

of 60 generations

Recoverable

4 / 8

by any strategy tested

The rule

Implementable as written. Every number below was measured on this model, on this run.

Do not gate generation on entropy. No entropy statistic separates this model's correct from incorrect output well enough to act on, so an entropy gate costs compute and buys nothing.
On a failed generation, repair in this order: skeleton_fill → test_retry. This ordering is greedy set cover over what actually recovered failures in this run, not a preference — skeleton_fill alone recovered 3 of 8.
When the failure is a logic error, route to test_retry first (1/4, 25%), ahead of skeleton_fill 25%, temp_retry 0%.
When the failure is an assertion error, route to test_retry first (2/3, 67%), ahead of skeleton_fill 67%, temp_retry 0%.
When the failure is a syntax error, nothing recovered it. All 3 strategies scored 0 of 1. Don't spend retries here — escalate or drop the task.
Stop after the ordered chain. 4 of 8 failures were recoverable by any strategy tested; the remaining 4 were not recovered by any of them, so further retries on those are spend without evidence behind it.

Where to cut, and what it costs

One signal, one curve. What changes is where you cut it and what you do with each side. Fitted and scored across cross-validation folds on 60 unique generations (8 failures) — each number is what the rule did on records its own fold never saw. Pick one row; they are three settings of the same dial, not three rules to stack.

Catch wrong — regenerate when margin_mean ≤ 0.9688. Chosen in 4 of 5 folds; only readable once the generation is complete, so the tokens are already spent when it fires.

Failures caught

80%

of all failing generations

Correct work lost

41%

regenerated needlessly

Net tokens

-95,210

per 1000 generations (-71.6%)

Per 1000 generations: 106.7 doomed runs aborted early, 357.9 correct ones thrown away and regenerated. Token counts are this run's own.

Trust right — ship untested when margin_mean > 0.9665. Chosen in 2 of 5 folds; only readable once the generation is complete, so the tokens are already spent when it fires.

Output covered

76%

shipped without testing

Purity

93.0%

of what ships is correct

Defects shipped

53.5

per 1000 generations

This saves test execution, not generation tokens — the generation is already paid for by the time any signal can be read. Worth it when the test suite is expensive or latency matters more than compute. The defect count is the price.

Both — one cut point, not two, on margin_mean. Chosen in 2 of 5 folds; only readable once the generation is complete, so the tokens are already spent when it fires.

The fit put both cut points at the same value, so there is no undecided band: every generation lands in trust or in bail. That makes this row the same single threshold as the two above, read from both sides — not a third, richer rule.

Zone	Condition	What to do	Share of output
Trust	`margin_mean > 0.9688`	ship untested — 71.9% correct	58%
Band	between the two	test it — the signal cannot decide	0%
Bail	`margin_mean ≤ 0.9688`	regenerate — catches 60% of all failures	42%

Net -45,638 tokens per 1000 generations from the bail zone (-34.3% of baseline).

Ship the trust zone untested, regenerate the bail zone, test the band between them. Both cut points come from one fit on one signal, ordered so the zones cannot overlap. The band is what the signal cannot decide — the honest residue, not a gap in the analysis.

No operating point here pays for itself in tokens. The best of them still nets -45,638 tokens per 1000 generations. That is not a flaw in the rule — this run's winning signal is only readable once a generation is complete, so aborting saves nothing that was not already spent, while every wrongly flagged generation pays for a full replacement. Use these rules to spend less on testing, or to decide what to ship, not to spend fewer tokens.

Does entropy predict correctness here?

CorrectIncorrect

Every probe generation, placed by its max_entropy. Vertical marks are group means.

Correct generations averaged max_entropy 0.794; incorrect ones 0.974. The groups are shifted, not merely overlapping — this is the separation the rule above is built on.

A permutation test puts that difference at p=0.1511, against an alpha of 0.05. Reshuffling which generations were labelled correct reproduces a gap this large routinely.

Across 146 candidate statistics the largest effect was plateau_start_quartile at d=-1.327 (high=bad), and it does not survive correction for the number tested.

4 of 8 failures were recoverable by at least one repair strategy. Recovery is measured directly — break it, repair it, re-run the tests — so these counts do not depend on any of the statistical argument above.

Power: underpowered. This run could only reliably detect an effect of d>=1.06; the effect it saw was d=-0.55. That means this run cannot tell 'no entropy signal' apart from 'too few failures to see one'. Re-running with --repeats 4 (this run used 1) would give ~80% power on an effect that size.

Every candidate signal, corrected

All 146 numeric statistics were tested against the same pass/fail outcome, so raw p-values are corrected with Benjamini-Hochberg FDR. A statistic is called usable only if its adjusted p clears 0.05 and |d| ≥ 0.2. The shaded band is what this run was too small to detect at all.

Table view — all signals

Signal	Direction	d	p	p adjusted	Usable
`margin_mean`	high=good	+1.230	0.0009	0.0584	no
`mass1_mean`	high=good	+1.187	0.0014	0.0584	no
`phi_mean`	high=good	+1.187	0.0014	0.0584	no
`surprisal_mean`	high=bad	-1.169	0.0016	0.0584	no
`full_ent_mean`	high=bad	-1.118	0.0029	0.0584	no
`kl_uniform_mean`	high=good	+1.119	0.0029	0.0584	no
`mean_entropy`	high=bad	-1.119	0.0029	0.0584	no
`margin_std`	high=bad	-1.114	0.0032	0.0584	no
`mass5_f10_max`	high=good	+1.167	0.0043	0.0698	no
`kl_step_mean`	high=good	+1.046	0.0049	0.0715	no
`mass1_std`	high=bad	-1.034	0.0068	0.0903	no
`surprisal_std`	high=bad	-0.976	0.0097	0.1123	no
`q3`	high=bad	-1.001	0.0100	0.1123	no
`spike_token_idx`	high=bad	-0.970	0.0131	0.1366	no
`kl_step_std`	high=bad	-0.906	0.0146	0.1421	no
`plateau_start_quartile`	high=bad	-1.327	0.0162	0.1478	no
`kl_uniform_std`	high=bad	-0.892	0.0186	0.1525	no
`full_ent_std`	high=bad	-0.891	0.0188	0.1525	no
`osc_rate`	high=good	+0.870	0.0213	0.1637	no
`kl_step_early_vs_rest`	high=bad	-0.861	0.0241	0.1759	no
`kl_cent_late_minus_early`	high=bad	-0.790	0.0371	0.2526	no
`margin_f10_mean`	high=good	+0.771	0.0396	0.2526	no
`mono_down_frac`	high=bad	-0.759	0.0398	0.2526	no
`spike_height`	high=good	+0.748	0.0474	0.2553	no
`mass1_f10_mean`	high=good	+0.740	0.0479	0.2553	no
`phi_first10_mean`	high=good	+0.740	0.0479	0.2553	no
`surprisal_f10_mean`	high=bad	-0.720	0.0540	0.2553	no
`spike_quartile`	high=bad	-0.748	0.0593	0.2553	no
`first10_mean`	high=bad	-0.711	0.0595	0.2553	no
`full_ent_f10_mean`	high=bad	-0.711	0.0595	0.2553	no
`kl_uniform_f10_mean`	high=good	+0.711	0.0595	0.2553	no
`head_entropy`	high=bad	-0.712	0.0601	0.2553	no
`q1`	high=bad	-0.713	0.0601	0.2553	no
`mass20_late_minus_early`	high=good	+0.753	0.0612	0.2553	no
`tail_mass_late_minus_early`	high=bad	-0.753	0.0612	0.2553	no
`mass20_min`	high=good	+0.650	0.0759	0.2995	no
`tail_mass_max`	high=bad	-0.650	0.0759	0.2995	no
`margin_min`	high=good	+0.652	0.0826	0.3135	no
`surprisal_f10_max`	high=bad	-0.661	0.0850	0.3135	no
`total_variation`	high=bad	-0.651	0.0859	0.3135	no
`q4`	high=bad	-0.608	0.0996	0.3547	no
`mass20_std`	high=bad	-0.482	0.1148	0.3869	no
`tail_mass_std`	high=bad	-0.482	0.1148	0.3869	no
`mass1_min`	high=good	+0.595	0.1166	0.3869	no
`spike_relative_pos`	high=bad	-0.575	0.1303	0.4085	no
`first10_std`	high=bad	-0.582	0.1344	0.4085	no
`surprisal_max`	high=bad	-0.572	0.1354	0.4085	no
`first10_max`	high=bad	-0.573	0.1409	0.4085	no
`full_ent_f10_max`	high=bad	-0.573	0.1409	0.4085	no
`mass5_slope`	high=good	+0.353	0.1426	0.4085	no
`trend_rho`	high=good	+0.556	0.1501	0.4085	no
`full_ent_max`	high=bad	-0.552	0.1506	0.4085	no
`kl_uniform_min`	high=good	+0.551	0.1511	0.4085	no
`max_entropy`	high=bad	-0.551	0.1511	0.4085	no
`n_semantic`	high=bad	-0.484	0.2108	0.5496	no
`n_tokens`	high=bad	-0.484	0.2108	0.5496	no
`kl_cent_slope`	high=bad	-0.411	0.2715	0.6954	no
`q2`	high=bad	-0.414	0.2863	0.7207	no
`mass5_late_minus_early`	high=good	+0.327	0.3135	0.7633	no
`kl_step_late_minus_early`	high=good	+0.388	0.3137	0.7633	no
`kl_cent_f5_mean`	high=good	+0.371	0.3434	0.8219	no
`q_slope`	high=good	+0.316	0.4121	0.9268	no
`early_slope`	high=bad	-0.319	0.4126	0.9268	no
`full_ent_slope`	high=bad	-0.319	0.4126	0.9268	no
`kl_uniform_slope`	high=good	+0.319	0.4126	0.9268	no
`kl_cent_std`	high=good	+0.310	0.4227	0.9351	no
`q_range`	high=bad	-0.305	0.4308	0.9388	no
`margin_early_vs_rest`	high=good	+0.286	0.4670	1.0000	no
`q_curvature`	high=good	+0.262	0.4735	1.0000	no
`kl_step_slope`	high=bad	-0.250	0.5001	1.0000	no
`mass1_early_vs_rest`	high=good	+0.262	0.5002	1.0000	no
`mass20_mean`	high=good	+0.244	0.5042	1.0000	no
`tail_mass_mean`	high=bad	-0.244	0.5042	1.0000	no
`surprisal_early_vs_rest`	high=bad	-0.239	0.5381	1.0000	no
`q1_over_mean`	high=good	+0.227	0.5566	1.0000	no
`mass1_slope`	high=good	+0.222	0.5705	1.0000	no
`early_vs_rest`	high=bad	-0.212	0.5807	1.0000	no
`kl_uniform_early_vs_rest`	high=good	+0.212	0.5807	1.0000	no
`full_ent_early_vs_rest`	high=bad	-0.211	0.5813	1.0000	no
`margin_slope`	high=good	+0.207	0.5970	1.0000	no
`mass20_early_vs_rest`	high=bad	-0.184	0.6126	1.0000	no
`tail_mass_early_vs_rest`	high=good	+0.184	0.6126	1.0000	no
`mass1_f5_mean`	high=good	+0.179	0.6183	1.0000	no
`margin_f5_mean`	high=good	+0.223	0.6211	1.0000	no
`surprisal_slope`	high=bad	-0.193	0.6244	1.0000	no
`surprisal_f5_mean`	high=bad	-0.174	0.6271	1.0000	no
`tail_entropy`	high=bad	-0.191	0.6440	1.0000	no
`late_vs_early_half`	high=bad	-0.175	0.6472	1.0000	no
`mass5_min`	high=good	+0.201	0.6551	1.0000	no
`first5_mean`	high=bad	-0.137	0.6587	1.0000	no
`full_ent_f5_mean`	high=bad	-0.137	0.6587	1.0000	no
`kl_uniform_f5_mean`	high=good	+0.137	0.6587	1.0000	no
`full_ent_late_minus_early`	high=bad	-0.165	0.6660	1.0000	no
`kl_uniform_late_minus_early`	high=good	+0.164	0.6679	1.0000	no
`kl_step_f5_mean`	high=good	+0.161	0.6784	1.0000	no
`mass20_slope`	high=good	+0.184	0.6916	1.0000	no
`tail_mass_slope`	high=bad	-0.184	0.6916	1.0000	no
`q4_over_mean`	high=bad	-0.155	0.6951	1.0000	no
`kl_cent_f10_max`	high=bad	-0.153	0.7143	1.0000	no
`kl_step_f10_max`	high=bad	-0.243	0.7488	1.0000	no
`q_early_drop`	high=bad	-0.124	0.7576	1.0000	no
`q_monotone_down`	high=bad	-0.157	0.7616	1.0000	no
`kl_step_f10_mean`	high=good	+0.105	0.7797	1.0000	no
`mass1_late_minus_early`	high=good	+0.101	0.7940	1.0000	no
`mass5_std`	high=bad	-0.112	0.7969	1.0000	no
`mass5_mean`	high=good	+0.111	0.8055	1.0000	no
`osc_sign_changes`	high=bad	-0.099	0.8073	1.0000	no
`n_spikes`	high=good	+0.167	0.8213	1.0000	no
`surprisal_late_minus_early`	high=bad	-0.084	0.8271	1.0000	no
`mass5_early_vs_rest`	high=bad	-0.106	0.8274	1.0000	no
`path_ratio`	high=bad	-0.106	0.8331	1.0000	no
`margin_late_minus_early`	high=good	+0.066	0.8646	1.0000	no
`kl_cent_early_vs_rest`	high=good	+0.063	0.8730	1.0000	no
`mass5_f5_mean`	high=bad	-0.144	0.8753	1.0000	no
`kl_cent_mean`	high=bad	-0.058	0.8779	1.0000	no
`margin_max`	high=bad	-0.176	0.8813	1.0000	no
`kl_cent_max`	high=bad	-0.073	0.8814	1.0000	no
`full_ent_min`	high=good	+0.168	0.8817	1.0000	no
`kl_uniform_max`	high=bad	-0.169	0.8817	1.0000	no
`mass1_max`	high=bad	-0.168	0.8817	1.0000	no
`surprisal_min`	high=good	+0.168	0.8820	1.0000	no
`kl_step_max`	high=bad	-0.168	0.8840	1.0000	no
`mass5_max`	high=bad	-0.190	0.8928	1.0000	no
`kl_uniform_f10_max`	high=good	+0.043	0.9317	1.0000	no
`mass5_first`	high=bad	-0.033	0.9501	1.0000	no
`first_token_entropy`	high=good	+0.171	0.9595	1.0000	no
`full_ent_first`	high=good	+0.171	0.9595	1.0000	no
`kl_uniform_first`	high=bad	-0.171	0.9595	1.0000	no
`first3_mean`	high=good	+0.172	0.9597	1.0000	no
`mass1_first`	high=bad	-0.189	0.9602	1.0000	no
`phi_first`	high=bad	-0.189	0.9602	1.0000	no
`surprisal_first`	high=good	+0.189	0.9602	1.0000	no
`mass1_f10_max`	high=good	+0.022	0.9624	1.0000	no
`mass20_f5_mean`	high=bad	-0.094	0.9630	1.0000	no
`tail_mass_f5_mean`	high=good	+0.094	0.9630	1.0000	no
`margin_first`	high=bad	-0.189	0.9640	1.0000	no
`mass20_first`	high=good	+0.027	0.9642	1.0000	no
`tail_mass_first`	high=bad	-0.027	0.9642	1.0000	no
`margin_f10_max`	high=bad	-0.014	0.9742	1.0000	no
`first_to_max_ratio`	high=good	+0.172	0.9761	1.0000	no
`kl_cent_f10_mean`	high=good	+0.007	0.9898	1.0000	no
`tail_mass_f10_max`	high=bad	-0.025	0.9945	1.0000	no
`mass20_f10_mean`	high=bad	-0.045	0.9967	1.0000	no
`tail_mass_f10_mean`	high=good	+0.045	0.9967	1.0000	no
`mass5_f10_mean`	high=bad	-0.015	0.9973	1.0000	no
`q_argmax`	high=good	+0.045	1.0000	1.0000	no

Which set of signals covers the most failures

Not run: needs ~12+ failures to cross-validate a combination (8 present). Raise --repeats.

What repairs failures

Direct measurement, not correlation: each failed generation was re-attempted with every strategy and re-tested. Nothing in this section depends on the entropy question above.

Routing by error type

logic errors

assertion errors

syntax errors

What shows promise

Leads, not findings. Each one names the run that would settle it.

margin_mean

d=+1.230, raw p=0.0009, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

To settle it: Pre-register it: decide in advance to test this one statistic on a fresh run and report that result whichever way it goes. Re-scanning and picking the new winner is how the original mistake happened.

mass1_mean

d=+1.187, raw p=0.0014, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

phi_mean

surprisal_mean

d=-1.169, raw p=0.0016, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

full_ent_mean

d=-1.118, raw p=0.0029, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

kl_uniform_mean

d=+1.119, raw p=0.0029, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

mean_entropy

d=-1.119, raw p=0.0029, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

margin_std

d=-1.114, raw p=0.0032, adjusted p=0.0584. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

mass5_f10_max

d=+1.167, raw p=0.0043, adjusted p=0.0698. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

kl_step_mean

d=+1.046, raw p=0.0049, adjusted p=0.0715. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

mass1_std

d=-1.034, raw p=0.0068, adjusted p=0.0903. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

surprisal_std

d=-0.976, raw p=0.0097, adjusted p=0.1123. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

d=-1.001, raw p=0.0100, adjusted p=0.1123. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

spike_token_idx

d=-0.970, raw p=0.0131, adjusted p=0.1366. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

kl_step_std

d=-0.906, raw p=0.0146, adjusted p=0.1421. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

plateau_start_quartile

d=-1.327, raw p=0.0162, adjusted p=0.1478. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

kl_uniform_std

d=-0.892, raw p=0.0186, adjusted p=0.1525. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

full_ent_std

d=-0.891, raw p=0.0188, adjusted p=0.1525. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

osc_rate

d=+0.870, raw p=0.0213, adjusted p=0.1637. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

kl_step_early_vs_rest

d=-0.861, raw p=0.0241, adjusted p=0.1759. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

kl_cent_late_minus_early

d=-0.790, raw p=0.0371, adjusted p=0.2526. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

margin_f10_mean

d=+0.771, raw p=0.0396, adjusted p=0.2526. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

mono_down_frac

d=-0.759, raw p=0.0398, adjusted p=0.2526. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

spike_height

d=+0.748, raw p=0.0474, adjusted p=0.2553. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

mass1_f10_mean

d=+0.740, raw p=0.0479, adjusted p=0.2553. Clears p<0.05 on its own but not after correcting for the 146 statistics tested against the same outcome. With that many tests, one or two hits at p<0.05 is what noise looks like.

phi_first10_mean

More failures would settle the entropy question

This run saw d=-0.55 and could only reliably detect d≥1.06. That gap is why the verdict is 'none' rather than 'no'.

To settle it: Re-run with --repeats 4 (this run used 1). The run checkpoints after every generation, so it can be stopped and resumed.

Settings this run used

Parameter	Value	Source
`min_p`	0.0	registry (card-verified)
`presence_penalty`	0.0	registry (card-verified)
`repeat_penalty`	1.0	registry (card-verified)
`temp`	0.0	registry (card-verified)
`top_k`	0	registry (card-verified)
`top_p`	1.0	registry (card-verified)
`thinking`	False	registry (card-verified)

Generated by the Local Model Calibration Kit. Raw records for every generation and repair attempt are in the run’s _raw.jsonl; calibrate.py reanalyze rebuilds this report from them without re-running the model.