How to read these studies

Read this section once. It defines the terms and the rules that every study below applies.

The three questions

A study asks three separate questions about cleverly. The counts are not interchangeable.

question	what it establishes about cleverly	what it cannot establish
Accuracy against known truth	cleverly clears declared bias and coverage-validity margins on a law whose truth is computed longhand	nothing about the derivation or exact nominal coverage. The canonical rows say nothing about cleverly
Agreement with the canonical implementation	cleverly and an independently maintained implementation compute the same thing on the same rows, within a declared margin	that either one is right. Two poor implementations can agree. This is why the accuracy question is asked first and separately
Theory properties	declared repeated-sampling properties, including robustness, root-n contraction, calibration, and error rates	behaviour outside the laws and compositions the cells declare. Efficiency requires a separate comparison with an independent bound

The scientific derivation is checked elsewhere. The exact-law, Gateaux, remainder, identity, and deliberate-mutation instruments are listed per estimand in the evidence manifest. A study measures a complete estimator under repeated sampling. It does not replace those instruments.

The verdict rules

Every rule is an equivalence rule bounded by a margin declared before the run. None of them tests whether a discrepancy is exactly zero, and the distinction is the design rather than a detail.

A Monte Carlo study accumulates evidence by adding replications. A verdict must therefore become easier, or at worst stay put, as replications grow. A significance test does the reverse. It converges on rejecting any estimator whose finite-sample remainder is not identically zero, which is every estimator. The study would eventually go red for the one reason that is not a defect. An earlier version of the first study carried two such rules. Quadrupling its replication count would have failed it without a line of estimator code changing. tests/unit/test_evidence_framework.py holds both rules side by side and asserts which way each one moves.

rule	what must hold	why it is bounded	its control
bias equivalence	the 99% Student interval for the error lies inside margin:standardized_bias empirical standard deviations of zero	a point test of zero bias fails every consistent estimator at enough replications	the same instrument read in the opposite direction: a control cell’s interval must lie entirely outside the margin
coverage validity	the lower endpoint of the exact Clopper-Pearson interval clears margin:coverage_floor	whether a nominal 95% interval is valid is the question. Whether it is 95% to the third decimal is a question no finite study answers	one-sided by design. Over-coverage is conservative, not invalid
SE sanity band	the bootstrap interval for mean reported SE over empirical SD lies inside margin:se_ratio_sanity_lower to margin:se_ratio_sanity_upper	this is a screen for a standard error wrong by an order of magnitude. The coverage floor binds first	a ratio of 0.80 corresponds to about 88% coverage, so the band cannot be tightened past what the floor implies
SE calibration	where both nuisances are correct, the SE ratio interval lies inside the calibration band and the exact coverage interval inside its own	this is the only two-sided calibration claim, and the only gate a uniform tenth-scale understatement fails	a shrunken_se_control cell multiplies the standard errors by a declared factor and must fail
efficiency	the empirical standard deviation of the estimates and the mean reported standard error both sit inside margin:efficiency_ratio_lower to margin:efficiency_ratio_upper of an independently computed efficient-influence-function bound	Monte Carlo error in the empirical spread, which a point test against the bound would reject at enough replications. Calibration and validity are asked separately, because an estimator can have root-n bias and calibrated intervals without attaining the bound	a noise_control cell adds one bound-unit of independent noise and must fail
paired similarity	the 99% interval for the mean paired difference lies within margin:paired_difference pooled empirical standard deviations	symmetric, because a large difference in either direction means the two implementations are not computing the same thing	a paired mutation makes cleverly materially worse and must fail
RMSE non-inferiority	the bootstrap upper bound for cleverly’s RMSE ratio is at most margin:rmse_noninferiority	one-sided, because cleverly performing better than the reference is a result rather than a failure	a reference-only mutation must fail the reference’s own column and leave cleverly standing
coverage non-inferiority	the lower bound for the coverage difference is at least margin:coverage_noninferiority	one-sided, for the same reason	as above
coverage superiority	the 99% lower bound for Cleverly minus reference coverage is positive, while Cleverly passes its truth gates and the RMSE and SE-calibration non-inferiority gates pass	superiority must improve a validity endpoint. A point-estimate difference or shorter runtime is descriptive	a comparison with equal coverage cannot pass this route
calibration non-inferiority	the upper bound for excess absolute SE-calibration error is at most margin:calibration_noninferiority	applied only where the two native inference scales are comparable	an exemption must be earned. A test requires the two implementations to report genuinely different scales
type-I error	the one-sided upper endpoint of the rejection rate under a confounded sharp null stays under margin:type_i_ceiling	the null law keeps its confounding, so the test is not an unadjusted comparison	a power cell under a real effect must reject. An inert test cannot pass by never firing
power	the rejection lower bound clears margin:minimum_power	this is the positive control the type-I cell needs	none. It is itself a control
root-n rate	the log-log slope interval lies within margin:root_n_slope_lower to margin:root_n_slope_upper	a band rather than containment of -1/2, which is a point test the reported-SE rate already fails at these replication counts	the interval must also exclude margin:excluded_slope, so a merely decreasing spread fails

The efficiency rule needs a bound this package did not compute from the estimator it judges. A study without one runs no efficiency cell, and its tables say so. The property family named root_n_and_efficiency is the exception in name only. The name is historical, it tests bias, coverage, and SE calibration across sample sizes, and the efficiency comparison lives in interval_calibration.

Each implementation is judged on its own terms as well as on the pairing. A reference that degrades is reported against the reference. It does not turn cleverly’s row red.

A paired row concludes equivalent, superior, inferior, or inconclusive. Equivalence and superiority are separate passing routes. A reporting study can publish a failed scientific verdict. It still refuses an unrecorded missing replication, invalid schemas, and broken provenance. Failure to establish non-inferiority is inconclusive, not affirmative evidence of inferiority; the inferior label is reserved for a future rule with a confidence bound in that direction.

The failure accounting checks each passing route alone. A superior row whose superiority does not survive its comparator failures concludes equivalent when equivalence survives. A row that keeps no passing route concludes unaccounted.

The harness has negative controls for all of it, in tests/unit/test_method_evidence.py. The fast tests corrupt bias, coverage, and reported standard errors for each implementation in turn. They require only that implementation to fail and require the untouched one to keep passing.

Reproducing a study

Each study page ends with a Reproduction section listing its own artifacts. Every study commits the same set, and each file answers the same question wherever it appears.

artifact	what it holds
the fixture README.md	the regeneration commands, in full and in smoke form
manifest.json	the configuration, every declared margin, every seed and package pin, the cleverly version and commit, the interpreter and library versions, and a hash of every published result and study module
replicates.csv.gz	one row per replication per implementation
performance-tests.csv	the accuracy verdicts against known truth
equivalence.csv	the paired verdicts against the canonical implementation
properties.csv	the theory-property cells and their controls
failures.csv	each recorded fitting failure of either implementation, with its replicates, stage, and message. Present only when a fit failed
failure-accounting.csv	each cell that lost a replicate, with each side’s count and bound, its legs at the complete cases and at the worst case, and its gate. Present only when a fit failed
property-failures.csv	each recorded fitting failure of a property cell, with its replicates, stage, and message. Present only when a property fit failed
property-failure-accounting.csv	each property cell that lost a replicate, with its count, its bound, and one row for each declared clause. Present only when a property fit failed
property-failure-witnesses.csv	the draw-level witness of each lost replicate. Present only when a property fit failed

A study page’s Reproduction section adds only what is specific to that study.

Property cells that lost a replicate

A property fit that raises a recordable error costs its cells one replicate each. The study declares FAILURE_BOUND (0.01) before the run, and a cell may lose floor(bound * R) replicates. The property failure rules give the full contract. Read a cell in property-failure-accounting.csv with these columns.

column	what it holds
leg	failure bound, then one row for each clause of the cell’s family rule. A clause with no candidate set reads failures not accountable
complete_case, worst_case	the slack of the clause on the survivors and under its least favourable fill. A negative value fails
imputation	the label of the leg. support fills a bounded variable over its full support and needs no assumption. range assumes that a lost value lies inside the survivor range, with the scale of the margin held
reason	within bound, bound exceeded, worst case fails, complete case fails, or failures not accountable
informative-failure witness rows	a statistic of the lost draw and its quantile among the survivors’ draws. The witness is a report. No gate reads it

A green cell with a range leg rests on the range assumption. A green cell with only support legs does not. Three clauses are not accountable: natural_course_identity, static_reduction, and the closed-score clause of correction_necessity. A lost replicate turns each of them red at any bound.

Terms

term	definition
replication	one simulated dataset drawn from a study’s law, fitted by every implementation. Seeds derive from the study’s own record, so replication k is a fixed sample whatever the study’s size
law	the data-generating process a scenario samples, with a parameter computed longhand rather than estimated
estimand	the parameter a row reports. Longitudinal keys carry their regimen in brackets
cell	one configuration of a property family, labelled positive when it must pass the family’s instrument and control when it must fail it
margin	an acceptance threshold declared before the run and recorded in the study’s manifest.json. Names beginning margin: in a measured-values table are these thresholds
standardized bias	mean error divided by the empirical standard deviation of the estimates. Scale-free, so it reads the same across estimands
SE ratio	mean reported standard error divided by the empirical standard deviation. One means the reported uncertainty matches the real spread
share of margin used	how much of a similarity margin a paired comparison consumed. Small means the two implementations agree far inside the bound
plug-in	the estimate the same nuisance fits produce with no targeting fluctuation. The distance from it measures what targeting did
lost replicate	a replication whose fit raised a recordable error. The cell keeps the survivors, and the accounting judges the verdict under the worst case of the lost values
leg	one clause of a verdict, judged separately in the accounting. A support leg needs no assumption about the lost values. A range leg assumes that they lie inside the survivor range