Full-refit bootstrap and derived contrasts

This study measures four post-fit outputs against known truth. The outputs are the log-scale ratio of two regimen levels (LongitudinalResult.ratio), the RMST and its contrast (LongitudinalResult.rmst), the percentile interval of the LTMLE full-refit bootstrap, and the percentile interval of the point-treatment TMLE bootstrap. No canonical implementation is compared. No pinned comparator ships these outputs, so the empty equivalence artifact records the absence of a comparator.

The ratio and RMST are exact delta-method and linear functionals of shipped influence curves. tests/unit/test_contrast_conveniences.py checks their arithmetic on exact laws. Their log-scale small-sample coverage is a simulation claim, so this study measures it. The bootstrap has no source for this estimator. The study measures its percentile interval with correctly specified nuisances at \(n = 1000\). No result covers a data-adaptive nuisance, and Cai and van der Laan (2020) is the warning for that case.

What was tested

setting	declaration
primary scenario	the log risk ratio and log odds ratio of always versus never on the two-node end-of-study law, 1,000 replications at \(n = 2000\)
laws	the two-node end-of-study law of tests/discrete_law_longitudinal.py, the four-node survival law of tests/studies/survival_grid_law.py, a clustered draw of the end-of-study law, and the point law of tests/discrete_law.py
learners	saturated cell means on the end-of-study and point laws. On the four-node law, cell means over the columns each true conditional reads. Every learner is correctly specified
bootstrap	200 full-refit replicates per study replicate. Each cell reads the shipped estimate.bootstrap.ci and estimate.bootstrap.std_error, and the RMST cell reads result.rmst(...).bootstrap. Each fit runs with n_jobs=1
clustered draw	60 clusters of 25 rows. A cluster’s latent sign moves the outcome probability by \(\pm 0.1 (A_1 + A_2 - 1)\), which leaves every regimen mean unchanged
replications	4,000 per property cell, chosen before the run so that a cell whose reported standard error is 2% short passes the calibration rule with probability of at least 0.95 (calibration_pass_probability). At 1,000 that probability is about 0.3
verdicts	every cell is an interval_calibration cell under the shared margins. A positive cell needs its SE-ratio and coverage intervals inside the calibration bands. A shrunken control multiplies the standard error, or the percentile interval about its median, by 0.70 and must fall below the band
failed-replicate cap	a bootstrap cell whose mean failed share of bootstrap replicates exceeds 1% reports bootstrap_conditional and claims no coverage

The cells and their sizes are the table.

cells	law	folds	n	replications
log risk and odds ratio, end of study	end of study	1	2,000	4,000
log risk ratio and log survival ratio at t = 4, RMST and RMST contrast up to t = 5	four-node survival	1	2,000	4,000
RMST and RMST contrast up to t = 5	four-node survival	5	2,000	4,000
bootstrap of the always mean and the contrast	end of study	1 and 5	1,000	4,000 each
bootstrap of the risks at t = 2 and 4 and the RMST	four-node survival	1	1,000	4,000
cluster bootstrap of the always mean and the contrast	clustered	1	1,500	4,000
bootstrap of the point ATE	point	1	1,000	4,000

The end-of-study ratios appear twice. The primary table measures bias and coverage against the shared floor at 1,000 replications. The property cells measure calibration at 4,000. The grid row cites the property cells for the calibration claim.

Red-cell policy

The policy was declared before the run, in RED_CELL_POLICY of tests/studies/canonical_full_refit_bootstrap.py.

rule	when it applies	what happens
1	always	the study publishes under reporting. No budget, margin, law or learner changes after a verdict is seen
2	a red ratio or RMST cell	the exact-law tests are checked first. A defect is fixed and the study is regenerated once. A red cell that the exact tests cannot explain is published red with owner X20-derived
3	a longitudinal bootstrap cell	a design kind enters LICENSED_BOOTSTRAP_DESIGNS only when all its bootstrap cells are green. A red cell keeps its own kind a diagnostic, with owner X20-bootstrap, and moves no other kind. This is the rule as declared. Since RM42b the bootstrap kind of a support row does this job, and a kind without a row prints experimental
4	a red point-TMLE bootstrap cell	the cell is published red with owner X20-point-bootstrap, and rule 3 applies to TMLEResult
5	a control that does not fail	the control is reported as underpowered by design, and its positive cell claims no verdict

Accuracy against known truth

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
two-node end-of-study law, saturated cell-mean learners	or_regimen[always vs never]	odds ratio, with inference on the log scale, between the plans “treat at both times” against “treat at neither time”	cleverly LTMLE ratio contrast	-0.0176 to 0.0214	0.9660	1.0171	pass
two-node end-of-study law, saturated cell-mean learners	rr_regimen[always vs never]	risk ratio, with inference on the log scale, between the plans “treat at both times” against “treat at neither time”	cleverly LTMLE ratio contrast	-0.0104 to 0.0060	0.9640	1.0202	pass

Theory properties

property	cell	role	what was tested	what must hold	measured	result
interval_calibration	boot_ate_clustered__correctly_specified	positive	cluster-bootstrap percentile interval of the contrast, 60 clusters: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9123 to 0.9342, SE ratio 0.9496 to 1.0065	fail
interval_calibration	boot_ate_crossfit__correctly_specified	positive	bootstrap percentile interval of the contrast, five folds: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9741 to 0.9857, SE ratio 1.2019 to 1.3336	fail
interval_calibration	boot_ate_end_of_study__correctly_specified	positive	bootstrap percentile interval of the always-never contrast: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9254 to 0.9456, SE ratio 0.9443 to 1.0028	pass
interval_calibration	boot_ate_end_of_study__shrunken_se_control	control	bootstrap percentile interval of the always-never contrast: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.7920 to 0.8243, SE ratio 0.6606 to 0.7022	pass
interval_calibration	boot_ate_point_tmle__correctly_specified	positive	bootstrap percentile interval of the point-treatment ATE: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9273 to 0.9472, SE ratio 0.9518 to 1.0077	pass
interval_calibration	boot_ey_clustered__correctly_specified	positive	cluster-bootstrap percentile interval of the always mean, 60 clusters: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9219 to 0.9426, SE ratio 0.9618 to 1.0199	pass
interval_calibration	boot_ey_crossfit__correctly_specified	positive	bootstrap percentile interval of the always mean, five folds: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9730 to 0.9848, SE ratio 1.1663 to 1.3066	fail
interval_calibration	boot_ey_end_of_study__correctly_specified	positive	bootstrap percentile interval of the always mean, end-of-study law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9208 to 0.9416, SE ratio 0.9423 to 1.0034	pass
interval_calibration	boot_ey_end_of_study__shrunken_se_control	control	bootstrap percentile interval of the always mean, end-of-study law: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.7938 to 0.8260, SE ratio 0.6592 to 0.7009	pass
interval_calibration	boot_risk2_survival__correctly_specified	positive	bootstrap percentile interval of the always risk at t = 2: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9307 to 0.9502, SE ratio 0.9695 to 1.0289	pass
interval_calibration	boot_risk4_survival__correctly_specified	positive	bootstrap percentile interval of the always risk at t = 4: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9289 to 0.9486, SE ratio 0.9601 to 1.0178	pass
interval_calibration	boot_rmst_survival__correctly_specified	positive	bootstrap percentile interval of the always RMST up to t = 5: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9240 to 0.9444, SE ratio 0.9594 to 1.0192	pass
interval_calibration	or_end_of_study__correctly_specified	positive	log odds ratio of always versus never, end-of-study law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9367 to 0.9553, SE ratio 0.9477 to 1.0046	pass
interval_calibration	or_end_of_study__shrunken_se_control	control	log odds ratio of always versus never, end-of-study law: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8067 to 0.8380, SE ratio 0.6629 to 0.7030	pass
interval_calibration	rmst_contrast_crossfit__correctly_specified	positive	RMST contrast up to t = 5, four-node law, five folds: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9456 to 0.9628, SE ratio 1.0070 to 1.0655	pass
interval_calibration	rmst_contrast_survival__correctly_specified	positive	RMST contrast of always versus never up to t = 5, four-node law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9369 to 0.9555, SE ratio 0.9767 to 1.0349	pass
interval_calibration	rmst_crossfit__correctly_specified	positive	RMST of always up to t = 5, four-node law, five folds: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9434 to 0.9610, SE ratio 1.0084 to 1.0685	pass
interval_calibration	rmst_survival__correctly_specified	positive	RMST of always up to t = 5, four-node survival law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9394 to 0.9575, SE ratio 0.9704 to 1.0256	pass
interval_calibration	rr_end_of_study__correctly_specified	positive	log risk ratio of always versus never, end-of-study law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9391 to 0.9573, SE ratio 0.9536 to 1.0105	pass
interval_calibration	rr_end_of_study__shrunken_se_control	control	log risk ratio of always versus never, end-of-study law: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8059 to 0.8373, SE ratio 0.6673 to 0.7080	pass
interval_calibration	rr_survival__correctly_specified	positive	log risk ratio of always versus never at t = 4, four-node survival law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9364 to 0.9550, SE ratio 0.9659 to 1.0246	pass
interval_calibration	rr_survival__shrunken_se_control	control	log risk ratio of always versus never at t = 4, four-node survival law: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8188 to 0.8494, SE ratio 0.6760 to 0.7166	pass
interval_calibration	survival_rr_survival__correctly_specified	positive	log ratio of the survival probabilities of always versus never at t = 4, four-node law: all four required nuisance functions are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9372 to 0.9557, SE ratio 0.9637 to 1.0236	pass

Result

Every ratio and RMST cell passed, and every shrunken control fell below the band. The in-sample end-of-study, in-sample survival and point-treatment bootstrap cells passed. So the support rows G01, G11.nodes4 and T01.binary list the bootstrap kind. The point cell is an in-sample TMLE fit, so the row covers the in-sample point TMLE fit only. A cross-fitted point bootstrap, a clustered bootstrap and every DRTMLE bootstrap keep the interval names and carry the label experimental. CTMLE prints every bootstrap as a diagnostic. tests/unit/test_ltmle_bootstrap.py checks each setting. This study measured only correctly specified nuisances, and roadmap F2 holds the data-adaptive case.

The table above is the policy declared before the run. LICENSED_BOOTSTRAP_DESIGNS, the set rule 3 names, no longer exists. The bootstrap kind of a support row now plays its role, as the paragraph above states.

Three cells are red, and rule 3 applies to each. The two cross-fitted end-of-study cells over-cover, with SE ratios near 1.25. The cluster contrast covers 0.924, with a 99% interval below the band, while the cluster mean passes. So end_of_study/cross_fit and end_of_study/cluster stay diagnostic, and the roadmap owner X20-bootstrap holds the three cells.

A diagnostic reading, which is not a registered result, explains the cross-fitted cells. On this cell the cross-fitted estimator is heavy-tailed. Its 4,000 committed estimates have excess kurtosis 6.7, against 0.1 in sample. About 5% of them lie at the probability boundary, because g_bounds=(1e-8, 1) with saturated cell means leaves sparse training cells unbounded. A bootstrap resample holds about 63% unique units, so more replicates reach the boundary, and the replicate spread exceeds the sampling spread. A probe that kept each unit’s copies in one fold raised the SE ratio to about 1.65, so the split of copies across folds is not the cause.

The cluster contrast has a calibrated SE ratio of 0.977, so its shortfall reads as a finite-sample limit of the percentile interval at 60 clusters.

Measured values

Names beginning margin: are thresholds declared before the run. Everything else is measured from the committed results and checked at the precision printed.

quantity	value	source
replicates	1000	primary replications
n	2000	observations per primary replication
independent_tests_total	2	primary ratio tests against truth
independent_tests_passed	2	of those, passing
paired_tests_total	0	external comparisons declared
paired_tests_passed	0	external comparisons passing
property_cells_total	23	repeated-sampling property cells
property_cells_passed	20	cells whose own and family verdicts pass
max_standardized_bias	0.0219	largest absolute primary bias in empirical standard deviations
min_coverage	0.9640	lowest measured primary-study coverage
min_coverage_ci_lower	0.9460	lowest exact 99% primary coverage endpoint
min_se_ratio_ci_lower	0.9609	lowest bootstrap primary SE-ratio endpoint
max_se_ratio_ci_upper	1.0809	highest bootstrap primary SE-ratio endpoint
margin:confidence_level	0.9900	confidence level of every Monte Carlo interval
margin:alpha	0.0500	nominal size of the reported intervals
margin:nominal_coverage	0.9500	nominal coverage those intervals claim
margin:bootstrap_replicates	10000	resamples behind every bootstrap interval
margin:standardized_bias	0.2500	bias equivalence margin, in empirical standard deviations
margin:coverage_floor	0.9000	validity floor the coverage lower endpoint must clear
margin:over_coverage_ceiling	0.9900	above this, coverage is conservative rather than invalid
margin:se_ratio_sanity_lower	0.8000	SE-ratio screen, lower limit
margin:se_ratio_sanity_upper	1.2000	SE-ratio screen, upper limit
margin:calibration_se_ratio_lower	0.9300	calibration-cell SE-ratio band, lower limit
margin:calibration_se_ratio_upper	1.0700	calibration-cell SE-ratio band, upper limit. A control must rise above it
margin:calibration_coverage_lower	0.9200	calibration-cell coverage band, lower limit
margin:calibration_coverage_upper	0.9800	calibration-cell coverage band, upper limit
margin:type_i_ceiling	0.1000	largest size a one-sided type-I bound may establish
margin:paired_difference	0.1500	paired similarity margin, in pooled empirical standard deviations
margin:rmse_noninferiority	1.1000	largest external-comparison RMSE ratio bound
margin:coverage_noninferiority	-0.0250	smallest external-comparison coverage difference bound
margin:calibration_noninferiority	0.0500	largest external-comparison calibration excess bound
margin:minimum_power	0.8000	rejection lower bound a power control must clear
margin:root_n_slope	-0.5000	contraction rate root-n asymptotics predict. No cell of this study reads it
margin:root_n_slope_lower	-0.6250	accepted root-n slope band, lower limit
margin:root_n_slope_upper	-0.3750	accepted root-n slope band, upper limit
margin:excluded_slope	-0.2500	slower rate a root-n interval must exclude
properties[interval_calibration/boot_ey_crossfit__correctly_specified]:se_ratio	1.2310	SE ratio, cross-fitted bootstrap of the always mean
properties[interval_calibration/boot_ate_crossfit__correctly_specified]:se_ratio	1.2652	SE ratio, cross-fitted bootstrap of the contrast
properties[interval_calibration/boot_ate_crossfit__correctly_specified]:coverage	0.9805	coverage, cross-fitted bootstrap of the contrast
properties[interval_calibration/boot_ate_clustered__correctly_specified]:coverage	0.9237	coverage, cluster bootstrap of the contrast
properties[interval_calibration/boot_ate_clustered__correctly_specified]:coverage_ci_lower	0.9123	its 99% lower endpoint
properties[interval_calibration/boot_risk4_survival__correctly_specified]:bootstrap_failed_fraction	0.0025	mean failed share of bootstrap replicates, survival
properties[interval_calibration/boot_ate_point_tmle__correctly_specified]:coverage	0.9377	coverage, point-treatment bootstrap of the ATE

Limits

limit	what it means
correctly specified nuisances only	the bootstrap interval is measured with known-form learners. No result covers a data-adaptive nuisance
one size per cell	each bootstrap cell runs at one \(n\). A licensed kind is licensed by design, and the study does not measure it at other sizes
unmeasured compositions	a working model, competing risks, weights, a dynamic rule, a Gaussian outcome and a cross-fitted survival fit have no licensed kind, so their bootstrap prints as a diagnostic
finite laws	every law has binary nodes and finite support
static regimens	the study fits never and always. A dynamic rule is not measured by these cells
node units	RMST is measured in node units on an equal grid

Reproduction

uv run --extra dev python -m tests.canonical.full_refit_bootstrap.regenerate