Stacked arm-indexed missing-outcome CV-TMLE#

This study validates the stacked CV-TMLE of arm-indexed means and contrasts under MAR. Stacked CV-TMLE for arm-indexed targets states the contract. The table gives the studied construction. Every fit declares it through the public TMLEMethod, CrossFitting, Targeting, and Inference API.

component	studied setting
partition	one package-generated, unstratified ten-fold split
outcome regression	fitted on the respondents in each training complement
treatment and response mechanisms	fitted separately on each training complement
targeting	one pooled logistic fluctuation with one column for each arm
evaluation	plug-in and influence curve over the whole sample
inference	pointwise Wald intervals from the centered influence-curve variance; ratios on the log scale
primary learners	separate depth-five trees with minimum leaf size 25 for each nuisance

Laws#

tests/studies/mar_arm_indexed_laws.py declares four laws. Each law has a three-level covariate W, and each nuisance is a table indexed by W and the arm. The smallest product g_a pi_a is 0.12 in L1 and L2, and 0.1125 in L3 and L4.

law	arms	outcome	estimands
L1	2	binary	`ey0`, `ey1`, `ate`, `rr`, and `or`
L2	2	`-2 + 10 B`, with `B` Beta-distributed about the L1 mean	`ey0`, `ey1`, and `ate`
L3	3	binary	`ey[a]` for each arm; `ate`, `rr`, and `or` against `high`
L4	3	`-2 + 10 B`, with `B` Beta-distributed about the L3 mean	`ey[a]` for each arm; `ate` against `high`

A continuous fit declares q_bounds=(-2, 8), the known support. Each law has nine or fewer covariate cells for the outcome regression. Each depth-five tree can therefore fit the saturated model, and that model is correct. The primary and calibration rows thus do not test calibration under a misspecified data-adaptive learner. The paired overfitting cells supply that evidence.

Comparator#

The study compares R tmle 2.1.1. The adapter supplies the same stitched out-of-fold outcome, treatment, and response predictions that cleverly targets. L1 and L2 use the native two-arm path. L3 and L4 use the population-mean path once for each arm. For arm a, the adapter passes a constant A, the indicator 1{A = a} Delta as Delta, and NA for every other outcome. It builds the joint covariance and each reference contrast from the three returned IC.EY1 curves.

R takes a continuous outcome scale from every non-NA Y (tmle.R line 1120). A continuous fit therefore sets Y to -2 and 8 on two rows whose response indicator is zero. Those rows leave the fluctuation and the curves. probe_scale_workaround.R checks the workaround on replication 0 of each continuous law, and it writes scale-probe.csv.

probe check	requirement
scale with the planted rows	equal to `q_bounds` exactly
scale without the planted rows	different from `q_bounds`, so the workaround does work
planted rows moved to two other eligible rows	every point and curve value bitwise unchanged
independent rebuild at the `q_bounds` scale	points and curves within `1e-12`

The regeneration refuses publication when a probe row fails. Every committed probe row passes.

This comparison conditions on the supplied nuisance predictions. It validates the pooled targeting, plug-in, influence curves, and joint covariance. It does not validate R-native cross-fit training. docs/references.md records the locator for each candidate below.

The paired tests alone cannot show that cleverly targets. The saturated trees leave the initial plug-in almost unbiased, so an estimate without its targeting step still passes every paired test and every truth test. The claim that cleverly reproduces the R targeting step rests on a separate check over the committed replications.

check	requirement
targeting move, `estimate - initial_estimate`	nonzero in every replication and estimand
`abs(cleverly - R) / abs(targeting move)`	below `1e-2` in every replication and estimand
initial estimates of the two implementations	equal to `1e-12`
mutation: `cleverly` reports its initial estimate	the ratio exceeds `0.9` in every row, so the check fails

R tmle stops its fluctuation glm at a relative deviance change of 1e-8. That stop resolves the fluctuation coefficient to about 1e-4 of its scale, and the bound allows a hundredfold over that. tests/unit/test_arm_indexed_cvtmle_method_study.py runs the check and the mutation.

candidate	disposition
R `tmle` 2.1.1	compared through its two-arm path for L1 and L2, and its population-mean path for each arm of L3 and L4
`tmle3` at commit `ed72f8a`	its generic treatment-specific outcome fit can use the `Delta = 0` pseudo-outcomes when it predicts under `Delta = 1`. `cleverly` fits that regression on respondents only
zEpid 0.9.1	its cross-fit TMLE targets inside each fold rather than with one pooled fluctuation
Newey and Robins (2018)	the construction fits the outcome and inverse response regressions on distinct subsamples. That is a different estimator

source	locator	what it supplies
Díaz and van der Laan (2017)	Section 2.1 and Equation (1)	the per-arm curve with missing outcomes
Gruber and van der Laan (2012)	Section 2.3	the clever covariate for a generic arm with a treatment mechanism conditional on `W`
Gruber and van der Laan (2012)	Appendix A	the delta-method form of each contrast
Zheng and van der Laan (2011)	Section 2 and Theorem 2	the vector parameter, the training-complement fits, and the joint expansion
Levy (2018)	Section 3.1	the stacked update and the whole-sample plug-in

Accuracy against known truth#

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
L3: three-arm binary-outcome law with MAR outcomes	`ate[low vs high]`	difference in counterfactual means, low versus high	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0041 to 0.0027	0.9550	1.0352	pass
L3: three-arm binary-outcome law with MAR outcomes	`ate[low vs high]`	difference in counterfactual means, low versus high	R `tmle` two-arm path, or its population-mean path once per arm	-0.0041 to 0.0027	0.9550	1.0352	pass
L3: three-arm binary-outcome law with MAR outcomes	`ate[mid vs high]`	difference in counterfactual means, mid versus high	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0044 to 0.0021	0.9613	1.0268	pass
L3: three-arm binary-outcome law with MAR outcomes	`ate[mid vs high]`	difference in counterfactual means, mid versus high	R `tmle` two-arm path, or its population-mean path once per arm	-0.0044 to 0.0021	0.9613	1.0268	pass
L3: three-arm binary-outcome law with MAR outcomes	`ey[high]`	counterfactual mean under treatment arm ‘high’	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0012 to 0.0036	0.9575	1.0236	pass
L3: three-arm binary-outcome law with MAR outcomes	`ey[high]`	counterfactual mean under treatment arm ‘high’	R `tmle` two-arm path, or its population-mean path once per arm	-0.0012 to 0.0036	0.9575	1.0236	pass
L3: three-arm binary-outcome law with MAR outcomes	`ey[low]`	counterfactual mean under treatment arm ‘low’	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0020 to 0.0029	0.9450	1.0318	pass
L3: three-arm binary-outcome law with MAR outcomes	`ey[low]`	counterfactual mean under treatment arm ‘low’	R `tmle` two-arm path, or its population-mean path once per arm	-0.0020 to 0.0029	0.9450	1.0318	pass
L3: three-arm binary-outcome law with MAR outcomes	`ey[mid]`	counterfactual mean under treatment arm ‘mid’	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0022 to 0.0022	0.9437	1.0215	pass
L3: three-arm binary-outcome law with MAR outcomes	`ey[mid]`	counterfactual mean under treatment arm ‘mid’	R `tmle` two-arm path, or its population-mean path once per arm	-0.0022 to 0.0022	0.9437	1.0215	pass
L3: three-arm binary-outcome law with MAR outcomes	`or[low vs high]`	marginal odds ratio, low versus high, reported on the log scale	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0229 to 0.0074	0.9613	1.0329	pass
L3: three-arm binary-outcome law with MAR outcomes	`or[low vs high]`	marginal odds ratio, low versus high, reported on the log scale	R `tmle` two-arm path, or its population-mean path once per arm	-0.0229 to 0.0074	0.9613	1.0329	pass
L3: three-arm binary-outcome law with MAR outcomes	`or[mid vs high]`	marginal odds ratio, mid versus high, reported on the log scale	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0223 to 0.0065	0.9613	1.0244	pass
L3: three-arm binary-outcome law with MAR outcomes	`or[mid vs high]`	marginal odds ratio, mid versus high, reported on the log scale	R `tmle` two-arm path, or its population-mean path once per arm	-0.0223 to 0.0065	0.9613	1.0244	pass
L3: three-arm binary-outcome law with MAR outcomes	`rr[low vs high]`	marginal risk ratio, low versus high, reported on the log scale	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0095 to 0.0050	0.9537	1.0360	pass
L3: three-arm binary-outcome law with MAR outcomes	`rr[low vs high]`	marginal risk ratio, low versus high, reported on the log scale	R `tmle` two-arm path, or its population-mean path once per arm	-0.0095 to 0.0050	0.9537	1.0360	pass
L3: three-arm binary-outcome law with MAR outcomes	`rr[mid vs high]`	marginal risk ratio, mid versus high, reported on the log scale	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0073 to 0.0034	0.9550	1.0305	pass
L3: three-arm binary-outcome law with MAR outcomes	`rr[mid vs high]`	marginal risk ratio, mid versus high, reported on the log scale	R `tmle` two-arm path, or its population-mean path once per arm	-0.0073 to 0.0034	0.9550	1.0305	pass
L1: two-arm binary-outcome law with MAR outcomes	`ate`	average treatment effect	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0037 to 0.0023	0.9500	1.0345	pass
L1: two-arm binary-outcome law with MAR outcomes	`ate`	average treatment effect	R `tmle` two-arm path, or its population-mean path once per arm	-0.0037 to 0.0023	0.9500	1.0345	pass
L1: two-arm binary-outcome law with MAR outcomes	`ey0`	counterfactual mean under no treatment	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0017 to 0.0028	0.9437	0.9731	pass
L1: two-arm binary-outcome law with MAR outcomes	`ey0`	counterfactual mean under no treatment	R `tmle` two-arm path, or its population-mean path once per arm	-0.0017 to 0.0028	0.9437	0.9731	pass
L1: two-arm binary-outcome law with MAR outcomes	`ey1`	counterfactual mean under treatment	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0023 to 0.0020	0.9425	1.0062	pass
L1: two-arm binary-outcome law with MAR outcomes	`ey1`	counterfactual mean under treatment	R `tmle` two-arm path, or its population-mean path once per arm	-0.0023 to 0.0020	0.9425	1.0062	pass
L1: two-arm binary-outcome law with MAR outcomes	`or`	marginal odds ratio, reported on the log scale	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0132 to 0.0119	0.9525	1.0341	pass
L1: two-arm binary-outcome law with MAR outcomes	`or`	marginal odds ratio, reported on the log scale	R `tmle` two-arm path, or its population-mean path once per arm	-0.0132 to 0.0119	0.9525	1.0341	pass
L1: two-arm binary-outcome law with MAR outcomes	`rr`	marginal risk ratio, reported on the log scale	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0070 to 0.0064	0.9500	1.0230	pass
L1: two-arm binary-outcome law with MAR outcomes	`rr`	marginal risk ratio, reported on the log scale	R `tmle` two-arm path, or its population-mean path once per arm	-0.0070 to 0.0064	0.9500	1.0230	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ate[low vs high]`	difference in counterfactual means, low versus high	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0237 to 0.0095	0.9387	0.9653	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ate[low vs high]`	difference in counterfactual means, low versus high	R `tmle` two-arm path, or its population-mean path once per arm	-0.0237 to 0.0095	0.9387	0.9653	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ate[mid vs high]`	difference in counterfactual means, mid versus high	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0148 to 0.0180	0.9475	0.9939	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ate[mid vs high]`	difference in counterfactual means, mid versus high	R `tmle` two-arm path, or its population-mean path once per arm	-0.0148 to 0.0180	0.9475	0.9939	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[high]`	counterfactual mean under treatment arm ‘high’	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0056 to 0.0171	0.9425	0.9927	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[high]`	counterfactual mean under treatment arm ‘high’	R `tmle` two-arm path, or its population-mean path once per arm	-0.0056 to 0.0171	0.9425	0.9927	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[low]`	counterfactual mean under treatment arm ‘low’	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0133 to 0.0106	0.9300	0.9565	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[low]`	counterfactual mean under treatment arm ‘low’	R `tmle` two-arm path, or its population-mean path once per arm	-0.0133 to 0.0106	0.9300	0.9565	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[mid]`	counterfactual mean under treatment arm ‘mid’	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0032 to 0.0178	0.9487	1.0190	pass
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[mid]`	counterfactual mean under treatment arm ‘mid’	R `tmle` two-arm path, or its population-mean path once per arm	-0.0032 to 0.0178	0.9487	1.0190	pass
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ate`	average treatment effect	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0114 to 0.0191	0.9413	0.9797	pass
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ate`	average treatment effect	R `tmle` two-arm path, or its population-mean path once per arm	-0.0114 to 0.0191	0.9413	0.9797	pass
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ey0`	counterfactual mean under no treatment	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0142 to 0.0057	0.9537	1.0180	pass
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ey0`	counterfactual mean under no treatment	R `tmle` two-arm path, or its population-mean path once per arm	-0.0142 to 0.0057	0.9537	1.0180	pass
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ey1`	counterfactual mean under treatment	`cleverly` stacked arm-indexed missing-outcome CV-TMLE	-0.0112 to 0.0104	0.9487	0.9806	pass
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ey1`	counterfactual mean under treatment	R `tmle` two-arm path, or its population-mean path once per arm	-0.0112 to 0.0104	0.9487	0.9806	pass

Agreement with the canonical implementation#

Comparator-backed: a paired comparison with tmle-r-stitched-arm-indexed on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.

law	estimand	what was compared	paired difference	share of margin used	RMSE ratio bound	comparison kind	calibration resolution	result
L3: three-arm binary-outcome law with MAR outcomes	`ate[low vs high]`	difference in counterfactual means, low versus high	-4.105e-10	7.373e-08	1.0000	related target	1.793e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`ate[mid vs high]`	difference in counterfactual means, mid versus high	-1.973e-10	3.676e-08	1.0000	related target	2.471e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`ey[high]`	counterfactual mean under treatment arm ‘high’	2.215e-10	5.617e-08	1.0000	matching implementation	3.387e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`ey[low]`	counterfactual mean under treatment arm ‘low’	-1.890e-10	4.745e-08	1.0000	matching implementation	1.410e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`ey[mid]`	counterfactual mean under treatment arm ‘mid’	2.416e-11	6.736e-09	1.0000	matching implementation	1.773e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`or[low vs high]`	marginal odds ratio, low versus high, reported on the log scale	-5.201e-10	7.413e-08	1.0000	related target	2.123e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`or[mid vs high]`	marginal odds ratio, mid versus high, reported on the log scale	-4.952e-10	3.866e-08	1.0000	related target	2.855e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`rr[low vs high]`	marginal risk ratio, low versus high, reported on the log scale	-4.554e-10	6.914e-08	1.0000	related target	1.803e-09 vs 0.0500	equivalent
L3: three-arm binary-outcome law with MAR outcomes	`rr[mid vs high]`	marginal risk ratio, mid versus high, reported on the log scale	-2.176e-10	3.121e-08	1.0000	related target	2.082e-09 vs 0.0500	equivalent
L1: two-arm binary-outcome law with MAR outcomes	`ate`	average treatment effect	9.460e-10	1.924e-07	1.0000	matching implementation	4.979e-09 vs 0.0500	equivalent
L1: two-arm binary-outcome law with MAR outcomes	`ey0`	counterfactual mean under no treatment	-3.362e-10	9.074e-08	1.0000	matching implementation	4.969e-09 vs 0.0500	equivalent
L1: two-arm binary-outcome law with MAR outcomes	`ey1`	counterfactual mean under treatment	6.099e-10	1.727e-07	1.0000	matching implementation	4.987e-09 vs 0.0500	equivalent
L1: two-arm binary-outcome law with MAR outcomes	`or`	marginal odds ratio, reported on the log scale	8.961e-09	1.837e-07	1.0000	matching implementation	4.783e-09 vs 0.0500	equivalent
L1: two-arm binary-outcome law with MAR outcomes	`rr`	marginal risk ratio, reported on the log scale	2.948e-09	1.713e-07	1.0000	matching implementation	4.019e-09 vs 0.0500	equivalent
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ate[low vs high]`	difference in counterfactual means, low versus high	-5.692e-10	2.083e-08	1.0000	related target	1.011e-09 vs 0.0500	equivalent
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ate[mid vs high]`	difference in counterfactual means, mid versus high	3.131e-10	1.162e-08	1.0000	related target	1.915e-10 vs 0.0500	equivalent
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[high]`	counterfactual mean under treatment arm ‘high’	-4.417e-13	2.375e-11	1.0000	matching implementation	5.510e-13 vs 0.0500	equivalent
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[low]`	counterfactual mean under treatment arm ‘low’	-5.697e-10	2.898e-08	1.0000	matching implementation	1.762e-09 vs 0.0500	equivalent
L4: three-arm bounded continuous-outcome law with MAR outcomes	`ey[mid]`	counterfactual mean under treatment arm ‘mid’	3.126e-10	1.808e-08	1.0000	matching implementation	4.701e-09 vs 0.0500	equivalent
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ate`	average treatment effect	1.598e-09	6.375e-08	1.0000	matching implementation	1.537e-08 vs 0.0500	equivalent
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ey0`	counterfactual mean under no treatment	-7.787e-10	4.754e-08	1.0000	matching implementation	9.779e-10 vs 0.0500	equivalent
L2: two-arm bounded continuous-outcome law with MAR outcomes	`ey1`	counterfactual mean under treatment	8.193e-10	4.628e-08	1.0000	matching implementation	7.143e-09 vs 0.0500	equivalent

In this comparator-backed study, both implementations use the centered, n - 1 divisor variance var(IC) / n. The R two-arm path bounds a binary mean interval to [0, 1] and an ATE interval to [-1, 1]. These laws stay away from those bounds. For L3 and L4 the adapter forms Wald intervals from the per-arm curves.

Theory properties#

property	cell	role	what was tested	what must hold	measured	result
`crossfit_overfitting`	`l1_ate__in_sample_control`	control	L1 average treatment effect: the same flexible learner fitted in sample, with no cross-fitting	SE ratio must fall below the overfitting ceiling	SE ratio 0.1100 to 0.1371	pass
`crossfit_overfitting`	`l1_ate__stacked_arm_indexed_cvtmle`	positive	L1 average treatment effect: stacked arm-indexed MAR CV-TMLE with a fully grown outcome tree	SE ratio clears the overfitting floor and stays inside the sanity band	SE ratio 0.8724 to 1.0501	pass
`crossfit_overfitting`	`l2_ate__in_sample_control`	control	L2 average treatment effect: the same flexible learner fitted in sample, with no cross-fitting	SE ratio must fall below the overfitting ceiling	SE ratio 0.1666 to 0.2116	pass
`crossfit_overfitting`	`l2_ate__stacked_arm_indexed_cvtmle`	positive	L2 average treatment effect: stacked arm-indexed MAR CV-TMLE with a fully grown outcome tree	SE ratio clears the overfitting floor and stays inside the sanity band	SE ratio 0.8550 to 1.0245	pass
`crossfit_overfitting`	`l3_ate_low__in_sample_control`	control	L3 difference, low versus high: the same flexible learner fitted in sample, with no cross-fitting	SE ratio must fall below the overfitting ceiling	SE ratio 0.1454 to 0.1820	pass
`crossfit_overfitting`	`l3_ate_low__stacked_arm_indexed_cvtmle`	positive	L3 difference, low versus high: stacked arm-indexed MAR CV-TMLE with a fully grown outcome tree	SE ratio clears the overfitting floor and stays inside the sanity band	SE ratio 0.9195 to 1.1130	pass
`crossfit_overfitting`	`l4_ate_low__in_sample_control`	control	L4 difference, low versus high: the same flexible learner fitted in sample, with no cross-fitting	SE ratio must fall below the overfitting ceiling	SE ratio 0.2333 to 0.3088	pass
`crossfit_overfitting`	`l4_ate_low__stacked_arm_indexed_cvtmle`	positive	L4 difference, low versus high: stacked arm-indexed MAR CV-TMLE with a fully grown outcome tree	SE ratio clears the overfitting floor and stays inside the sanity band	SE ratio 0.8759 to 1.0735	pass
`interval_calibration`	`l1_ate__learned_nuisances`	positive	L1 average treatment effect: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9361 to 0.9617, SE ratio 0.9413 to 1.0198	pass
`interval_calibration`	`l1_ate__shrunken_se_control`	control	L1 average treatment effect: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8058 to 0.8497, SE ratio 0.6594 to 0.7139	pass
`interval_calibration`	`l1_ey0__learned_nuisances`	positive	L1 mean under arm 0: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9311 to 0.9578, SE ratio 0.9465 to 1.0261	pass
`interval_calibration`	`l1_ey0__shrunken_se_control`	control	L1 mean under arm 0: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.7943 to 0.8392, SE ratio 0.6620 to 0.7170	pass
`interval_calibration`	`l1_ey1__learned_nuisances`	positive	L1 mean under arm 1: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9316 to 0.9582, SE ratio 0.9549 to 1.0399	pass
`interval_calibration`	`l1_ey1__shrunken_se_control`	control	L1 mean under arm 1: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.7953 to 0.8402, SE ratio 0.6681 to 0.7282	pass
`interval_calibration`	`l1_or__learned_nuisances`	positive	L1 log odds ratio: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9361 to 0.9617, SE ratio 0.9416 to 1.0195	pass
`interval_calibration`	`l1_or__shrunken_se_control`	control	L1 log odds ratio: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8047 to 0.8487, SE ratio 0.6596 to 0.7136	pass
`interval_calibration`	`l1_rr__learned_nuisances`	positive	L1 log risk ratio: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9322 to 0.9586, SE ratio 0.9396 to 1.0186	pass
`interval_calibration`	`l1_rr__shrunken_se_control`	control	L1 log risk ratio: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8005 to 0.8449, SE ratio 0.6581 to 0.7130	pass
`interval_calibration`	`l2_ate__learned_nuisances`	positive	L2 average treatment effect: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9383 to 0.9635, SE ratio 0.9684 to 1.0485	pass
`interval_calibration`	`l2_ate__shrunken_se_control`	control	L2 average treatment effect: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8131 to 0.8563, SE ratio 0.6786 to 0.7348	pass
`interval_calibration`	`l2_ey0__learned_nuisances`	positive	L2 mean under arm 0: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9428 to 0.9670, SE ratio 0.9800 to 1.0614	pass
`interval_calibration`	`l2_ey0__shrunken_se_control`	control	L2 mean under arm 0: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8152 to 0.8582, SE ratio 0.6855 to 0.7429	pass
`interval_calibration`	`l2_ey1__learned_nuisances`	positive	L2 mean under arm 1: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9422 to 0.9665, SE ratio 0.9806 to 1.0627	pass
`interval_calibration`	`l2_ey1__shrunken_se_control`	control	L2 mean under arm 1: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8173 to 0.8601, SE ratio 0.6862 to 0.7442	pass
`interval_calibration`	`l3_ate_low__learned_nuisances`	positive	L3 difference, low versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9383 to 0.9635, SE ratio 0.9835 to 1.0666	pass
`interval_calibration`	`l3_ate_low__shrunken_se_control`	control	L3 difference, low versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8162 to 0.8591, SE ratio 0.6877 to 0.7468	pass
`interval_calibration`	`l3_ate_mid__learned_nuisances`	positive	L3 difference, mid versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9406 to 0.9652, SE ratio 0.9718 to 1.0570	pass
`interval_calibration`	`l3_ate_mid__shrunken_se_control`	control	L3 difference, mid versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8110 to 0.8544, SE ratio 0.6798 to 0.7413	pass
`interval_calibration`	`l3_ey_high__learned_nuisances`	positive	L3 mean under arm high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9294 to 0.9564, SE ratio 0.9517 to 1.0417	pass
`interval_calibration`	`l3_ey_high__shrunken_se_control`	control	L3 mean under arm high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8068 to 0.8506, SE ratio 0.6676 to 0.7280	pass
`interval_calibration`	`l3_ey_low__learned_nuisances`	positive	L3 mean under arm low: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9372 to 0.9626, SE ratio 0.9820 to 1.0616	pass
`interval_calibration`	`l3_ey_low__shrunken_se_control`	control	L3 mean under arm low: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8173 to 0.8601, SE ratio 0.6880 to 0.7442	pass
`interval_calibration`	`l3_ey_mid__learned_nuisances`	positive	L3 mean under arm mid: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9394 to 0.9643, SE ratio 0.9724 to 1.0565	pass
`interval_calibration`	`l3_ey_mid__shrunken_se_control`	control	L3 mean under arm mid: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8168 to 0.8596, SE ratio 0.6808 to 0.7400	pass
`interval_calibration`	`l3_or_low__learned_nuisances`	positive	L3 log odds ratio, low versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9389 to 0.9639, SE ratio 0.9826 to 1.0648	pass
`interval_calibration`	`l3_or_low__shrunken_se_control`	control	L3 log odds ratio, low versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8152 to 0.8582, SE ratio 0.6883 to 0.7452	pass
`interval_calibration`	`l3_or_mid__learned_nuisances`	positive	L3 log odds ratio, mid versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9445 to 0.9683, SE ratio 0.9693 to 1.0571	pass
`interval_calibration`	`l3_or_mid__shrunken_se_control`	control	L3 log odds ratio, mid versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8100 to 0.8534, SE ratio 0.6789 to 0.7411	pass
`interval_calibration`	`l3_rr_low__learned_nuisances`	positive	L3 log risk ratio, low versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9456 to 0.9691, SE ratio 0.9877 to 1.0688	pass
`interval_calibration`	`l3_rr_low__shrunken_se_control`	control	L3 log risk ratio, low versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8194 to 0.8619, SE ratio 0.6928 to 0.7493	pass
`interval_calibration`	`l3_rr_mid__learned_nuisances`	positive	L3 log risk ratio, mid versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9411 to 0.9657, SE ratio 0.9738 to 1.0627	pass
`interval_calibration`	`l3_rr_mid__shrunken_se_control`	control	L3 log risk ratio, mid versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8189 to 0.8615, SE ratio 0.6823 to 0.7432	pass
`interval_calibration`	`l4_ate_low__learned_nuisances`	positive	L4 difference, low versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9344 to 0.9604, SE ratio 0.9672 to 1.0461	pass
`interval_calibration`	`l4_ate_low__shrunken_se_control`	control	L4 difference, low versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8021 to 0.8463, SE ratio 0.6763 to 0.7331	pass
`interval_calibration`	`l4_ate_mid__learned_nuisances`	positive	L4 difference, mid versus high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9394 to 0.9643, SE ratio 0.9807 to 1.0590	pass
`interval_calibration`	`l4_ate_mid__shrunken_se_control`	control	L4 difference, mid versus high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8126 to 0.8558, SE ratio 0.6865 to 0.7419	pass
`interval_calibration`	`l4_ey_high__learned_nuisances`	positive	L4 mean under arm high: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9316 to 0.9582, SE ratio 0.9740 to 1.0564	pass
`interval_calibration`	`l4_ey_high__shrunken_se_control`	control	L4 mean under arm high: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8079 to 0.8515, SE ratio 0.6823 to 0.7382	pass
`interval_calibration`	`l4_ey_low__learned_nuisances`	positive	L4 mean under arm low: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9327 to 0.9591, SE ratio 0.9625 to 1.0433	pass
`interval_calibration`	`l4_ey_low__shrunken_se_control`	control	L4 mean under arm low: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8162 to 0.8591, SE ratio 0.6739 to 0.7320	pass
`interval_calibration`	`l4_ey_mid__learned_nuisances`	positive	L4 mean under arm mid: separate depth-five trees fit the outcome, treatment and observation nuisances out of fold; each law has nine or fewer covariate cells, so each tree can fit the saturated model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9411 to 0.9657, SE ratio 0.9783 to 1.0615	pass
`interval_calibration`	`l4_ey_mid__shrunken_se_control`	control	L4 mean under arm mid: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8126 to 0.8558, SE ratio 0.6854 to 0.7427	pass
`mar_robustness`	`l1_ate__only_outcome_wrong`	positive	L1 average treatment effect: only the outcome regression is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0036 to 0.0025, margin 0.0101, SE ratio 0.9953	pass
`mar_robustness`	`l1_ate__only_response_wrong`	positive	L1 average treatment effect: only the observation mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0026 to 0.0010, margin 0.0060, SE ratio 1.6416	pass
`mar_robustness`	`l1_ate__only_treatment_wrong`	positive	L1 average treatment effect: only the treatment mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0025 to 0.0020, margin 0.0075, SE ratio 1.4196	pass
`mar_robustness`	`l1_ate__outcome_and_response_wrong`	control	L1 average treatment effect: the outcome regression and the observation mechanism are both wrong	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias -0.4808 to -0.4769, margin 0.0066, SE ratio 1.7487	pass
`mar_robustness`	`l2_ate__only_outcome_wrong`	positive	L2 average treatment effect: only the outcome regression is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0437 to -0.0013, margin 0.0711, SE ratio 0.9797	pass
`mar_robustness`	`l2_ate__only_response_wrong`	positive	L2 average treatment effect: only the observation mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0174 to 0.0015, margin 0.0316, SE ratio 1.4776	pass
`mar_robustness`	`l2_ate__only_treatment_wrong`	positive	L2 average treatment effect: only the treatment mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0074 to 0.0154, margin 0.0382, SE ratio 1.3065	pass
`mar_robustness`	`l2_ate__outcome_and_response_wrong`	control	L2 average treatment effect: the outcome regression and the observation mechanism are both wrong	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias -4.8100 to -4.7846, margin 0.0425, SE ratio 1.7963	pass
`mar_robustness`	`l3_ate_low__only_outcome_wrong`	positive	L3 difference, low versus high: only the outcome regression is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0039 to 0.0025, margin 0.0109, SE ratio 0.9868	pass
`mar_robustness`	`l3_ate_low__only_response_wrong`	positive	L3 difference, low versus high: only the observation mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0027 to 0.0022, margin 0.0082, SE ratio 1.7628	pass
`mar_robustness`	`l3_ate_low__only_treatment_wrong`	positive	L3 difference, low versus high: only the treatment mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0021 to 0.0026, margin 0.0079, SE ratio 1.4943	pass
`mar_robustness`	`l3_ate_low__outcome_and_response_wrong`	control	L3 difference, low versus high: the outcome regression and the observation mechanism are both wrong	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias 0.2911 to 0.2969, margin 0.0098, SE ratio 1.7518	pass
`mar_robustness`	`l3_ate_mid__only_outcome_wrong`	positive	L3 difference, mid versus high: only the outcome regression is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0035 to 0.0029, margin 0.0108, SE ratio 0.9779	pass
`mar_robustness`	`l3_ate_mid__only_response_wrong`	positive	L3 difference, mid versus high: only the observation mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0030 to 0.0021, margin 0.0086, SE ratio 1.5259	pass
`mar_robustness`	`l3_ate_mid__only_treatment_wrong`	positive	L3 difference, mid versus high: only the treatment mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0030 to 0.0012, margin 0.0072, SE ratio 1.3532	pass
`mar_robustness`	`l3_ate_mid__outcome_and_response_wrong`	control	L3 difference, mid versus high: the outcome regression and the observation mechanism are both wrong	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias 0.1499 to 0.1558, margin 0.0100, SE ratio 1.4921	pass
`mar_robustness`	`l4_ate_low__only_outcome_wrong`	positive	L4 difference, low versus high: only the outcome regression is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0219 to 0.0187, margin 0.0681, SE ratio 0.9865	pass
`mar_robustness`	`l4_ate_low__only_response_wrong`	positive	L4 difference, low versus high: only the observation mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0130 to 0.0087, margin 0.0365, SE ratio 1.7824	pass
`mar_robustness`	`l4_ate_low__only_treatment_wrong`	positive	L4 difference, low versus high: only the treatment mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0183 to 0.0030, margin 0.0359, SE ratio 1.4934	pass
`mar_robustness`	`l4_ate_low__outcome_and_response_wrong`	control	L4 difference, low versus high: the outcome regression and the observation mechanism are both wrong	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias 2.9086 to 2.9477, margin 0.0656, SE ratio 1.7332	pass
`mar_robustness`	`l4_ate_mid__only_outcome_wrong`	positive	L4 difference, mid versus high: only the outcome regression is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0101 to 0.0313, margin 0.0694, SE ratio 1.0074	pass
`mar_robustness`	`l4_ate_mid__only_response_wrong`	positive	L4 difference, mid versus high: only the observation mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0238 to 0.0019, margin 0.0432, SE ratio 1.4191	pass
`mar_robustness`	`l4_ate_mid__only_treatment_wrong`	positive	L4 difference, mid versus high: only the treatment mechanism is wrong	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.0122 to 0.0101, margin 0.0373, SE ratio 1.2520	pass
`mar_robustness`	`l4_ate_mid__outcome_and_response_wrong`	control	L4 difference, mid versus high: the outcome regression and the observation mechanism are both wrong	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias 1.4891 to 1.5278, margin 0.0648, SE ratio 1.4437	pass
`simultaneous_coverage`	`l1__pointwise_joint_control`	control	L1, two arms and a binary outcome: the pointwise 95% intervals of the same fits, read jointly	joint coverage upper endpoint must fall below the nominal rate	joint coverage 0.8575 to 0.8957	pass
`simultaneous_coverage`	`l1__simultaneous_band`	positive	L1, two arms and a binary outcome: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves	joint coverage interval inside the calibration coverage band	joint coverage 0.9255 to 0.9533	pass
`simultaneous_coverage`	`l2__pointwise_joint_control`	control	L2, two arms and a bounded continuous outcome: the pointwise 95% intervals of the same fits, read jointly	joint coverage upper endpoint must fall below the nominal rate	joint coverage 0.8761 to 0.9119	pass
`simultaneous_coverage`	`l2__simultaneous_band`	positive	L2, two arms and a bounded continuous outcome: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves	joint coverage interval inside the calibration coverage band	joint coverage 0.9422 to 0.9665	pass
`simultaneous_coverage`	`l3__pointwise_joint_control`	control	L3, three arms and a binary outcome: the pointwise 95% intervals of the same fits, read jointly	joint coverage upper endpoint must fall below the nominal rate	joint coverage 0.8047 to 0.8487	pass
`simultaneous_coverage`	`l3__simultaneous_band`	positive	L3, three arms and a binary outcome: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves	joint coverage interval inside the calibration coverage band	joint coverage 0.9344 to 0.9604	pass
`simultaneous_coverage`	`l4__pointwise_joint_control`	control	L4, three arms and a bounded continuous outcome: the pointwise 95% intervals of the same fits, read jointly	joint coverage upper endpoint must fall below the nominal rate	joint coverage 0.8021 to 0.8463	pass
`simultaneous_coverage`	`l4__simultaneous_band`	positive	L4, three arms and a bounded continuous outcome: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves	joint coverage interval inside the calibration coverage band	joint coverage 0.9372 to 0.9626	pass

The table below states what each family checks.

family	fits	cells
`interval_calibration`	2,000 samples of 2,000 rows for each law, with the primary trees	one positive cell for each of the 22 estimands, and a control that multiplies each standard error by `margin:shrunken_se_factor`
`simultaneous_coverage`	the same fits, which also declare `Inference(simultaneous=True)`	the default multiplier band over each law’s estimands, and a control that reads the same fits’ pointwise intervals jointly
`mar_robustness`	1,200 samples of 2,000 rows for each law and configuration, with exact or deliberately wrong nuisance tables	only `Q`, only `g`, or only `pi` wrong for each ATE contrast, and a control with `Q` and `pi` both wrong
`crossfit_overfitting`	400 samples of 500 rows for each law, with eight noise covariates	a fully grown outcome tree beside the exact mechanisms, fitted out of fold and in sample on the same draws

A ratio row is on the log scale, which is the scale its interval uses. A joint cell has no scalar estimand. Its row records the largest standardized deviation over the law’s estimands, so the table quotes its joint coverage alone.

The pointwise joint control is designed to under-cover. In the Gaussian limit, the pointwise intervals of one law cover every truth at once in less than 90% of samples (tests/unit/test_arm_indexed_cvtmle_method_study.py). The band cell therefore shows that the band’s wider critical value restores the nominal joint rate.

The control cannot fail a wrong band. A Šidák or Bonferroni band ignores the correlation between the estimands. Each one is wider than the max-t band and passes three of the four coverage cells. A second check reads the committed critical values of each law.

check	requirement
target	the limiting max-t quantile, from each law’s exact influence-curve correlation, at the level that `np.quantile` of 1,000 multiplier draws reaches on average (0.94910)
mean committed critical value	within `0.005` of the target for each law
mutation: a Šidák or a Bonferroni critical value	at least `0.08` from the target for each law, so the check fails

Each critical value has a Monte Carlo standard deviation near 0.054 from its 1,000 draws. The tolerance is four standard errors of a mean over 2,000 replications. tests/unit/test_arm_indexed_cvtmle_method_study.py runs the check and both mutations.

A wrong outcome regression is 1 - mu on the [0, 1] scale. Each wrong mechanism is a fixed table that tests/studies/mar_arm_indexed_laws.py declares. The large-sample limit of each single-wrong fit equals the truth to 1e-12. The limit of the control moves each ATE contrast by more than a tenth of the outcome range. The unit tests above check both statements.

The four overfitting pairs share one resampling label for their coverage-gain intervals. Their bootstrap draws are therefore correlated across the pairs.

Measured values and declared margins#

quantity	value	source
`replicates`	800	primary replications
`n`	2000	observations per primary replication
`independent_tests_passed`	44	truth tests passing
`independent_tests_total`	44	truth tests reported
`paired_tests_passed`	22	paired comparisons passing
`paired_tests_total`	22	paired comparisons reported
`property_cells_passed`	84	property cells passing
`property_cells_total`	84	property cells reported
`max_standardized_bias`	0.0636	largest primary standardized bias
`min_coverage`	0.9300	lowest primary coverage
`max_margin_utilization`	1.924e-07	largest share of the paired margin used
`properties[simultaneous_coverage/l1__simultaneous_band]:coverage`	0.9405	L1 band joint coverage
`properties[simultaneous_coverage/l2__simultaneous_band]:coverage`	0.9555	L2 band joint coverage
`properties[simultaneous_coverage/l3__simultaneous_band]:coverage`	0.9485	L3 band joint coverage
`properties[simultaneous_coverage/l4__simultaneous_band]:coverage`	0.9510	L4 band joint coverage
`properties[simultaneous_coverage/l3__pointwise_joint_control]:coverage`	0.8275	L3 pointwise joint coverage
`properties[mar_robustness/l1_ate__outcome_and_response_wrong]:bias`	-0.4788	L1 ATE bias of the control
`properties[mar_robustness/l3_ate_mid__outcome_and_response_wrong]:bias`	0.1529	L3 mid-versus-high bias of the control
`margin:confidence_level`	0.9900	Monte Carlo confidence level
`margin:alpha`	0.0500	nominal test size
`margin:nominal_coverage`	0.9500	nominal interval coverage
`margin:bootstrap_replicates`	10000	bootstrap replications
`margin:standardized_bias`	0.2500	standardized-bias margin
`margin:coverage_floor`	0.9000	primary coverage floor
`margin:over_coverage_ceiling`	0.9900	descriptive overcoverage threshold
`margin:se_ratio_sanity_lower`	0.8000	primary SE-ratio lower screen
`margin:se_ratio_sanity_upper`	1.2000	primary SE-ratio upper screen
`margin:calibration_se_ratio_lower`	0.9300	calibration SE-ratio lower bound
`margin:calibration_se_ratio_upper`	1.0700	calibration SE-ratio upper bound
`margin:calibration_coverage_lower`	0.9200	calibration and band coverage lower bound
`margin:calibration_coverage_upper`	0.9800	calibration and band coverage upper bound
`margin:type_i_ceiling`	0.1000	type-I upper bound
`margin:paired_difference`	0.1500	paired similarity margin in pooled SDs
`margin:rmse_noninferiority`	1.1000	RMSE-ratio noninferiority bound
`margin:coverage_noninferiority`	-0.0250	coverage-difference noninferiority bound
`margin:calibration_noninferiority`	0.0500	calibration-excess noninferiority bound
`margin:minimum_power`	0.8000	minimum power lower bound
`margin:root_n_slope`	-0.5000	expected root-n slope
`margin:root_n_slope_lower`	-0.6250	root-n slope lower bound
`margin:root_n_slope_upper`	-0.3750	root-n slope upper bound
`margin:excluded_slope`	-0.2500	slower rate the interval must exclude
`margin:overfit_se_floor`	0.8500	cross-fit SE-ratio floor
`margin:overfit_control_ceiling`	0.7500	in-sample SE-ratio ceiling
`margin:overfit_coverage_gain`	0.1500	paired coverage-gain floor
`margin:union_model_se_lower`	0.1000	union-model SE-ratio screen, lower limit
`margin:union_model_se_upper`	10	union-model SE-ratio screen, upper limit
`margin:shrunken_se_factor`	0.7000	negative-control SE multiplier
`bound:l1_ey0_standard_error`	0.0238	exact EIF standard error at primary n
`bound:l1_ey1_standard_error`	0.0233	exact EIF standard error at primary n
`bound:l1_ate_standard_error`	0.0335	exact EIF standard error at primary n
`bound:l1_rr_standard_error`	0.0743	exact EIF standard error at primary n
`bound:l1_or_standard_error`	0.1403	exact EIF standard error at primary n
`bound:l2_ey0_standard_error`	0.1100	exact EIF standard error at primary n
`bound:l2_ey1_standard_error`	0.1144	exact EIF standard error at primary n
`bound:l2_ate_standard_error`	0.1619	exact EIF standard error at primary n
`bound:l3_ey_high_standard_error`	0.0265	exact EIF standard error at primary n
`bound:l3_ey_low_standard_error`	0.0269	exact EIF standard error at primary n
`bound:l3_ey_mid_standard_error`	0.0241	exact EIF standard error at primary n
`bound:l3_ate_low_standard_error`	0.0378	exact EIF standard error at primary n
`bound:l3_ate_mid_standard_error`	0.0362	exact EIF standard error at primary n
`bound:l3_rr_low_standard_error`	0.0807	exact EIF standard error at primary n
`bound:l3_rr_mid_standard_error`	0.0596	exact EIF standard error at primary n
`bound:l3_or_low_standard_error`	0.1677	exact EIF standard error at primary n
`bound:l3_or_mid_standard_error`	0.1580	exact EIF standard error at primary n
`bound:l4_ey_high_standard_error`	0.1214	exact EIF standard error at primary n
`bound:l4_ey_low_standard_error`	0.1237	exact EIF standard error at primary n
`bound:l4_ey_mid_standard_error`	0.1156	exact EIF standard error at primary n
`bound:l4_ate_low_standard_error`	0.1734	exact EIF standard error at primary n
`bound:l4_ate_mid_standard_error`	0.1760	exact EIF standard error at primary n

Limits#

The evidence covers four finite laws with one three-level baseline covariate. Two laws have two arms and two have three. Two have a binary outcome and two a Beta outcome on a known support.
It covers one package-generated unstratified ten-fold draw, pooled targeting, whole-sample evaluation, and fixed q_bounds.
The primary fits refuse a treatment prediction that reaches g_bounds=(0.01, 0.99) at either end, and a response prediction below nuisance_bound=0.01. The R runner also refuses a product g_a pi_a below its gbound of 0.001. No check covers the outcome-regression predictions or the property fits. The study does not show that those predictions stay clear of every bound.
Every fit declares Runtime(random_state=0), and an unstratified fold draw depends only on the row count and the seed. Every replication of one size therefore uses the same assignment of rows to ten equal folds. The rows are iid, so that fixed assignment has the distribution of a random partition of each sample. The study does not measure how the estimate varies across fold draws of one sample.
The primary and calibration rows use depth-five trees that can fit the saturated, correct model. Those rows do not test calibration under nuisance misspecification. The overfitting pairs are the data-adaptive evidence, and they use the exact mechanisms. The study does not establish performance for every learner library or tuning procedure.
The exact efficient standard errors describe the finite laws. No calibration cell claims efficiency-bound attainment.
The band cells measure the default Rademacher multiplier band over one law’s reported estimands. They do not cover other multiplier kinds, other estimand sets, or bands across laws.
The external comparison conditions on the stitched nuisance predictions from cleverly. It does not compare nuisance training. The continuous comparison rests on the planted-row scale workaround, which the probe checks on replication 0 only.
The study excludes repeated splits, stratified or supplied folds, fold targeting or evaluation, att and atc, continuous rr and or, weights, clusters, baseline strata, bootstrap intervals, intermediates, missing treatment, MNAR outcomes, and C-TMLE or DR-TMLE fits.

Reproduction#

The fixture README, manifest, replications, scale probe, performance decisions, and property results carry the protocol, provenance, and every published row.