Stacked missing-outcome natural-course CV-TMLE

This study validates the stacked CV-TMLE of the observed outcome mean under MAR. The table gives the studied construction. The primary rows declare it through the public TMLEMethod and CrossFitting API.

component	studied setting
partition	one package-generated, unstratified ten-fold split
outcome regression	fitted on the respondents in each training complement
response mechanism	fitted separately on each training complement
targeting	one pooled logistic fluctuation over the stacked held-out rows
evaluation	plug-in and influence curve over the whole sample
primary learners	separate classification trees with depth five and minimum leaf size 25

The law has six (A, W) covariate cells. The smallest respondent cell has an expected 72 rows in a 1,800-row training complement, which exceeds the minimum leaf size. Each depth-five tree can therefore fit the saturated model, and that model is correct for this law. The primary and flexible-learning rows thus do not test calibration under a misspecified data-adaptive learner. The paired overfitting arm supplies the data-adaptive evidence. It fits a fully grown outcome tree with eight noise covariates beside the exact response mechanism.

The study compares R tmle 2.1.1 through its population-mean path. The adapter supplies the same stitched out-of-fold outcome and response predictions that cleverly targets. It passes a constant synthetic treatment and duplicates each realized-arm prediction across the two required input columns. The original treatment remains beside W, preserving the conditioning set of the supplied predictions. R then fits one pooled respondent-only fluctuation with clever covariate 1 / pi and reports EY1 as the population mean.

This comparison is conditional on the supplied nuisance predictions. It validates the pooled targeting, plug-in, and influence-curve path. It does not validate R-native cross-fit training. The table gives the status of the other candidates. docs/references.md records each source locator.

candidate	disposition
R tmle 2.1.1	compared through its population-mean path with the same stitched out-of-fold outcome and response predictions
tmle3 at commit ed72f8a	its generic treatment-specific outcome fit can use the Delta = 0 pseudo-outcomes when it predicts under Delta = 1. cleverly fits that regression on respondents only
zEpid 0.9.1	its cross-fit TMLE targets inside each fold rather than with one pooled fluctuation
Newey and Robins (2018)	the construction fits the outcome and inverse response regressions on distinct subsamples. That is a different estimator

source	locator	what it supplies
Díaz, Carone and van der Laan (2016)	Section 2, Equations (1)–(5)	the MAR natural-course functional, efficient influence curve, targeting update, and two-nuisance remainder
Levy (2018)	abstract	the stacked validation update and the whole-sample plug-in
Levy (2018)	Section 3.1	the asymptotic expansion after the pooled score is solved, which supports the influence-curve interval
Zheng and van der Laan (2011)	Sections 2 and 2.1	the training-complement nuisance fits and the untargeted empirical-distribution component
Zheng and van der Laan (2011)	Theorem 2	the partial-targeting expansion and its conditions

Accuracy against known truth

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
binary-outcome observational natural-course law with learned MAR nuisances	ey_obs	observed outcome mean under the natural course	cleverly stacked missing-outcome natural-course CV-TMLE	-0.0018 to 0.0014	0.9537	0.9943	pass
binary-outcome observational natural-course law with learned MAR nuisances	ey_obs	observed outcome mean under the natural course	R tmle population-mean path	-0.0018 to 0.0014	0.9537	0.9946	pass

Agreement with the canonical implementation

Comparator-backed: a paired comparison with tmle-r-population-mean on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.

law	estimand	what was compared	paired difference	share of margin used	RMSE ratio bound	coverage difference	comparison kind	calibration resolution	result
binary-outcome observational natural-course law with learned MAR nuisances	ey_obs	observed outcome mean under the natural course	-1.770e-11	6.736e-09	1.0000	0	matching implementation	0.000001 vs 0.0500	equivalent

In this comparator-backed study, R tmle uses the centered, n - 1 divisor sample variance var(IC) / n. cleverly uses the uncentered second moment mean(IC^2) / n. The two rules differ by the finite-sample factor n / (n - 1) when the targeted score is zero. R also bounds its binary interval to [0, 1], while cleverly reports an unbounded Wald interval. This law stays away from the bounds.

Theory properties

property	cell	role	what was tested	what must hold	measured	result
crossfit_overfitting	in_sample_control	control	the same flexible learner fitted in sample, with no cross-fitting	SE ratio must fall below the overfitting ceiling	SE ratio 0.4918 to 0.5870	pass
crossfit_overfitting	stacked_mar_natural_course_cvtmle	positive	stacked MAR natural-course CV-TMLE with a fully grown outcome tree	SE ratio clears the overfitting floor and stays inside the sanity band	SE ratio 0.9297 to 1.1005	pass
double_robustness	both_wrong	control	both nuisances are misspecified	bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread	bias -0.0156 to -0.0135, margin 0.0034, SE ratio 1.5963	pass
double_robustness	outcome_correct	positive	only the outcome regression is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.000595 to 0.0013, margin 0.0032, SE ratio 1.4842	pass
double_robustness	response_correct	positive	only the outcome-observation mechanism is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0012 to 0.0019, margin 0.0053, SE ratio 0.9666	pass
interval_calibration	ey_obs__flexible_learning	positive	observed outcome mean under the natural course: separate depth-five trees fit the outcome and response nuisances out of fold; on this six-cell law each tree fits the saturated, correct model	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9422 to 0.9665, SE ratio 0.9807 to 1.0602	pass
interval_calibration	ey_obs__shrunken_se_control	control	observed outcome mean under the natural course: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8126 to 0.8558, SE ratio 0.6864 to 0.7411	pass

The flexible-learning cell checks empirical coverage and reported-SE calibration together. Its trees fit the correct saturated model, as the first section states. The derived shrunken-SE arm multiplies each reported standard error by 0.70 and must fall below the calibration band.

The paired overfitting experiment is the data-adaptive check. It draws 400 samples of 500 rows and adds eight noise covariates to each. Both arms fit the same fully grown outcome tree and the exact response mechanism on each sample. Only cross-fitting differs between the arms. Coverage is 0.9500 with cross-fitting and 0.6950 in sample.

The union-model cells make either the outcome regression or the response mechanism correct. The both-wrong control passes on bias discrimination. Its 99% bias interval, -0.0156 to -0.0135, lies outside the 0.0034 margin. Its standardized bias is -1.0593. Its interval coverage stays at 0.9858, because its reported standard error is 1.5963 times the empirical spread. The control therefore shows detectable bias rather than an interval failure.

Measured values and declared margins

quantity	value	source
replicates	800	primary replications
n	2000	observations per primary replication
independent_tests_passed	2	truth tests passing
independent_tests_total	2	truth tests reported
paired_tests_passed	1	paired comparisons passing
paired_tests_total	1	paired comparisons reported
property_cells_passed	7	property cells passing
property_cells_total	7	property cells reported
max_standardized_bias	0.0105	largest primary standardized bias
min_coverage	0.9537	lowest primary coverage
margin:confidence_level	0.9900	Monte Carlo confidence level
margin:alpha	0.0500	nominal test size
margin:nominal_coverage	0.9500	nominal interval coverage
margin:bootstrap_replicates	10000	bootstrap replications
margin:standardized_bias	0.2500	standardized-bias margin
margin:coverage_floor	0.9000	primary coverage floor
margin:over_coverage_ceiling	0.9900	descriptive overcoverage threshold
margin:se_ratio_sanity_lower	0.8000	primary SE-ratio lower screen
margin:se_ratio_sanity_upper	1.2000	primary SE-ratio upper screen
margin:calibration_se_ratio_lower	0.9300	calibration SE-ratio lower bound
margin:calibration_se_ratio_upper	1.0700	calibration SE-ratio upper bound
margin:calibration_coverage_lower	0.9200	calibration coverage lower bound
margin:calibration_coverage_upper	0.9800	calibration coverage upper bound
margin:type_i_ceiling	0.1000	type-I upper bound
margin:paired_difference	0.1500	paired similarity margin in pooled SDs
margin:rmse_noninferiority	1.1000	RMSE-ratio noninferiority bound
margin:coverage_noninferiority	-0.0250	coverage-difference noninferiority bound
margin:calibration_noninferiority	0.0500	calibration-excess noninferiority bound
margin:minimum_power	0.8000	minimum power lower bound
margin:root_n_slope	-0.5000	expected root-n slope
margin:root_n_slope_lower	-0.6250	root-n slope lower bound
margin:root_n_slope_upper	-0.3750	root-n slope upper bound
margin:excluded_slope	-0.2500	slower rate the interval must exclude
margin:overfit_se_floor	0.8500	cross-fit SE-ratio floor
margin:overfit_control_ceiling	0.7500	in-sample SE-ratio ceiling
margin:overfit_coverage_gain	0.1500	paired coverage-gain floor
margin:union_model_se_lower	0.1000	union-model SE-ratio screen, lower limit
margin:union_model_se_upper	10	union-model SE-ratio screen, upper limit
margin:shrunken_se_factor	0.7000	negative-control SE multiplier
bound:ey_obs_standard_error	0.0173	exact binary-law EIF standard error at primary n

Limits

• The evidence covers one observational binary-treatment, binary-outcome MAR law with one three-level baseline covariate.

• It covers exactly one package-generated unstratified ten-fold draw, pooled targeting, whole-sample evaluation, pointwise Wald inference, and bounded nuisance predictions.

• The primary and flexible-learning rows use depth-five trees that fit the saturated, correct model on this six-cell law. Those rows do not test calibration under nuisance misspecification. The overfitting arm is the data-adaptive evidence, and it uses the exact response mechanism. The study does not establish performance for every learner library or tuning procedure.

• The exact EIF bound describes the finite law. The flexible-learning cell does not claim efficiency-bound attainment.

• The external comparison conditions on the stitched nuisance predictions from cleverly. It does not compare nuisance training implementations.

• The R variance and interval conventions differ as the agreement section states. The study does not claim bitwise interval equality.

• The study excludes repeated splits, stratified or supplied folds, and fold targeting or evaluation. It excludes continuous or multinomial treatment, continuous outcomes, weights, clusters, and baseline strata. It also excludes intermediates, missing treatment, MNAR outcomes, and simultaneous intervals.

Reproduction

The fixture README, manifest, replications, performance decisions, and property results carry the protocol, provenance, and every published row.