Ordinary missing-outcome natural-course TMLE#

This study validates ordinary, non-cross-fitted TMLE for the observed outcome mean under MAR. The target averages the outcome regression over the empirical joint distribution of treatment and baseline covariates. Its score therefore uses the outcome and response nuisances, but no treatment mechanism. Binary and bounded-continuous primary laws share the same conditional means and observation process; the continuous outcome is beta distributed and uses fixed bounds (0, 1).

The study compares both primary laws with the R tmle 2.1.1 population-mean path. The comparison conditions on the outcome and response predictions of the cleverly fit. Comparator gives the adapter and its checks.

Díaz, Carone and van der Laan (2016), Section 2 and Equations (1)–(5) give the estimator construction. tests/unit/test_influence_gateaux_natural_course_mar.py pins the target, the efficient influence curve, and the targeting score against the exact law. tests/unit/test_remainder_natural_course_mar.py pins the signed second-order remainder and both double-robustness halves. This repeated-sampling study supplies the rest.

Comparator#

The study compares R tmle 2.1.1. The adapter tests/canonical/tmle_mar_natural_course/run_study.R supplies the outcome and response predictions that cleverly targets. The R tmle audit gives the source lines of this path. The table gives the arguments of each call.

argument	value
`A`	`rep(1, n)`, a constant synthetic treatment that selects the population-mean path
`W`	the original treatment and `W`
`Q`	the realized-arm outcome prediction, in both columns
`g1W`	`rep(1, n)`
`pDelta1`	the response prediction, in both columns
`family`	`binomial` for the binary law, and `gaussian` for the continuous law
`fluctuation`	`logistic`
`Qbounds`, `gbound`, and `alpha`	`c(0, 1)`, `0.01`, and `0.9995`

The runner stops when R refits a supplied nuisance. It also stops when a response prediction is at or below gbound, because R would clip that prediction.

R takes a continuous outcome scale from every non-NA Y (tmle.R line 1120). A continuous fit therefore sets Y to 0 and 1 on the first two rows whose response indicator is zero. The indicator removes those rows from the fluctuation and the curve. Before each fit, the runner checks that R takes the scale c(0, 1). probe_scale_workaround.R checks the workaround on every continuous replication, and it writes scale-probe.csv.

probe check	requirement
scale with the planted rows	equal to `c(0, 1)` exactly
scale without the planted rows	different from `c(0, 1)`
planted rows moved to two other eligible rows	point and curve bitwise unchanged
independent rebuild at the `c(0, 1)` scale	point and curve within `1e-12`
`unplanted_point_difference`, the point shift without the workaround	above zero

The regeneration refuses publication when a probe row fails or a continuous replication has no row. tests/unit/test_mar_natural_course_method_study.py checks that every committed row passes.

The witness unplanted_point_difference is small for this law. The observed range of the Beta outcome already lies close to (0, 1). The planted rows therefore move the estimate very little. scale-probe.csv records the unplanted scale and the witness of each replication.

The probe shows that the workaround sets the R scale exactly. The targeting-ratio check below cannot detect a missing workaround. In almost every replication, the witness is below 1e-2 of the targeting move, so that check admits it. The source is the witness in scale-probe.csv and the targeting move in replicates.csv.gz.

A separate witness check compares abs(cleverly - R) with the witness in each continuous replication. An estimate at the unplanted scale sits about one witness from the R point. The check requires abs(cleverly - R) at or below 1e-2 of the witness in every replication. This bound was chosen after the regeneration, from the committed rows. The witness has no lower bound for this law, so the check covers the committed replications only.

This comparison conditions on the supplied nuisance predictions. It validates the targeting, the plug-in, the influence curve, and the variance. It does not compare nuisance training.

The paired tests alone cannot show that cleverly targets. The oracle nuisances leave the initial plug-in unbiased, so an estimate without its targeting step still passes every paired test and every truth test. Separate checks over the committed replications carry that claim and the variance rule.

check	requirement
targeting move, `estimate - initial_estimate`	nonzero in every replication
`abs(cleverly - R) / abs(targeting move)`	below `1e-2` in every replication
initial estimates of the two implementations	equal to `1e-12`
`abs(SE_cleverly - SE_R) / SE_R`	below `1e-6` in every replication
`abs(cleverly - R) / unplanted_point_difference`, continuous law	at or below `1e-2` in every replication
mutation: `cleverly` reports its initial estimate	the ratio exceeds `0.9` in every row, so the check fails
mutation: the standard error times `sqrt((n - 1) / n)`	the relative difference reaches `1e-6` in every row, so the check fails
mutation: the `cleverly` estimate moves by the witness, in either direction	the witness ratio exceeds `0.9` in every row, so the check fails

R tmle stops its fluctuation glm at a relative deviance change of 1e-8. That stop resolves the fluctuation coefficient to about 1e-4 of its scale, and the ratio bound allows a hundredfold over that. The second mutation is the uncentered second-moment rule of the stacked study. At n = 2000 it differs from R by a relative 2.5e-4, and the SE bound sits two orders below that gap. Commit 0605650 declared every bound except the witness bound before the regeneration. tests/unit/test_mar_natural_course_method_study.py runs the checks and the three mutations.

Accuracy against known truth#

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
binary-outcome observational natural-course law with MAR outcomes	`ey_obs`	observed outcome mean under the natural course	`cleverly` missing-outcome natural-course TMLE	-0.000474 to 0.0029	0.9350	0.9468	pass
binary-outcome observational natural-course law with MAR outcomes	`ey_obs`	observed outcome mean under the natural course	R `tmle` population-mean path	-0.000474 to 0.0029	0.9350	0.9468	pass
bounded continuous-outcome observational natural-course law with MAR outcomes	`ey_obs`	observed outcome mean under the natural course	`cleverly` missing-outcome natural-course TMLE	-0.000306 to 0.000726	0.9375	0.9687	pass
bounded continuous-outcome observational natural-course law with MAR outcomes	`ey_obs`	observed outcome mean under the natural course	R `tmle` population-mean path	-0.000306 to 0.000726	0.9375	0.9687	pass

Agreement with the canonical implementation#

Comparator-backed: a paired comparison with tmle-r-population-mean on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.

law	estimand	what was compared	paired difference	share of margin used	RMSE ratio bound	coverage difference	comparison kind	calibration resolution	result
binary-outcome observational natural-course law with MAR outcomes	`ey_obs`	observed outcome mean under the natural course	-4.741e-13	1.734e-10	1.0000	0	matching implementation	2.429e-11 vs 0.0500	equivalent
bounded continuous-outcome observational natural-course law with MAR outcomes	`ey_obs`	observed outcome mean under the natural course	-6.072e-14	7.158e-11	1.0000	0	matching implementation	7.497e-12 vs 0.0500	equivalent

In this comparator-backed study, both implementations use the centered, n - 1 divisor variance var(IC) / n. A non-cross-fitted natural-course fit in cleverly declares that rule (covariance_rule in src/cleverly/estimators/tmle.py). R bounds a binary-outcome interval to [0, 1], and cleverly reports an unbounded Wald interval. No committed interval reaches either bound.

Theory properties#

property	cell	role	what was tested	what must hold	measured	result
`double_robustness`	`both_correct`	positive	both the outcome regression and the outcome-response mechanism are correct	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0017 to 0.000937, margin 0.0044, SE ratio 0.9913	pass
`double_robustness`	`both_wrong`	control	both nuisances are misspecified	bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread	bias -0.0156 to -0.0135, margin 0.0034, SE ratio 1.5919	pass
`double_robustness`	`outcome_correct`	positive	only the outcome regression is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.000619 to 0.0012, margin 0.0031, SE ratio 1.5107	pass
`double_robustness`	`response_correct`	positive	only the outcome-observation mechanism is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0010 to 0.0020, margin 0.0051, SE ratio 1.0152	pass
`interval_calibration`	`ey_obs__correctly_specified`	positive	observed outcome mean under the natural course: both nuisances are correctly specified with an independently computed efficiency bound	SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands	coverage 0.9440 to 0.9614, SE ratio 0.9745 to 1.0307, empirical efficiency ratio 0.9690 to 1.0250, reported efficiency ratio 0.9977 to 0.9996	pass
`interval_calibration`	`ey_obs__noise_control`	control	observed outcome mean under the natural course: one efficiency-bound unit of independent noise is added to each estimate	the empirical efficiency ratio must rise above the band	coverage 0.8251 to 0.8551, SE ratio 0.6943 to 0.7341, empirical efficiency ratio 1.3604 to 1.4383, reported efficiency ratio 0.9977 to 0.9996	pass
`interval_calibration`	`ey_obs__shrunken_se_control`	control	observed outcome mean under the natural course: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8113 to 0.8424, SE ratio 0.6820 to 0.7218, empirical efficiency ratio 0.9686 to 1.0248, reported efficiency ratio 0.6984 to 0.6997	pass
`missingness_necessity`	`ey_obs__complete_case_control`	control	observed outcome mean under the natural course: the identical estimator silently discards unobserved outcomes and ignores selection	bias interval must fall entirely outside the margin	bias -0.0075 to -0.0052, margin 0.0040	pass
`missingness_necessity`	`ey_obs__declared`	positive	observed outcome mean under the natural course: the observation indicator is declared, so correct mechanisms carry a wrong outcome model	bias interval inside the equivalence margin	bias -0.0013 to 0.0017, margin 0.0051	pass
`root_n_and_efficiency`	`n_2000`	positive	bias, coverage and SE calibration at n = 2,000	bias inside the margin, coverage clears the floor, SE ratio inside the sanity band	bias -0.000060, coverage 0.9356 to 0.9737, SE ratio 1.0358	pass
`root_n_and_efficiency`	`n_500`	control	bias, coverage and SE calibration at n = 500	coverage interval lies below nominal or clears the declared floor	bias -0.0012, coverage 0.9341 to 0.9727, SE ratio 1.0312	pass
`root_n_and_efficiency`	`n_8000`	positive	bias, coverage and SE calibration at n = 8,000	bias inside the margin, coverage clears the floor, SE ratio inside the sanity band	bias 0.000133, coverage 0.9121 to 0.9575, SE ratio 0.9658	pass
`root_n_rate`	`empirical_sd`	positive	log empirical spread of the estimates regressed on log n across three sizes	slope interval inside the root-n band and excluding -1/4	slope -0.5092 to -0.4417	pass
`root_n_rate`	`reported_se`	positive	the same regression applied to the mean reported standard error	slope interval inside the root-n band and excluding -1/4	slope -0.5003 to -0.4973	pass
`targeting_necessity`	`ey_obs__targeted`	positive	observed outcome mean under the natural course: the estimator fluctuates a misspecified outcome model, so targeting does all the adjusting	bias interval inside the equivalence margin	bias -0.0019 to 0.0012, margin 0.0052	pass
`targeting_necessity`	`ey_obs__untargeted`	control	observed outcome mean under the natural course: the identical fit with every fluctuation step removed	bias interval must fall entirely outside the margin	bias 0.0117 to 0.0123, margin 0.0011	pass

The property law tests the two-nuisance union model: either the outcome regression or the response mechanism may be correct, and a both-wrong cell must discriminate the resulting bias. Separate cells test three-size root-n behavior, exact-EIF efficiency, repeated-sampling interval calibration and its two mutations, the targeting update, and the bias induced by silently analyzing complete cases.

Measured values and declared margins#

quantity	value	source
`replicates`	800	primary replications per law
`n`	2000	observations per primary replication
`independent_tests_passed`	4	truth tests passing
`independent_tests_total`	4	truth tests reported
`paired_tests_passed`	2	paired comparisons passing
`paired_tests_total`	2	paired comparisons reported
`property_cells_passed`	16	property cells passing
`property_cells_total`	16	property cells reported
`max_standardized_bias`	0.0653	largest primary standardized bias
`min_coverage`	0.9350	lowest primary coverage
`margin:confidence_level`	0.9900	Monte Carlo confidence level
`margin:alpha`	0.0500	nominal test size
`margin:nominal_coverage`	0.9500	nominal interval coverage
`margin:bootstrap_replicates`	10000	bootstrap replications
`margin:standardized_bias`	0.2500	standardized-bias margin
`margin:coverage_floor`	0.9000	primary coverage floor
`margin:over_coverage_ceiling`	0.9900	descriptive overcoverage threshold
`margin:se_ratio_sanity_lower`	0.8000	primary SE-ratio lower screen
`margin:se_ratio_sanity_upper`	1.2000	primary SE-ratio upper screen
`margin:calibration_se_ratio_lower`	0.9300	calibration SE-ratio lower bound
`margin:calibration_se_ratio_upper`	1.0700	calibration SE-ratio upper bound
`margin:calibration_coverage_lower`	0.9200	calibration coverage lower bound
`margin:calibration_coverage_upper`	0.9800	calibration coverage upper bound
`margin:type_i_ceiling`	0.1000	type-I upper bound
`margin:paired_difference`	0.1500	paired similarity margin in pooled SDs
`margin:rmse_noninferiority`	1.1000	RMSE-ratio noninferiority bound
`margin:coverage_noninferiority`	-0.0250	coverage-difference noninferiority bound
`margin:calibration_noninferiority`	0.0500	calibration-excess noninferiority bound
`margin:minimum_power`	0.8000	minimum power lower bound
`margin:root_n_slope`	-0.5000	expected root-n slope
`margin:root_n_slope_lower`	-0.6250	root-n slope lower bound
`margin:root_n_slope_upper`	-0.3750	root-n slope upper bound
`margin:excluded_slope`	-0.2500	slower rate the interval must exclude
`margin:union_model_se_lower`	0.1000	union-model SE-ratio screen, lower limit
`margin:union_model_se_upper`	10	union-model SE-ratio screen, upper limit
`margin:shrunken_se_factor`	0.7000	negative-control SE multiplier
`margin:efficiency_ratio_lower`	0.9000	efficiency-ratio lower bound
`margin:efficiency_ratio_upper`	1.1000	efficiency-ratio upper bound
`margin:targeting_displacement`	0.2500	minimum targeting displacement
`margin:missingness_displacement`	0.2500	minimum complete-case displacement
`bound:ey_obs_standard_error`	0.0173	exact binary-law EIF standard error at primary n

Limits#

The primary study covers binary treatment, one three-level baseline covariate, binary and beta-distributed bounded-continuous outcomes, and one observational MAR response mechanism.
The response probabilities and binary outcome regression are finite-law oracles. The continuous primary law uses a correctly specified affine outcome learner fit to the scaled observed outcomes. This does not validate flexible learner wrappers or cross-fitting.
The exact efficiency comparison is for the binary property law; the continuous primary law is checked for truth recovery, coverage, and reported-SE calibration without claiming the same bound.
The external comparison conditions on the supplied outcome and response predictions. It does not compare nuisance training.
The scale probe shows that the planted rows set the R scale exactly. For this law the unplanted fit differs only slightly. Neither the paired tests nor the targeting-ratio check would detect a missing workaround. The witness check carries that claim, with a bound chosen after the regeneration, for the committed replications only.
The study uses ordinary pointwise Wald intervals and excludes weights, clusters, missing treatment, multinomial treatment, MNAR outcomes, sensitivity analysis, and longitudinal data.

Reproduction#

The fixture README, manifest, replications, performance decisions, paired comparisons, scale probe, and property results carry the protocol, provenance, and every published row.