Canonical point-treatment TMLE

This study validates ordinary, non-cross-fitted point-treatment TMLE in cleverly. The canonical comparison uses R tmle3 0.2.0 at pinned commit ed72f8a. The independent claims come from the parameter oracles and from the properties in van der Laan and Rubin’s original TMLE paper.

The row is deliberately named ordinary TMLE. The public default cross-fits its nuisances. The comparison disables cross-fitting in cleverly because tmle3’s ordinary specs are not CV-TMLE. Cross-fitted and CV-TMLE constructions are separate methods and do not inherit this result.

What was compared

setting	cleverly	R tmle3
estimator	ordinary TMLE, cross-fitting disabled	ordinary tmle3 spec
datasets	generated in Python	the identical rows and all DGP covariates
nuisance regressions	GLM	corresponding GLM
treatment contrast	1 versus 0	1 versus 0
overlap	comfortable, no active bound	the same law
intervals	pointwise 95% Wald	pointwise 95% Wald
RR and OR scale	log, exponentiated	log, exponentiated
PAF scale	identity, from the PAF influence curve	negative-log-complement, transformed by 1 - exp(-x)
ATT and ATC updater	default	the constrained one-dimensional updater the package’s own tests exercise

Both laws are point-treatment laws with complete outcomes. The bounded continuous-outcome law with effect modification covers ey1, ey0, ate, att, atc, ey_obs, and par. The binary-outcome law covers those plus paf, rr, and or.

Accuracy against known truth

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
binary-outcome law	atc	average effect on the untreated	cleverly	-0.0026 to 0.0013	0.9506	0.9997	pass
binary-outcome law	atc	average effect on the untreated	R tmle3	-0.0026 to 0.0013	0.9544	1.0049	pass
binary-outcome law	ate	average treatment effect	cleverly	-0.0026 to 0.0012	0.9494	1.0067	pass
binary-outcome law	ate	average treatment effect	R tmle3	-0.0026 to 0.0012	0.9487	1.0066	pass
binary-outcome law	att	average effect on the treated	cleverly	-0.0028 to 0.0012	0.9506	1.0088	pass
binary-outcome law	att	average effect on the treated	R tmle3	-0.0028 to 0.0012	0.9500	1.0141	pass
binary-outcome law	ey0	counterfactual mean under no treatment	cleverly	-0.000802 to 0.0020	0.9531	1.0102	pass
binary-outcome law	ey0	counterfactual mean under no treatment	R tmle3	-0.000802 to 0.0020	0.9531	1.0101	pass
binary-outcome law	ey1	counterfactual mean under treatment	cleverly	-0.0016 to 0.0013	0.9425	0.9861	pass
binary-outcome law	ey1	counterfactual mean under treatment	R tmle3	-0.0016 to 0.0013	0.9425	0.9861	pass
binary-outcome law	ey_obs	observed outcome mean under the natural course	cleverly	-0.000858 to 0.0012	0.9450	0.9919	pass
binary-outcome law	ey_obs	observed outcome mean under the natural course	R tmle3	-0.000858 to 0.0012	0.9450	0.9919	pass
binary-outcome law	or	marginal odds ratio, reported on the log scale	cleverly	-0.0095 to 0.0064	0.9500	1.0074	pass
binary-outcome law	or	marginal odds ratio, reported on the log scale	R tmle3	-0.0095 to 0.0064	0.9500	1.0074	pass
binary-outcome law	paf	population attributable fraction	cleverly	-0.0030 to 0.0013	0.9494	1.0118	pass
binary-outcome law	paf	population attributable fraction	R tmle3	-0.0029 to 0.0026	0.9487	1.0114	pass
binary-outcome law	par	population attributable risk	cleverly	-0.0014 to 0.000586	0.9469	1.0082	pass
binary-outcome law	par	population attributable risk	R tmle3	-0.0014 to 0.000587	0.9469	1.0081	pass
binary-outcome law	rr	marginal risk ratio, reported on the log scale	cleverly	-0.0053 to 0.0033	0.9556	1.0117	pass
binary-outcome law	rr	marginal risk ratio, reported on the log scale	R tmle3	-0.0053 to 0.0033	0.9556	1.0116	pass
bounded continuous-outcome law with effect modification	atc	average effect on the untreated	cleverly	-0.000400 to 0.000508	0.9463	0.9893	pass
bounded continuous-outcome law with effect modification	atc	average effect on the untreated	R tmle3	0.000387 to 0.0013	0.9475	0.9957	pass
bounded continuous-outcome law with effect modification	ate	average treatment effect	cleverly	-0.000459 to 0.000392	0.9481	1.0015	pass
bounded continuous-outcome law with effect modification	ate	average treatment effect	R tmle3	-0.000442 to 0.000406	0.9500	1.0125	pass
bounded continuous-outcome law with effect modification	att	average effect on the treated	cleverly	-0.000602 to 0.000314	0.9394	0.9706	pass
bounded continuous-outcome law with effect modification	att	average effect on the treated	R tmle3	-0.0013 to -0.000431	0.9400	0.9726	pass
bounded continuous-outcome law with effect modification	ey0	counterfactual mean under no treatment	cleverly	-0.000147 to 0.000743	0.9531	1.0146	pass
bounded continuous-outcome law with effect modification	ey0	counterfactual mean under no treatment	R tmle3	-0.000147 to 0.000742	0.9537	1.0149	pass
bounded continuous-outcome law with effect modification	ey1	counterfactual mean under treatment	cleverly	-0.000260 to 0.000788	0.9519	1.0107	pass
bounded continuous-outcome law with effect modification	ey1	counterfactual mean under treatment	R tmle3	-0.000244 to 0.000802	0.9519	1.0122	pass
bounded continuous-outcome law with effect modification	ey_obs	observed outcome mean under the natural course	cleverly	-0.000234 to 0.000779	0.9506	0.9908	pass
bounded continuous-outcome law with effect modification	ey_obs	observed outcome mean under the natural course	R tmle3	-0.000234 to 0.000779	0.9506	0.9908	pass
bounded continuous-outcome law with effect modification	par	population attributable risk	cleverly	-0.000327 to 0.000276	0.9344	0.9451	pass
bounded continuous-outcome law with effect modification	par	population attributable risk	R tmle3	-0.000325 to 0.000277	0.9375	0.9526	pass

Agreement with the canonical implementation

Comparator-backed: a paired comparison with tmle3 on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.

law	estimand	what was compared	paired difference	share of margin used	RMSE ratio bound	coverage difference	comparison kind	calibration resolution	result
binary-outcome law	atc	average effect on the untreated	0.000023	0.0050	1.0066	-0.0037	different estimator	0.0107 vs 0.0500	equivalent
binary-outcome law	ate	average treatment effect	-6.992e-07	0.000157	1.0000	0.000625	matching implementation	0.000096 vs 0.0500	equivalent
binary-outcome law	att	average effect on the treated	-0.000022	0.0049	1.0067	0.000625	different estimator	0.0115 vs 0.0500	equivalent
binary-outcome law	ey0	counterfactual mean under no treatment	9.290e-08	0.000029	1.0000	0	matching implementation	0.000010 vs 0.0500	equivalent
binary-outcome law	ey1	counterfactual mean under treatment	-4.085e-10	1.216e-07	1.0000	0	matching implementation	0.000071 vs 0.0500	equivalent
binary-outcome law	ey_obs	observed outcome mean under the natural course	-8.722e-12	3.654e-09	1.0000	0	matching implementation	5.378e-10 vs 0.0500	equivalent
binary-outcome law	or	marginal odds ratio, reported on the log scale	-5.723e-07	0.000014	1.0000	0	matching implementation	0.000006 vs 0.0500	equivalent
binary-outcome law	paf	population attributable fraction	-6.674e-07	0.000132	1.0002	0.000625	related target	n/a	equivalent
binary-outcome law	par	population attributable risk	-2.965e-07	0.000126	1.0002	0	matching implementation	0.000105 vs 0.0500	equivalent
binary-outcome law	rr	marginal risk ratio, reported on the log scale	-2.861e-07	0.000019	1.0000	0	matching implementation	0.000007 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	atc	average effect on the untreated	-0.000783	0.7445	1.0088	-0.0012	different estimator	0.0033 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	ate	average treatment effect	-0.000016	0.0161	1.0062	-0.0019	matching implementation	0.0239 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	att	average effect on the treated	0.000744	0.6995	1.0019	-0.000625	different estimator	0.0033 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	ey0	counterfactual mean under no treatment	-6.584e-08	0.000064	1.0030	-0.000625	matching implementation	0.0019 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	ey1	counterfactual mean under treatment	-0.000015	0.0122	1.0032	0	matching implementation	0.0040 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	ey_obs	observed outcome mean under the natural course	-5.672e-13	4.814e-10	1.0000	0	matching implementation	1.179e-10 vs 0.0500	equivalent
bounded continuous-outcome law with effect modification	par	population attributable risk	-0.000001	0.0020	1.0045	-0.0031	matching implementation	0.0020 vs 0.0500	equivalent

Theory properties

The double-robustness cells use a bounded nonlinear confounded law with exact ATE 1.75. Its treatment mechanism stays between 0.076 and 0.924, so the configured bounds do not clip it. The wrong main-effects outcome regression imposes a constant contrast, while the true contrast varies with W1 and I(W2 > 0). The treatment-correct cell uses n = 2,000. The other three cells use n = 700. Each cell uses 1,200 replications and the existing predeclared margin.

property	cell	role	what was tested	what must hold	measured	result
double_robustness	both_correct	positive	both the outcome regression and the treatment mechanism are correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0043 to 0.0087, margin 0.0218, SE ratio 0.9982	pass
double_robustness	both_wrong	control	both nuisances are misspecified	bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread	bias -0.3017 to -0.2835, margin 0.0306, SE ratio 0.9811	pass
double_robustness	outcome_correct	positive	only the outcome regression is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0058 to 0.0077, margin 0.0226, SE ratio 0.9250	pass
double_robustness	treatment_correct	positive	only the treatment mechanism is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0094 to 0.0024, margin 0.0199, SE ratio 0.9974	pass
interval_calibration	correctly_specified	positive	both nuisances are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9420 to 0.9645, SE ratio 0.9678 to 1.0384	pass
power	alternative	positive	the same test applied to a law with a real effect	rejection lower bound clears the minimum power	rejection 1, 0.9868 to 1	pass
root_n_and_efficiency	n_2000	positive	bias, coverage and SE calibration at n = 2,000	bias inside the margin, coverage clears the floor, SE ratio inside the sanity band	bias -0.000513, coverage 0.9194 to 0.9627, SE ratio 0.9777	pass
root_n_and_efficiency	n_500	positive	bias, coverage and SE calibration at n = 500	bias inside the margin, coverage clears the floor, SE ratio inside the band	bias -0.000770, coverage 0.9223 to 0.9647, SE ratio 0.9913	pass
root_n_and_efficiency	n_8000	positive	bias, coverage and SE calibration at n = 8,000	bias inside the margin, coverage clears the floor, SE ratio inside the sanity band	bias 0.000285, coverage 0.9297 to 0.9698, SE ratio 1.0077	pass
root_n_rate	empirical_sd	positive	log empirical spread of the estimates regressed on log n across three sizes	slope interval inside the root-n band and excluding -1/4	slope -0.5374 to -0.4725	pass
root_n_rate	reported_se	positive	the same regression applied to the mean reported standard error	slope interval inside the root-n band and excluding -1/4	slope -0.4998 to -0.4974	pass
type_i_error	sharp_null	positive	a confounded law whose true contrast is exactly zero	one-sided rejection bound stays under the declared type-I ceiling	rejection 0.0325, 0.0141 to 0.0627	pass

Measured values

Every figure this section quotes is resolved by name and checked at the precision it is printed to. Names beginning margin: are thresholds declared before the run. Everything else is measured from the committed results. Both resolve the same way, so moving a threshold in the code changes this table rather than leaving it asserting a rule the study never applied.

quantity	value	source
replicates	1600	replications per law
n	1000	observations per replication
independent_tests_total	34	implementation-estimand tests against truth
independent_tests_passed	34	of those, passing
paired_tests_total	17	scenario-estimand cells compared with tmle3
paired_tests_passed	17	of those, passing
property_cells_total	12	repeated-sampling property cells
property_cells_passed	12	of those, passing
max_standardized_bias	0.1254	largest bias, in empirical standard deviations
min_coverage	0.9344	lowest measured coverage of a nominal 95% interval
min_coverage_ci_lower	0.9168	lowest exact 99% coverage endpoint, against a floor of 0.90
min_se_ratio_ci_lower	0.9040	lowest bootstrap SE-ratio endpoint, against a band of 0.80–1.20
max_se_ratio_ci_upper	1.0649	highest bootstrap SE-ratio endpoint
max_se_ratio_resolution	0.0960	how far from 1.0 the widest SE-ratio interval still reaches
max_margin_utilization	0.7445	largest share of the paired similarity margin used
max_rmse_ratio_upper	1.0088	largest one-sided RMSE-ratio bound, against a margin of 1.10
min_coverage_difference_lower	-0.0081	smallest one-sided coverage-difference bound, against -0.025
max_calibration_excess_upper	0.0129	largest comparable SE-calibration-excess bound, against 0.05
summary_cells	34	summary cells over both laws and both implementations
cells_with_se_ratio_below_one	13	of those, reporting a standard error below the empirical spread
cells_with_coverage_below_nominal	17	of those, covering below 95%
summary[cleverly/continuous/par]:coverage	0.9344	the study’s lowest coverage, cleverly
summary[tmle3/continuous/par]:coverage	0.9375	the same cell in tmle3
summary[cleverly/continuous/att]:bias	-0.000144	continuous ATT bias against known truth, cleverly
summary[tmle3/continuous/att]:bias	-0.000888	the same, tmle3
summary[cleverly/continuous/atc]:bias	0.000054	continuous ATC bias against known truth, cleverly
summary[tmle3/continuous/atc]:bias	0.000837	the same, tmle3
continuous_summary_cells	14	continuous-law summary cells, both implementations
continuous_cells_with_se_ratio_below_one	8	of those, reporting a standard error below the empirical spread
continuous_cells_with_coverage_below_nominal	7	of those, covering below 95%
properties[double_robustness/both_correct]:bias	0.0022	bias with both nuisances correct
properties[double_robustness/outcome_correct]:bias	0.000949	bias with only the outcome regression correct
properties[double_robustness/treatment_correct]:bias	-0.0035	bias with only the treatment mechanism correct
properties[double_robustness/both_wrong]:bias	-0.2926	bias with both nuisances wrong; this is the negative control
properties[double_robustness/both_wrong]:coverage	0.3167	coverage of the same control
properties[root_n_rate/empirical_sd]:slope	-0.5045	fitted log-log contraction rate of the sampling spread
properties[root_n_rate/empirical_sd]:slope_ci_lower	-0.5374	its 99% lower endpoint
properties[root_n_rate/empirical_sd]:slope_ci_upper	-0.4725	its 99% upper endpoint
properties[root_n_rate/reported_se]:slope	-0.4986	the same rate for the mean reported standard error
properties[root_n_and_efficiency/n_500]:se_ratio	0.9913	SE calibration at n = 500
properties[root_n_and_efficiency/n_2000]:se_ratio	0.9777	SE calibration at n = 2,000
properties[root_n_and_efficiency/n_8000]:se_ratio	1.0077	SE calibration at n = 8,000
properties[interval_calibration/correctly_specified]:se_ratio	1.0025	SE calibration where both nuisances are correct
properties[interval_calibration/correctly_specified]:se_ratio_ci_lower	0.9678	its 99% lower endpoint, against a band of 0.93–1.07
properties[interval_calibration/correctly_specified]:se_ratio_ci_upper	1.0384	its 99% upper endpoint
properties[interval_calibration/correctly_specified]:coverage	0.9542	coverage of the same cell
properties[type_i_error/sharp_null]:rejection_rate	0.0325	rejection under a confounded sharp null
properties[type_i_error/sharp_null]:rejection_ci_upper	0.0627	its 99% upper endpoint, against 0.05 + 0.05
properties[power/alternative]:rejection_rate	1	rejection under a real effect; this is the positive control
margin:confidence_level	0.9900	confidence level of every Monte Carlo interval below
margin:alpha	0.0500	nominal size of the estimator’s own intervals
margin:nominal_coverage	0.9500	nominal coverage those intervals claim
margin:bootstrap_replicates	10000	resamples behind every bootstrap interval
margin:standardized_bias	0.2500	bias equivalence margin, in empirical standard deviations
margin:coverage_floor	0.9000	validity floor the exact coverage lower endpoint must clear
margin:over_coverage_ceiling	0.9900	above this, coverage is conservative rather than invalid
margin:se_ratio_sanity_lower	0.8000	SE-ratio screen, lower limit
margin:se_ratio_sanity_upper	1.2000	SE-ratio screen, upper limit
margin:calibration_se_ratio_lower	0.9300	calibration-cell SE-ratio band, lower limit
margin:calibration_se_ratio_upper	1.0700	calibration-cell SE-ratio band, upper limit
margin:calibration_coverage_lower	0.9200	calibration-cell coverage band, lower limit
margin:calibration_coverage_upper	0.9800	calibration-cell coverage band, upper limit
margin:type_i_ceiling	0.1000	largest size the one-sided type-I bound may establish
margin:paired_difference	0.1500	paired similarity margin, in pooled empirical standard deviations
margin:rmse_noninferiority	1.1000	largest RMSE ratio the one-sided upper bound may reach
margin:coverage_noninferiority	-0.0250	smallest coverage difference the one-sided lower bound may reach
margin:calibration_noninferiority	0.0500	largest excess SE-calibration error the upper bound may reach
margin:minimum_power	0.8000	rejection lower bound the power control must clear
margin:root_n_slope	-0.5000	the contraction rate root-n asymptotics predict
margin:root_n_slope_lower	-0.6250	accepted slope band, lower limit
margin:root_n_slope_upper	-0.3750	accepted slope band, upper limit
margin:excluded_slope	-0.2500	the slower rate the interval must exclude
margin:union_model_se_lower	0.1000	union-model SE-ratio screen, lower limit
margin:union_model_se_upper	10	union-model SE-ratio screen, upper limit

Limitations

limitation	what it means for use
The SE sanity band is a screen, not a calibration proof	The widest bootstrap SE-ratio interval still reaches max_se_ratio_resolution from 1.0. These 34 cells cannot rule out a systematic misstatement smaller than about a tenth. The interval_calibration cell carries that claim instead, and it excludes exactly that misstatement
PAF is compared on different native scales	Point performance and coverage are compared. Raw standard-error parity is not claimed. The two intervals share a first-order delta-method limit but need not share finite-sample endpoints
Continuous ATT and ATC use the most paired margin	max_margin_utilization of it at the widest, and both stay inside. Measured against known truth, cleverly’s absolute bias on those two estimands is an order of magnitude smaller than the constrained R path’s, with essentially unchanged RMSE and coverage. The gate stays asymmetric so a future change reversing this fails the suite
The lowest coverage is continuous-law PAR	Both implementations sit low together and both clear the declared floor. What remains is a property of ordinary non-cross-fitted TMLE on this law, not of either implementation. The root_n_and_efficiency cells are the discriminator: a gap that shrinks with n is finite-sample noise, and one that persisted would be a wrong variance formula
Interval calibration shows no systematic direction	The reported standard error falls below the empirical spread in cells_with_se_ratio_below_one of summary_cells cells and above it in the rest. An earlier run at a quarter of the replication count had every continuous-law cell below one and was read as systematic understatement. Raising the replication count dissolved it
The row is bounded to ordinary TMLE on two laws	It does not establish cross-fitting, repeated cross-fitting, CV-TMLE evaluation, simultaneous intervals, bootstrap inference, missing outcomes, weights, clustering, strata, multi-valued treatment, or flexible learners. Each is a separate method or composition and needs its own study. The default-band study measures the default band of this study’s subject fit.

Reproduction

The container pins R 4.5.2 by digest, tmle3 at ed72f8a, and sl3 at 0e8f236. The fixture README gives the regeneration commands. manifest.json records the configuration and the provenance. The replications, performance tests, paired decisions and property results carry every row the tables above publish.

R uses its public specifications except for ATT and ATC. The public ATT convenience path fails to converge on a small fraction of bounded-continuous samples, so both use the constrained path the pinned package’s own tests exercise. That path is fixed for all replications.

The R harness records a failed replication. The committed run applied no failure bound, so one failure would make its cells red. The study declares a bound of 0.01 for its next run (comparator fitting failures). A failed property fit costs its cells one replicate, and the property failure accounting judges each cell.