Outcome-adaptive point-treatment C-TMLE

This study measures CTMLE(strategy="oat"), whose treatment mechanism is fitted on the vector of arm-specific outcome-regression predictions rather than on a selected subset of the original covariates. The canonical comparison uses archived tlverse ctmle3 0.1.0 at commit a4ea77b, with contemporaneous tmle3 at 3a61005 and sl3 at 821ca89.

The archive supplies implementation provenance for this OAT construction. Ju et al. (2019) derive greedy and preordered selector paths for the binary ATE, not this generated-design construction. Benkeser, Cai and van der Laan (2020) study the binary scalar case, where the mechanism already uses the estimated outcome regression. The drtmle adapt_g option builds the same design for any number of arms and reports the ordinary influence-curve covariance. Neither source covers the shipped joint all-arm targeting or simultaneous inference.

This study fits the shared design, oat_design="shared". Every shared fit has the generated_design_plugin status. The coverage and standard-error ratios below measure plugin_interval and plugin_std_error as diagnostics; their pass labels do not authorize ci, pvalue, or std_error. F19 holds the missing inference result, including the two red generated-design property cells.

What was compared

setting	cleverly	R ctmle3
construction	outcome-adaptive, non-cross-fitted	the archived non-cross-fitted OAT construction
datasets	binary samples generated in Python with exact truths	the identical rows
outcome regression	three-covariate logistic GLM	the same three-covariate logistic GLM
estimands	both treatment-specific means, ATE, marginal RR, marginal OR	the same
ratio scale	log, delta-method curves	log, delta-method curves
propensity bounds	0.025 to 0.975	no truncation of its own

The bound difference is declared rather than hidden. On this law the true propensity spans 0.085 to 0.904, so the bound is never active and no paired comparison is affected. The manifest’s g_bounds entry therefore describes the subject’s setting and not a shared one.

The primary fit draws no split at all. It is not cross-fitted, and the outcome-adaptive strategy scores no candidate path, so it needs no selection folds.

The property cells are a different setting. Twelve cells draw a sample, and the table below publishes fourteen rows, because the two root-n rate rows are derived from three of the twelve rather than fitted. Eleven of the twelve are cross-fitted, so each of them samples a proportion and declares an outcome support of 0 to 1. Each of them also declares stratify_folds="none", which reaches its outer folds and the Super Learner’s inner folds. The twelfth is the in-sample overfitting control. It draws no split and declares the same support, because the two arms of that claim must differ in whether the tree saw the rows it predicts and in nothing else.

Accuracy against known truth

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
binary-outcome law	ate	average treatment effect	cleverly outcome-adaptive C-TMLE	-0.0031 to 0.0024	0.9425	0.9787	pass
binary-outcome law	ate	average treatment effect	R ctmle3	-0.0032 to 0.0024	0.9425	0.9785	pass
binary-outcome law	ey0	counterfactual mean under no treatment	cleverly outcome-adaptive C-TMLE	-0.000870 to 0.0031	0.9625	1.0049	pass
binary-outcome law	ey0	counterfactual mean under no treatment	R ctmle3	-0.000869 to 0.0031	0.9625	1.0049	pass
binary-outcome law	ey1	counterfactual mean under treatment	cleverly outcome-adaptive C-TMLE	-0.0013 to 0.0028	0.9475	0.9814	pass
binary-outcome law	ey1	counterfactual mean under treatment	R ctmle3	-0.0013 to 0.0028	0.9475	0.9813	pass
binary-outcome law	or	marginal odds ratio, reported on the log scale	cleverly outcome-adaptive C-TMLE	-0.0117 to 0.0115	0.9437	0.9808	pass
binary-outcome law	or	marginal odds ratio, reported on the log scale	R ctmle3	-0.0117 to 0.0114	0.9437	0.9806	pass
binary-outcome law	rr	marginal risk ratio, reported on the log scale	cleverly outcome-adaptive C-TMLE	-0.0070 to 0.0055	0.9475	0.9865	pass
binary-outcome law	rr	marginal risk ratio, reported on the log scale	R ctmle3	-0.0070 to 0.0055	0.9463	0.9864	pass

Agreement with the canonical implementation

Comparator-backed: a paired comparison with tlverse-ctmle3-oat on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.

law	estimand	what was compared	paired difference	share of margin used	RMSE ratio bound	coverage difference	comparison kind	calibration resolution	result
binary-outcome law	ate	average treatment effect	0.000005	0.0011	1.0003	0	matching implementation	0.000826 vs 0.0500	equivalent
binary-outcome law	ey0	counterfactual mean under no treatment	3.839e-08	0.000012	1.0005	0	matching implementation	0.000388 vs 0.0500	equivalent
binary-outcome law	ey1	counterfactual mean under treatment	0.000005	0.0016	1.0002	0	matching implementation	0.000605 vs 0.0500	equivalent
binary-outcome law	or	marginal odds ratio, reported on the log scale	0.000043	0.0011	1.0004	0	matching implementation	0.000804 vs 0.0500	equivalent
binary-outcome law	rr	marginal risk ratio, reported on the log scale	0.000014	0.000940	1.0004	0.0012	matching implementation	0.000755 vs 0.0500	equivalent

Theory properties

property	cell	role	what was tested	what must hold	measured	result
crossfit_overfitting	cross_fitted_oat	positive	outcome-adaptive C-TMLE with cross-fitted nuisances and a flexible learner	SE ratio clears the overfitting floor and stays inside the sanity band	SE ratio 0.9066 to 1.0969	pass
crossfit_overfitting	in_sample_control	control	the same flexible learner fitted in sample, with no cross-fitting	SE ratio must fall below the overfitting ceiling	SE ratio 0.4327 to 0.5200	pass
generated_design	estimated	positive	the same design is estimated from the data, as a real fit does	SE ratio and coverage intervals both inside their calibration bands; the paired deficit is reported only	coverage 0.9295 to 0.9475, SE ratio 0.9381 to 0.9904	pass
generated_design	oracle_design	positive	the outcome-adaptive design is supplied rather than estimated	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9380 to 0.9549, SE ratio 0.9632 to 1.0153	pass
interval_calibration	correctly_specified	positive	both nuisances are correctly specified	SE ratio and coverage intervals both inside their calibration bands	coverage 0.9269 to 0.9522, SE ratio 0.9328 to 1.0042	pass
power	alternative	positive	the same test applied to a law with a real effect	rejection lower bound clears the minimum power	rejection 1, 0.9868 to 1	pass
robustness_contract	outcome_correct	positive	the outcome regression is correct and the mechanism is not	bias interval inside the equivalence margin, SE ratio must remain between 0.1 and 10.0	bias -0.000598 to 0.000924, margin 0.0026, SE ratio 0.9850	pass
robustness_contract	outcome_wrong	control	the outcome regression is misspecified	bias interval must fall entirely outside the margin, SE ratio must remain between 0.1 and 10.0	bias -0.0216 to -0.0195, margin 0.0036, SE ratio 0.9977	pass
root_n_and_efficiency	n_2000	positive	bias, coverage and SE calibration at n = 2,000	bias inside the margin, coverage clears the floor, SE ratio inside the sanity band	bias -0.000200, coverage 0.9208 to 0.9637, SE ratio 0.9749	pass
root_n_and_efficiency	n_500	positive	bias, coverage and SE calibration at n = 500	bias inside the margin, coverage clears the floor, SE ratio inside the band	bias 0.000727, coverage 0.9136 to 0.9585, SE ratio 0.9753	pass
root_n_and_efficiency	n_8000	positive	bias, coverage and SE calibration at n = 8,000	bias inside the margin, coverage clears the floor, SE ratio inside the sanity band	bias -0.000082, coverage 0.9371 to 0.9747, SE ratio 1.0179	pass
root_n_rate	empirical_sd	positive	log empirical spread of the estimates regressed on log n across three sizes	slope interval inside the root-n band and excluding -1/4	slope -0.5516 to -0.4852	pass
root_n_rate	reported_se	positive	the same regression applied to the mean reported standard error	slope interval inside the root-n band and excluding -1/4	slope -0.5040 to -0.5020	pass
type_i_error	sharp_null	positive	a confounded law whose true contrast is exactly zero	one-sided rejection bound stays under the declared type-I ceiling	rejection 0.0712, 0.0498 to 0.0980	pass

Measured values

Names beginning margin: are thresholds declared before the run. Everything else is measured from the committed results and checked at the precision printed.

quantity	value	source
replicates	800	replications of the binary law
n	1000	observations per replication
independent_tests_total	10	implementation-estimand tests against truth
independent_tests_passed	10	of those, passing
paired_tests_total	5	paired estimand tests
paired_tests_passed	5	of those, passing
property_cells_total	14	independent property cells
property_cells_passed	14	of those, passing
properties[generated_design/oracle_design]:replicates	4800	replications behind the generated-design pair, which the rule in tests/studies/bounded_cv_laws.py sizes from the control’s paired deficit
max_standardized_bias	0.0513	largest absolute primary bias in empirical standard deviations
min_coverage	0.9425	lowest primary coverage
min_coverage_ci_lower	0.9179	lowest exact 99% coverage endpoint
min_se_ratio_ci_lower	0.9211	lowest bootstrap SE-ratio endpoint
max_se_ratio_ci_upper	1.0769	highest bootstrap SE-ratio endpoint
max_margin_utilization	0.0016	largest share of the paired similarity margin used
max_rmse_ratio_upper	1.0005	largest paired RMSE-ratio upper bound
min_coverage_difference_lower	0	smallest paired coverage-difference lower bound
max_calibration_excess_upper	0.000592	largest paired SE-calibration-excess upper bound
properties[robustness_contract/outcome_correct]:standardized_bias	0.0160	outcome-correct OAT bias
properties[robustness_contract/outcome_wrong]:standardized_bias	-1.4208	outcome-wrong negative control
properties[root_n_rate/empirical_sd]:slope_ci_lower	-0.5516	99% lower endpoint of the empirical-spread rate
properties[root_n_rate/empirical_sd]:slope_ci_upper	-0.4852	its upper endpoint, which must also exclude -0.25
properties[root_n_rate/empirical_sd]:slope	-0.5184	fitted empirical-spread rate
properties[root_n_rate/reported_se]:slope	-0.5030	fitted reported-SE rate
properties[interval_calibration/correctly_specified]:coverage	0.9404	calibration-cell coverage
properties[interval_calibration/correctly_specified]:se_ratio	0.9665	calibration-cell SE ratio
properties[interval_calibration/correctly_specified]:se_ratio_ci_lower	0.9328	its 99% lower endpoint, against a band of 0.93–1.07
properties[interval_calibration/correctly_specified]:se_ratio_ci_upper	1.0042	its 99% upper endpoint
properties[type_i_error/sharp_null]:rejection_rate	0.0712	rejection under the confounded sharp null
properties[type_i_error/sharp_null]:rejection_ci_upper	0.0980	its 99% upper endpoint, against 0.05 + 0.05
properties[type_i_error/sharp_null]:coverage_ci_lower	0.9020	its exact 99% coverage endpoint, against a floor of 0.90
properties[power/alternative]:rejection_rate	1	rejection under the power control
properties[crossfit_overfitting/cross_fitted_oat]:coverage	0.9200	coverage with cross-fitted tree predictions
properties[crossfit_overfitting/cross_fitted_oat]:standardized_bias	0.3398	that arm’s bias, in empirical standard deviations
properties[crossfit_overfitting/in_sample_control]:coverage	0.5400	coverage with the in-sample tree control
properties[crossfit_overfitting/cross_fitted_oat]:se_ratio	0.9916	SE ratio with cross-fitting
properties[crossfit_overfitting/in_sample_control]:se_ratio	0.4728	SE ratio without cross-fitting
properties[crossfit_overfitting/cross_fitted_oat]:coverage_gain_ci_lower	0.3050	paired 99% lower bound for coverage gained by cross-fitting
properties[generated_design/oracle_design]:se_ratio	0.9889	SE ratio with the design pinned at the truth
properties[generated_design/oracle_design]:se_ratio_ci_lower	0.9632	its 99% lower endpoint, against a band of 0.93–1.07
properties[generated_design/oracle_design]:se_ratio_ci_upper	1.0153	its 99% upper endpoint
properties[generated_design/estimated]:se_ratio	0.9635	the same ratio with the design estimated
properties[generated_design/estimated]:se_ratio_ci_lower	0.9381	its 99% lower endpoint
properties[generated_design/estimated]:se_ratio_ci_upper	0.9904	its 99% upper endpoint
properties[generated_design/estimated]:se_ratio_deficit_lower	-0.0346	paired 99% lower endpoint for estimated minus pinned
properties[generated_design/estimated]:se_ratio_deficit_upper	-0.0164	its upper endpoint, reported as a paired finite-sample diagnostic
margin:confidence_level	0.9900	confidence level of Monte Carlo intervals
margin:alpha	0.0500	nominal estimator size
margin:nominal_coverage	0.9500	nominal estimator coverage
margin:bootstrap_replicates	10000	resamples per bootstrap interval
margin:standardized_bias	0.2500	bias equivalence margin in empirical standard deviations
margin:union_model_se_lower	0.1000	union-model SE-ratio screen, lower limit
margin:union_model_se_upper	10	union-model SE-ratio screen, upper limit
margin:coverage_floor	0.9000	validity floor for the exact coverage lower endpoint
margin:over_coverage_ceiling	0.9900	coverage above this is labeled conservative
margin:se_ratio_sanity_lower	0.8000	primary SE-ratio screen, lower limit
margin:se_ratio_sanity_upper	1.2000	primary SE-ratio screen, upper limit
margin:calibration_se_ratio_lower	0.9300	calibration SE-ratio band, lower limit
margin:calibration_se_ratio_upper	1.0700	calibration SE-ratio band, upper limit
margin:calibration_coverage_lower	0.9200	calibration coverage band, lower limit
margin:calibration_coverage_upper	0.9800	calibration coverage band, upper limit
margin:type_i_ceiling	0.1000	one-sided type-I error ceiling
margin:paired_difference	0.1500	paired mean-difference margin in pooled SDs
margin:rmse_noninferiority	1.1000	paired RMSE-ratio upper limit
margin:coverage_noninferiority	-0.0250	paired coverage-difference lower limit
margin:calibration_noninferiority	0.0500	paired calibration-excess upper limit
margin:minimum_power	0.8000	power-control rejection lower bound
margin:root_n_slope	-0.5000	predicted root-n slope
margin:root_n_slope_lower	-0.6250	accepted slope band, lower limit
margin:root_n_slope_upper	-0.3750	accepted slope band, upper limit
margin:excluded_slope	-0.2500	slower rate the interval must exclude
margin:overfit_se_floor	0.8500	cross-fit SE-ratio lower limit
margin:overfit_control_ceiling	0.7500	in-sample SE-ratio upper limit
margin:overfit_coverage_gain	0.1500	cross-fit coverage-gain lower limit

The final generated-design law and replication budget followed an earlier branch run. That earlier declaration used the positive oracle cell’s SE-ratio verdict to choose both. Commit 308467c replaced the choice with a rule based on law quadrature and the control’s paired deficit, then regenerated this study. The separately labelled pilot and its results are reported in tests/studies/bounded_cv_laws.py; this branch commits no pilot script or log.

Limitations

limitation	what it means for use
This is reporting evidence, and every cell is green	Fourteen of fourteen property cells pass, and the reporting policy means no test asserts that. On the confounded sharp-null law, the rejection rate is properties[type_i_error/sharp_null]:rejection_rate; its 99% upper endpoint is properties[type_i_error/sharp_null]:rejection_ci_upper, under the predeclared margin:type_i_ceiling, and its coverage lower endpoint is properties[type_i_error/sharp_null]:coverage_ci_lower, above the margin:coverage_floor. That cell was red on the Gaussian law this row sampled before, at the same budget of 800 and against the same two thresholds. Neither threshold moved
The property cells need a known outcome support	Eleven of the twelve cells that draw a sample are cross-fitted, so each draws a proportion and declares q_bounds of 0 to 1. The in-sample control declares the same support without cross-fitting, so the pair differs in one thing. The row says nothing about a continuous outcome whose support the analyst does not know, because the package refuses that composition under cross-fitting
One unstratified split does the cross-fitting	The outer folds read neither the treatment nor the outcome. The row does not establish a stratified split, a grouped split, or a supplied plan. The primary fit draws no split at all, so the parity claim below says nothing about any of them
OAT has a narrower robustness contract than selector C-TMLE	With the outcome regression correct, the bias interval must fit inside the equivalence margin. With it wrong, the control must be discriminated outside it. No treatment-correct-only claim is made, because OAT’s mechanism is a projection on the generated outcome-regression design rather than a fit of treatment on the original covariates
No cell measures published inference	These cells now measure a diagnostic that the package does not publish as inference. Every shared-design strategy="oat" fit takes the "generated_design_plugin" status, which RM20 decides. ci, pvalue, and std_error refuse, and the fit reports plugin_std_error and plugin_interval instead. The std_error, ci_lower, ci_upper, and covered columns of this study read those two accessors, and the null-size cell reads the p-value body that pvalue calls. The numbers are the ones the committed artifacts carry. Each accessor calls the same body as the refused one, so the reframing changed no arithmetic. The coverage, calibration, null-size, and power cells therefore describe that diagnostic
The reported curve is the ordinary EIF plug-in curve	OAT fits the treatment mechanism on estimated Qbar columns. Benkeser, Cai and van der Laan (2020) prove one binary treatment-specific-mean curve under six explicit conditions, and the implementation now follows their fold-local nuisance nesting. The package’s two-column joint fluctuation and means, ATE, RR, and OR vector are a finite-dimensional extension rather than literally that theorem. The package does not diagnose the paper’s rate, smoothness, remainder, and empirical-process conditions, whose setting is iid, complete-outcome, and unweighted. The generated_design cells compare finite-sample behaviour: with the design pinned the SE ratio is properties[generated_design/oracle_design]:se_ratio, and with it estimated properties[generated_design/estimated]:se_ratio. F19 records the remaining exact-fit and multi-arm boundary
Neither design cell’s interval on its own excludes 1	An SE ratio’s Monte Carlo error is dominated by the empirical spread in its denominator, worth about two percent at these replication counts. The paired difference resolves, because the two cells share their draws and that common error cancels. It runs from properties[generated_design/estimated]:se_ratio_deficit_lower to properties[generated_design/estimated]:se_ratio_deficit_upper, entirely below zero. This is a finite-sample generated-design effect, not proof of an omitted first-order term, and it does not show up as invalid coverage
The design comparison is not a negative control	The estimated design is judged by calibration to its own sampling spread. Its paired difference from the oracle design remains a finite-sample diagnostic and is not required to be negative. A source-backed resolution of F19 must determine whether the shipped multi-arm and cross-fitted construction needs an additional representation contribution
The cross-fit overfitting cells are relative evidence	These cells use the nonlinear bounded law, whose true propensity lies in 0.05 to 0.95. A fully grown tree on that law gives the cross-fitted estimate a bias of properties[crossfit_overfitting/cross_fitted_oat]:standardized_bias empirical standard deviations. The cell is gated on its SE ratio and on the paired gain rather than on the coverage floor. The primary GLM study carries the absolute gate
The parity claim is narrow	It is binary, two-arm, complete-outcome, GLM, and non-cross-fitted. The archived stack fails the analogous continuous law because its length-two outcome bounds enter a scalar if condition; the runner treats that as a reference limitation and not a dropped replication. The row does not establish continuous or multi-arm parity, missing outcomes, weights, clusters, strata, simultaneous or bootstrap intervals, broad learner libraries, or severe practical-positivity behaviour. Cross-fitted public behaviour rests on the property study

A nonparametric bootstrap reruns the generated design and the rest of the adaptive construction. No registered cell measures its calibration here, so this page claims nothing about it. No reviewed source validates that bootstrap for the shipped construction. The targeted-HAL bootstrap result cited in the audit holds its data-adaptive complexity bound fixed rather than reselecting it.

Reproduction

The fixture README and manifest record the provenance and the regeneration commands. The replications, paired decisions and property results carry every published row.