Cross-fitted point-treatment survival#

This study validates cross-fitted longitudinal TMLE on the held point-treatment design of point-treatment survival. The fit uses five outer folds. Each fold runs an untargeted backward recursion on its training rows. One pooled fluctuation for each node then targets the stitched out-of-fold predictions (Díaz, Williams, Hoffman and Schenck 2023, Section 5.2).

No canonical implementation is compared, because no maintained package ships this construction for a held baseline treatment. The row rests on the truth tests and the property cells.

What was compared#

setting	`cleverly`
datasets	1,600 samples of 2,000 rows for each scenario
scenarios	`survival`: the five-visit law of `point-treatment-survival`, the risks of both arms and their difference at visits 1, 3 and 5. `three_arm`: three arms with a binary covariate at each visit on the wide layout, the risks of arms 0 and 2 and the differences of arms 1 and 2 against arm 0 at visits 1 and 3
nuisances	the saturated cell means, which are correct on these finite laws
targeting	one pooled fluctuation for each node, over five outer folds
intervals	pointwise 95% Wald from the stitched influence curve

Accuracy against known truth#

law	estimand	what was tested	implementation	bias (99% interval)	coverage	SE ratio	result
five visits, a binary baseline treatment held over every node, visit dropout	`ate_regimen[arm1 vs arm0 @ t=1]`	difference in cumulative risk between the plans “assign arm 1 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.0011 to 0.000826	0.9431	0.9888	pass
five visits, a binary baseline treatment held over every node, visit dropout	`ate_regimen[arm1 vs arm0 @ t=3]`	difference in cumulative risk between the plans “assign arm 1 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.0018 to 0.0012	0.9406	0.9681	pass
five visits, a binary baseline treatment held over every node, visit dropout	`ate_regimen[arm1 vs arm0 @ t=5]`	difference in cumulative risk between the plans “assign arm 1 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 5	`cleverly` cross-fitted point-treatment survival	-0.0014 to 0.0019	0.9394	0.9728	pass
five visits, a binary baseline treatment held over every node, visit dropout	`risk_regimen[arm0 @ t=1]`	cumulative risk under the plan assign arm 0 at baseline and hold it at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.000463 to 0.000986	0.9469	0.9991	pass
five visits, a binary baseline treatment held over every node, visit dropout	`risk_regimen[arm0 @ t=3]`	cumulative risk under the plan assign arm 0 at baseline and hold it at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.000297 to 0.0018	0.9431	0.9875	pass
five visits, a binary baseline treatment held over every node, visit dropout	`risk_regimen[arm0 @ t=5]`	cumulative risk under the plan assign arm 0 at baseline and hold it at horizon t = 5	`cleverly` cross-fitted point-treatment survival	-0.0010 to 0.0012	0.9500	0.9920	pass
five visits, a binary baseline treatment held over every node, visit dropout	`risk_regimen[arm1 @ t=1]`	cumulative risk under the plan assign arm 1 at baseline and hold it at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.000465 to 0.000743	0.9500	0.9882	pass
five visits, a binary baseline treatment held over every node, visit dropout	`risk_regimen[arm1 @ t=3]`	cumulative risk under the plan assign arm 1 at baseline and hold it at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.000505 to 0.0015	0.9431	0.9761	pass
five visits, a binary baseline treatment held over every node, visit dropout	`risk_regimen[arm1 @ t=5]`	cumulative risk under the plan assign arm 1 at baseline and hold it at horizon t = 5	`cleverly` cross-fitted point-treatment survival	-0.000831 to 0.0015	0.9487	0.9891	pass
three visits, three arms held over every node, a binary L_t at each later visit	`ate_regimen[arm1 vs arm0 @ t=1]`	difference in cumulative risk between the plans “assign arm 1 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.000666 to 0.0015	0.9575	1.0228	pass
three visits, three arms held over every node, a binary L_t at each later visit	`ate_regimen[arm1 vs arm0 @ t=3]`	difference in cumulative risk between the plans “assign arm 1 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.0019 to 0.0016	0.9644	1.0481	pass
three visits, three arms held over every node, a binary L_t at each later visit	`ate_regimen[arm2 vs arm0 @ t=1]`	difference in cumulative risk between the plans “assign arm 2 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.0015 to 0.000888	0.9463	0.9883	pass
three visits, three arms held over every node, a binary L_t at each later visit	`ate_regimen[arm2 vs arm0 @ t=3]`	difference in cumulative risk between the plans “assign arm 2 at baseline and hold it” against “assign arm 0 at baseline and hold it” at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.0021 to 0.0017	0.9519	0.9962	pass
three visits, three arms held over every node, a binary L_t at each later visit	`risk_regimen[arm0 @ t=1]`	cumulative risk under the plan assign arm 0 at baseline and hold it at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.0012 to 0.000404	0.9500	0.9989	pass
three visits, three arms held over every node, a binary L_t at each later visit	`risk_regimen[arm0 @ t=3]`	cumulative risk under the plan assign arm 0 at baseline and hold it at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.0013 to 0.0012	0.9569	1.0229	pass
three visits, three arms held over every node, a binary L_t at each later visit	`risk_regimen[arm2 @ t=1]`	cumulative risk under the plan assign arm 2 at baseline and hold it at horizon t = 1	`cleverly` cross-fitted point-treatment survival	-0.0015 to 0.000109	0.9463	1.0156	pass
three visits, three arms held over every node, a binary L_t at each later visit	`risk_regimen[arm2 @ t=3]`	cumulative risk under the plan assign arm 2 at baseline and hold it at horizon t = 3	`cleverly` cross-fitted point-treatment survival	-0.0017 to 0.0011	0.9513	0.9995	pass

Theory properties#

property	cell	role	what was tested	what must hold	measured	result
`double_robustness`	`survival_t5__both_correct`	positive	difference of arm 1 and arm 0 at visit 5, five-visit held law: both the outcome regression and the treatment mechanism are correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0036 to 0.000022, margin 0.0060, SE ratio 1.0285	pass
`double_robustness`	`survival_t5__both_wrong`	control	difference of arm 1 and arm 0 at visit 5, five-visit held law: both nuisances are misspecified	bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread	bias 0.0306 to 0.0343, margin 0.0063, SE ratio 0.9561	pass
`double_robustness`	`survival_t5__mechanism_correct`	positive	difference of arm 1 and arm 0 at visit 5, five-visit held law: only the treatment and censoring mechanisms are correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.0034 to 0.000118, margin 0.0060, SE ratio 1.0450	pass
`double_robustness`	`survival_t5__outcome_correct`	positive	difference of arm 1 and arm 0 at visit 5, five-visit held law: only the outcome regression is correctly specified	bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread	bias -0.000512 to 0.0031, margin 0.0061, SE ratio 0.9879	pass
`interval_calibration`	`survival_t5__correctly_specified`	positive	difference of arm 1 and arm 0 at visit 5, five-visit held law: both nuisances are correctly specified with an independently computed efficiency bound	SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands	coverage 0.9462 to 0.9575, SE ratio 0.9986 to 1.0383, empirical efficiency ratio 0.9816 to 1.0207, reported efficiency ratio 1.0189 to 1.0196	pass
`interval_calibration`	`survival_t5__noise_control`	control	difference of arm 1 and arm 0 at visit 5, five-visit held law: one efficiency-bound unit of independent noise is added to each estimate	the empirical efficiency ratio must rise above the band	coverage 0.8361 to 0.8552, SE ratio 0.7099 to 0.7373, empirical efficiency ratio 1.3824 to 1.4359, reported efficiency ratio 1.0189 to 1.0196	pass
`interval_calibration`	`survival_t5__shrunken_se_control`	control	difference of arm 1 and arm 0 at visit 5, five-visit held law: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8312 to 0.8505, SE ratio 0.6988 to 0.7258, empirical efficiency ratio 0.9831 to 1.0210, reported efficiency ratio 0.7133 to 0.7137	pass
`interval_calibration`	`three_arm_t3__correctly_specified`	positive	difference of arm 2 and arm 0 at visit 3, three arms with L_t: both nuisances are correctly specified with an independently computed efficiency bound	SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands	coverage 0.9421 to 0.9689, SE ratio 0.9650 to 1.0560, empirical efficiency ratio 1.0101 to 1.1085, reported efficiency ratio 1.0592 to 1.0764	fail
`interval_calibration`	`three_arm_t3__noise_control`	control	difference of arm 2 and arm 0 at visit 3, three arms with L_t: one efficiency-bound unit of independent noise is added to each estimate	the empirical efficiency ratio must rise above the band	coverage 0.8250 to 0.8716, SE ratio 0.7010 to 0.7659, empirical efficiency ratio 1.3923 to 1.5239, reported efficiency ratio 1.0590 to 1.0764	pass
`interval_calibration`	`three_arm_t3__shrunken_se_control`	control	difference of arm 2 and arm 0 at visit 3, three arms with L_t: the reported standard errors are multiplied by a declared factor below one	the SE-ratio interval must fall below the calibration band	coverage 0.8025 to 0.8517, SE ratio 0.6757 to 0.7382, empirical efficiency ratio 1.0108 to 1.1065, reported efficiency ratio 0.7415 to 0.7533	pass
`simultaneous_coverage`	`all_reported__pointwise_joint_control`	control	every parameter the calibration fit reports: the pointwise 95% intervals of the same fits, read jointly	joint coverage upper endpoint must fall below the nominal rate	joint coverage 0.6652 to 0.6899	pass
`simultaneous_coverage`	`all_reported__simultaneous_band`	positive	every parameter the calibration fit reports: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves	joint coverage interval inside the calibration coverage band	joint coverage 0.9418 to 0.9536	pass

The three_arm_t3__correctly_specified calibration cell is red on its empirical efficiency band. The empirical ratio is 1.058, and its 99% interval is 1.010 to 1.108 against an upper bound of 1.10. The SE ratio is 1.008 and the coverage is 0.957, so the interval is calibrated. The reported standard error is 6.7% above the bound, so the reported and the empirical spreads rise together.

The update is pooled by construction. Each outer fold fits an untargeted recursion, the fit stitches the held-out predictions, and one fluctuation for each node is solved over every follower (_pooled_targeting). So a fold-local update is not a candidate.

The diagnostic tests/diagnostics/x13_crossfit_three_arm/ refits the cell’s configuration on fresh draws, with the study’s own fit and bound. It separates a mis-scaled curve, a wrong truth or a wrong bound from a finite-sample cost. Its module records the seeds and the command. It reads the mean reported standard error over the bound in each design.

design	reported SE over the bound
n = 2,000, in sample	1.0003
n = 2,000, five folds	1.082
n = 2,000, ten folds	1.055
n = 8,000, five folds	1.0085

The in-sample fit reaches the bound, so the truth and the bound are right. The excess falls when the training folds grow, and it nearly disappears at n = 8,000. A mis-scaled curve would keep it at every size. The cell has many saturated cells in each training fold, because the three-arm law has a covariate after baseline. The diagnosis finds no defect, and the cell stays red under reporting, with the owner row X13-finite-sample.

Measured values#

Names beginning margin: are thresholds declared before the run. Everything else is measured from the committed results and checked at the precision printed.

quantity	value	source
`replicates`	1600	primary replications per scenario
`n`	2000	observations per primary replication
`independent_tests_total`	17	implementation-estimand tests against the truth
`independent_tests_passed`	17	of those, passing
`property_cells_total`	12	repeated-sampling property cells
`property_cells_passed`	11	cells whose own and family verdicts pass
`max_standardized_bias`	0.0559	largest absolute primary bias in empirical standard deviations
`min_coverage`	0.9394	lowest measured primary-study coverage
`min_se_ratio_ci_lower`	0.9267	lowest bootstrap primary SE-ratio endpoint
`max_se_ratio_ci_upper`	1.0956	highest bootstrap primary SE-ratio endpoint
`margin:alpha`	0.0500	nominal size of the reported intervals, and the family level of the rule
`margin:bootstrap_replicates`	10000	resamples behind every bootstrap interval
`margin:calibration_coverage_lower`	0.9200	calibration-cell coverage band, lower limit. No cell of this study reads it
`margin:calibration_coverage_upper`	0.9800	calibration-cell coverage band, upper limit. No cell of this study reads it
`margin:calibration_noninferiority`	0.0500	largest external-comparison calibration excess bound
`margin:calibration_se_ratio_lower`	0.9300	calibration-cell SE-ratio band, lower limit. No cell of this study reads it
`margin:calibration_se_ratio_upper`	1.0700	calibration-cell SE-ratio band, upper limit. No cell of this study reads it
`margin:confidence_level`	0.9900	confidence level of every Monte Carlo interval
`margin:coverage_floor`	0.9000	validity floor the coverage lower endpoint must clear
`margin:coverage_noninferiority`	-0.0250	smallest external-comparison coverage difference bound
`margin:efficiency_ratio_lower`	0.9000	efficiency-ratio lower bound
`margin:efficiency_ratio_upper`	1.1000	efficiency-ratio upper bound
`margin:excluded_slope`	-0.2500	slower rate a root-n interval must exclude
`margin:minimum_power`	0.8000	rejection lower bound a power cell must clear
`margin:nominal_coverage`	0.9500	nominal coverage those intervals claim
`margin:over_coverage_ceiling`	0.9900	above this, coverage is conservative rather than invalid
`margin:paired_difference`	0.1500	paired similarity margin, in pooled empirical standard deviations
`margin:rmse_noninferiority`	1.1000	largest external-comparison RMSE ratio bound
`margin:root_n_slope`	-0.5000	contraction rate root-n asymptotics predict
`margin:root_n_slope_lower`	-0.6250	accepted root-n slope band, lower limit
`margin:root_n_slope_upper`	-0.3750	accepted root-n slope band, upper limit
`margin:se_ratio_sanity_lower`	0.8000	SE-ratio screen, lower limit
`margin:se_ratio_sanity_upper`	1.2000	SE-ratio screen, upper limit
`margin:shrunken_se_factor`	0.7000	negative-control SE multiplier
`margin:standardized_bias`	0.2500	bias equivalence margin, in empirical standard deviations
`margin:type_i_ceiling`	0.1000	the rate a positive cell must bound and a control must exceed
`margin:union_model_se_lower`	0.1000	union-model SE-ratio screen, lower limit
`margin:union_model_se_upper`	10	union-model SE-ratio screen, upper limit

Limitations#

limit	what it means for use
no comparator	no maintained package fits this construction, so no row is paired
saturated cell means only	the evidence covers correct nonparametric learners on finite laws. A flexible learner is outside it
one calibration cell is red under `reporting`	the three-arm cell pays a finite-sample cost of cross-fitting at n = 2,000. The interval stays calibrated. The `X13-finite-sample` owner holds it
five folds	the fold count is fixed. The probe shows the cost falls with larger training folds

Reproduction#

The manifest records the seeds, the margins, the estimator configuration, the source hashes and the result hashes. Run python -m tests.canonical.point_survival_crossfit.regenerate to regenerate the artifacts. The replications and the property results carry every published row.