# Stacked point-treatment CV-TMLE

This study validates `cleverly`'s default cross-validated point-treatment construction, described
by Levy (2018). Nuisance predictions are out of fold, one targeting regression is fitted over the
stacked validation rows, and the updated regression is evaluated over the whole sample. Zheng and
van der Laan (2011) supply the wider CV-TMLE framework. The source boundary is mapped in
[CV-TMLE and cross-fitting](../cv-tmle.md#the-algorithm-as-implemented).

## What was compared

| setting | `cleverly` | R `tmle3` CV-TMLE |
| --- | --- | --- |
| construction | stacked update, whole-sample plug-in evaluation | `tmle3_Update(cvtmle = TRUE)` |
| folds | unstratified ten-fold, generated in Python | the identical validation indices, rebuilt with `origami` and asserted on the task |
| nuisance learners | GLM | corresponding GLM wrapped in `Lrnr_cv` |
| propensity bounds | 0.025 to 0.975 | 0.025 to 0.975 |
| continuous outcome scale | declared `q_bounds` of 0 to 1 | `variable_type("continuous", bounds = c(0, 1))` |
| binary outcome scale | identity, and no declaration | the inferred binomial type |
| intervals | pointwise 95% Wald | pointwise 95% Wald |
| PAF scale | identity, from the PAF influence curve | negative-log-complement, transformed |

Supplying the exact row-to-fold assignment is what makes this a fold-matched comparison. A common
seed or fold count is not enough when splitters or dependency versions differ. The R runner aborts
the whole study on a changed fold, a missing estimand, or a dropped replication. The harness
records a failed fit. The committed run applied no failure bound, so one failure would make its
cells red. The study declares a bound of 0.01 for its next run.

The split reads neither the treatment nor the outcome. A split that reads either one makes each
fold's training rows a function of the rows it predicts. The fit asserts the realized scheme
rather than trusting the keyword it was given.

Both implementations also declare the continuous outcome scale. The continuous law draws a
proportion, so 0 to 1 is the law's own support. `cleverly` refuses a cross-fitted continuous
outcome that declares no support. `tmle3` would take the scale from its realized outcomes, and the
pair would then compare two scales rather than two targeting steps. The binary law needs no declaration, because both implementations already hold a
binary outcome on the unit interval.

## Accuracy against known truth

<!-- generated: accuracy -->
| law | estimand | what was tested | implementation | bias (99% interval) | coverage | SE ratio | result |
| --- | --- | --- | --- | --- | --- | --- | --- |
| binary-outcome law | `atc` | average effect on the untreated | `cleverly` stacked CV-TMLE | -0.0043 to -0.000413 | 0.9463 | 1.0126 | pass |
| binary-outcome law | `atc` | average effect on the untreated | R `tmle3` CV-TMLE | -0.0043 to -0.000397 | 0.9469 | 1.0185 | pass |
| binary-outcome law | `ate` | average treatment effect | `cleverly` stacked CV-TMLE | -0.0040 to -0.000215 | 0.9525 | 1.0166 | pass |
| binary-outcome law | `ate` | average treatment effect | R `tmle3` CV-TMLE | -0.0040 to -0.000217 | 0.9525 | 1.0166 | pass |
| binary-outcome law | `att` | average effect on the treated | `cleverly` stacked CV-TMLE | -0.0039 to 0.000104 | 0.9525 | 1.0223 | pass |
| binary-outcome law | `att` | average effect on the treated | R `tmle3` CV-TMLE | -0.0039 to 0.000088 | 0.9519 | 1.0265 | pass |
| binary-outcome law | `ey0` | counterfactual mean under no treatment | `cleverly` stacked CV-TMLE | -0.000698 to 0.0021 | 0.9563 | 1.0094 | pass |
| binary-outcome law | `ey0` | counterfactual mean under no treatment | R `tmle3` CV-TMLE | -0.000698 to 0.0021 | 0.9563 | 1.0094 | pass |
| binary-outcome law | `ey1` | counterfactual mean under treatment | `cleverly` stacked CV-TMLE | -0.0029 to 0.000034 | 0.9356 | 0.9891 | pass |
| binary-outcome law | `ey1` | counterfactual mean under treatment | R `tmle3` CV-TMLE | -0.0029 to 0.000034 | 0.9356 | 0.9891 | pass |
| binary-outcome law | `ey_obs` | observed outcome mean under the natural course | `cleverly` stacked CV-TMLE | -0.0013 to 0.000802 | 0.9456 | 0.9815 | pass |
| binary-outcome law | `ey_obs` | observed outcome mean under the natural course | R `tmle3` CV-TMLE | -0.0013 to 0.000802 | 0.9456 | 0.9815 | pass |
| binary-outcome law | `or` | marginal odds ratio, reported on the log scale | `cleverly` stacked CV-TMLE | -0.0152 to 0.000780 | 0.9500 | 1.0170 | pass |
| binary-outcome law | `or` | marginal odds ratio, reported on the log scale | R `tmle3` CV-TMLE | -0.0152 to 0.000779 | 0.9500 | 1.0169 | pass |
| binary-outcome law | `paf` | population attributable fraction | `cleverly` stacked CV-TMLE | -0.0039 to 0.000431 | 0.9525 | 1.0242 | pass |
| binary-outcome law | `paf` | population attributable fraction | R `tmle3` CV-TMLE | -0.0040 to 0.0014 | 0.9531 | 1.0241 | pass |
| binary-outcome law | `par` | population attributable risk | `cleverly` stacked CV-TMLE | -0.0020 to 0.000065 | 0.9506 | 1.0212 | pass |
| binary-outcome law | `par` | population attributable risk | R `tmle3` CV-TMLE | -0.0020 to 0.000064 | 0.9494 | 1.0216 | pass |
| binary-outcome law | `rr` | marginal risk ratio, reported on the log scale | `cleverly` stacked CV-TMLE | -0.0079 to 0.000792 | 0.9494 | 1.0215 | pass |
| binary-outcome law | `rr` | marginal risk ratio, reported on the log scale | R `tmle3` CV-TMLE | -0.0079 to 0.000792 | 0.9494 | 1.0214 | pass |
| bounded continuous-outcome law with effect modification | `atc` | average effect on the untreated | `cleverly` stacked CV-TMLE | -0.000577 to 0.000322 | 0.9531 | 1.0084 | pass |
| bounded continuous-outcome law with effect modification | `atc` | average effect on the untreated | R `tmle3` CV-TMLE | 0.000148 to 0.0010 | 0.9525 | 1.0095 | pass |
| bounded continuous-outcome law with effect modification | `ate` | average treatment effect | `cleverly` stacked CV-TMLE | -0.000520 to 0.000313 | 0.9525 | 1.0328 | pass |
| bounded continuous-outcome law with effect modification | `ate` | average treatment effect | R `tmle3` CV-TMLE | -0.000511 to 0.000323 | 0.9544 | 1.0424 | pass |
| bounded continuous-outcome law with effect modification | `att` | average effect on the treated | `cleverly` stacked CV-TMLE | -0.000500 to 0.000404 | 0.9425 | 0.9956 | pass |
| bounded continuous-outcome law with effect modification | `att` | average effect on the treated | R `tmle3` CV-TMLE | -0.0012 to -0.000309 | 0.9413 | 0.9943 | pass |
| bounded continuous-outcome law with effect modification | `ey0` | counterfactual mean under no treatment | `cleverly` stacked CV-TMLE | -0.000440 to 0.000477 | 0.9456 | 0.9869 | pass |
| bounded continuous-outcome law with effect modification | `ey0` | counterfactual mean under no treatment | R `tmle3` CV-TMLE | -0.000439 to 0.000479 | 0.9456 | 0.9869 | pass |
| bounded continuous-outcome law with effect modification | `ey1` | counterfactual mean under treatment | `cleverly` stacked CV-TMLE | -0.000609 to 0.000438 | 0.9544 | 1.0168 | pass |
| bounded continuous-outcome law with effect modification | `ey1` | counterfactual mean under treatment | R `tmle3` CV-TMLE | -0.000607 to 0.000441 | 0.9544 | 1.0169 | pass |
| bounded continuous-outcome law with effect modification | `ey_obs` | observed outcome mean under the natural course | `cleverly` stacked CV-TMLE | -0.000462 to 0.000556 | 0.9431 | 0.9862 | pass |
| bounded continuous-outcome law with effect modification | `ey_obs` | observed outcome mean under the natural course | R `tmle3` CV-TMLE | -0.000462 to 0.000556 | 0.9431 | 0.9862 | pass |
| bounded continuous-outcome law with effect modification | `par` | population attributable risk | `cleverly` stacked CV-TMLE | -0.000265 to 0.000322 | 0.9400 | 0.9826 | pass |
| bounded continuous-outcome law with effect modification | `par` | population attributable risk | R `tmle3` CV-TMLE | -0.000295 to 0.000291 | 0.9406 | 0.9859 | pass |
<!-- /generated -->

## Agreement with the canonical implementation

<!-- generated: agreement -->
Comparator-backed: a paired comparison with `tmle3-cvtmle` on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.

| law | estimand | what was compared | paired difference | share of margin used | RMSE ratio bound | coverage difference | comparison kind | calibration resolution | result |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| binary-outcome law | `atc` | average effect on the untreated | -0.000028 | 0.0061 | 1.0072 | -0.000625 | matching implementation | 0.0124 vs 0.0500 | equivalent |
| binary-outcome law | `ate` | average treatment effect | 0.000002 | 0.000381 | 1.0001 | 0 | matching implementation | 0.000275 vs 0.0500 | equivalent |
| binary-outcome law | `att` | average effect on the treated | 0.000007 | 0.0015 | 1.0055 | 0.000625 | matching implementation | 0.0089 vs 0.0500 | equivalent |
| binary-outcome law | `ey0` | counterfactual mean under no treatment | -7.683e-08 | 0.000023 | 1.0000 | 0 | matching implementation | 0.000013 vs 0.0500 | equivalent |
| binary-outcome law | `ey1` | counterfactual mean under treatment | 4.784e-09 | 0.000001 | 1.0000 | 0 | matching implementation | 0.000077 vs 0.0500 | equivalent |
| binary-outcome law | `ey_obs` | observed outcome mean under the natural course | -6.048e-12 | 2.508e-09 | 1.0000 | 0 | matching implementation | 1.267e-10 vs 0.0500 | equivalent |
| binary-outcome law | `or` | marginal odds ratio, reported on the log scale | 0.000001 | 0.000030 | 1.0000 | 0 | matching implementation | 0.000013 vs 0.0500 | equivalent |
| binary-outcome law | `paf` | population attributable fraction | 4.002e-07 | 0.000079 | 1.0009 | -0.000625 | related target | n/a | equivalent |
| binary-outcome law | `par` | population attributable risk | 2.046e-07 | 0.000087 | 1.0009 | 0.0013 | matching implementation | 0.000961 vs 0.0500 | equivalent |
| binary-outcome law | `rr` | marginal risk ratio, reported on the log scale | 4.883e-07 | 0.000032 | 1.0000 | 0 | matching implementation | 0.000014 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `atc` | average effect on the untreated | -0.000726 | 0.6935 | 1.0024 | 0.000625 | matching implementation | 0.0031 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `ate` | average treatment effect | -0.000010 | 0.0098 | 1.0005 | -0.0019 | matching implementation | 0.0176 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `att` | average effect on the treated | 0.000714 | 0.6785 | 0.9994 | 0.0012 | matching implementation | 0.0039 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `ey0` | counterfactual mean under no treatment | -0.000001 | 0.0013 | 1.0002 | 0 | matching implementation | 0.000124 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `ey1` | counterfactual mean under treatment | -0.000002 | 0.0019 | 1.0002 | 0 | matching implementation | 0.000221 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `ey_obs` | observed outcome mean under the natural course | -7.127e-13 | 6.017e-10 | 1.0000 | 0 | matching implementation | 4.062e-11 vs 0.0500 | equivalent |
| bounded continuous-outcome law with effect modification | `par` | population attributable risk | 0.000030 | 0.0441 | 1.0025 | -0.000625 | matching implementation | 0.000680 vs 0.0500 | equivalent |
<!-- /generated -->

## Theory properties

Every cross-fitted cell samples an outcome in the open interval 0 to 1, drawn as a beta variate
around the law's conditional mean. A proportion has a known support, so each cell declares
`q_bounds` of 0 to 1 and the outcome scaler is the identity. Each law is a bounded twin of the
Gaussian law the ordinary point-treatment row samples, and it keeps that law's treatment
mechanism unchanged. `tests/studies/bounded_cv_laws.py` records the pilot measurement behind
every declared constant.

The double-robustness cells use a bounded nonlinear confounded law with exact ATE 0.2368749. Its
treatment mechanism stays between 0.076 and 0.924, so the configured bounds do not clip it. Its
conditional mean stays between 0.088 and 0.924 on the quadrature grid, so both beta shapes stay
away from zero.

The wrong main-effects outcome regression imposes a constant contrast, while the true contrast
varies with `W1` and `I(W2 > 0)`. The treatment-correct cell uses n = 2,000. The other three
cells use n = 700. Each cell uses 1,200 replications and the existing predeclared margin.

The root-n ladder and the calibration cell sample a bounded linear law with exact ATE 0.1072047.
A quasibinomial solver recovers that law's six coefficients to 1.4e-15 on a quadrature grid, so
both nuisances are correctly specified there.

The fold-policy cells report and do not gate. They run three outer-split policies on one binary
law and one set of draws: unstratified, treatment-stratified, and treatment-outcome-stratified.
The package refuses the second and the third, so the study reaches them through a study-only
subclass rather than through a supported setting. Each cell publishes its coverage interval, and
each stratified cell publishes its paired coverage difference from the unstratified arm. No cell
carries a margin, because a difference inside its own interval is equally consistent with both
policies being right and with both being wrong in the same direction.

The pass counts below therefore leave them out. A diagnostic row has no rule to fail, so counting
one would raise the published ratio by adding a cell that cannot fail. The three are counted
separately, as `property_cells_reported`.

<!-- generated: properties -->
| property | cell | role | what was tested | what must hold | measured | result |
| --- | --- | --- | --- | --- | --- | --- |
| `crossfit_overfitting` | `in_sample_control` | control | the same flexible learner fitted in sample, with no cross-fitting | SE ratio must fall below the overfitting ceiling | SE ratio 0.4535 to 0.5401 | pass |
| `crossfit_overfitting` | `stacked_cvtmle` | positive | stacked CV-TMLE with a flexible learner | SE ratio clears the overfitting floor and stays inside the sanity band | SE ratio 0.8909 to 1.0819 | pass |
| `double_robustness` | `both_correct` | positive | both the outcome regression and the treatment mechanism are correctly specified | bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread | bias -0.000589 to 0.0011, margin 0.0029, SE ratio 0.9687 | pass |
| `double_robustness` | `both_wrong` | control | both nuisances are misspecified | bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread | bias -0.0358 to -0.0335, margin 0.0040, SE ratio 1.0108 | pass |
| `double_robustness` | `outcome_correct` | positive | only the outcome regression is correctly specified | bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread | bias -0.000840 to 0.000725, margin 0.0026, SE ratio 1.0523 | pass |
| `double_robustness` | `treatment_correct` | positive | only the treatment mechanism is correctly specified | bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread | bias -0.000825 to 0.000659, margin 0.0025, SE ratio 0.9717 | pass |
| `fold_policy` | `treatment_outcome_stratified` | diagnostic | each split is balanced on the treatment crossed with the outcome, which the package refuses | reported, not gated: the coverage interval and its paired difference from the unstratified arm are published and no margin is applied | coverage 0.9252 to 0.9667, paired coverage difference -0.0025 to 0.0163 | reported |
| `fold_policy` | `treatment_stratified` | diagnostic | each split is balanced on the treatment, which the package refuses | reported, not gated: the coverage interval and its paired difference from the unstratified arm are published and no margin is applied | coverage 0.9179 to 0.9616, paired coverage difference -0.0063 to 0.0063 | reported |
| `fold_policy` | `unstratified` | diagnostic | each split is drawn without reading the treatment or the outcome | reported, not gated: the coverage interval is published and no margin is applied | coverage 0.9179 to 0.9616 | reported |
| `interval_calibration` | `correctly_specified` | positive | both nuisances are correctly specified | SE ratio and coverage intervals both inside their calibration bands | coverage 0.9346 to 0.9585, SE ratio 0.9672 to 1.0404 | pass |
| `power` | `alternative` | positive | the same test applied to a law with a real effect | rejection lower bound clears the minimum power | rejection 1, 0.9868 to 1 | pass |
| `root_n_and_efficiency` | `n_2000` | positive | bias, coverage and SE calibration at n = 2,000 | bias inside the margin, coverage clears the floor, SE ratio inside the sanity band | bias 0.000046, coverage 0.9121 to 0.9575, SE ratio 0.9709 | pass |
| `root_n_and_efficiency` | `n_500` | positive | bias, coverage and SE calibration at n = 500 | bias inside the margin, coverage clears the floor, SE ratio inside the band | bias -0.000300, coverage 0.9401 to 0.9767, SE ratio 1.0193 | pass |
| `root_n_and_efficiency` | `n_8000` | positive | bias, coverage and SE calibration at n = 8,000 | bias inside the margin, coverage clears the floor, SE ratio inside the sanity band | bias 0.000056, coverage 0.9238 to 0.9657, SE ratio 0.9996 | pass |
| `root_n_rate` | `empirical_sd` | positive | log empirical spread of the estimates regressed on log n across three sizes | slope interval inside the root-n band and excluding -1/4 | slope -0.5331 to -0.4697 | pass |
| `root_n_rate` | `reported_se` | positive | the same regression applied to the mean reported standard error | slope interval inside the root-n band and excluding -1/4 | slope -0.5093 to -0.5070 | pass |
| `type_i_error` | `sharp_null` | positive | a confounded law whose true contrast is exactly zero | one-sided rejection bound stays under the declared type-I ceiling | rejection 0.0400, 0.0191 to 0.0725 | pass |
<!-- /generated -->

## Measured values

Names beginning `margin:` are thresholds declared before the run. Everything else is measured from
the committed results and checked at the precision printed.

| quantity | value | source |
| --- | --- | --- |
| `replicates` | 1600 | replications per law |
| `n` | 1000 | observations per replication |
| `independent_tests_total` | 34 | implementation-estimand tests against truth |
| `independent_tests_passed` | 34 | of those, passing |
| `paired_tests_total` | 17 | scenario-estimand paired tests |
| `paired_tests_passed` | 17 | of those, passing |
| `property_cells_total` | 14 | repeated-sampling property cells |
| `property_cells_passed` | 14 | of those, passing |
| `property_cells_reported` | 3 | fold-policy rows reported rather than gated, and excluded from the two counts above |
| `max_standardized_bias` | 0.1085 | largest absolute bias in empirical standard deviations |
| `min_coverage` | 0.9356 | lowest measured coverage of a nominal 95% interval |
| `min_coverage_ci_lower` | 0.9182 | lowest exact 99% coverage endpoint, against 0.90 |
| `min_se_ratio_ci_lower` | 0.9377 | lowest bootstrap SE-ratio endpoint |
| `max_se_ratio_ci_upper` | 1.0925 | highest bootstrap SE-ratio endpoint |
| `max_se_ratio_resolution` | 0.0925 | farthest the widest SE-ratio interval reaches from 1.0 |
| `max_margin_utilization` | 0.6935 | largest share of a paired similarity margin used |
| `max_rmse_ratio_upper` | 1.0072 | largest one-sided RMSE-ratio bound, against 1.10 |
| `min_coverage_difference_lower` | -0.0069 | smallest one-sided coverage-difference bound, against -0.025 |
| `max_calibration_excess_upper` | 0.0080 | largest SE-calibration-excess bound, against 0.05 |
| `properties[double_robustness/outcome_correct]:bias` | -0.000057 | bias with only the outcome nuisance correct |
| `properties[double_robustness/treatment_correct]:bias` | -0.000083 | bias with only the treatment nuisance correct |
| `properties[double_robustness/both_wrong]:bias` | -0.0347 | both-wrong negative control |
| `properties[root_n_rate/empirical_sd]:slope` | -0.5011 | fitted log-log sampling-spread rate |
| `properties[root_n_rate/empirical_sd]:slope_ci_lower` | -0.5331 | its 99% lower endpoint |
| `properties[root_n_rate/empirical_sd]:slope_ci_upper` | -0.4697 | its 99% upper endpoint |
| `properties[root_n_rate/reported_se]:slope` | -0.5081 | fitted reported-SE rate |
| `properties[interval_calibration/correctly_specified]:se_ratio` | 1.0023 | SE calibration where both nuisances are correct |
| `properties[interval_calibration/correctly_specified]:se_ratio_ci_lower` | 0.9672 | its 99% lower endpoint, against a band of 0.93--1.07 |
| `properties[interval_calibration/correctly_specified]:se_ratio_ci_upper` | 1.0404 | its 99% upper endpoint |
| `properties[interval_calibration/correctly_specified]:coverage` | 0.9475 | coverage of the same cell |
| `properties[type_i_error/sharp_null]:rejection_rate` | 0.0400 | rejection under the confounded sharp null |
| `properties[type_i_error/sharp_null]:rejection_ci_upper` | 0.0725 | its 99% upper endpoint, against 0.10 |
| `properties[power/alternative]:rejection_rate` | 1 | rejection under the positive control |
| `properties[crossfit_overfitting/stacked_cvtmle]:coverage` | 0.9450 | coverage with cross-fitted tree predictions |
| `properties[crossfit_overfitting/stacked_cvtmle]:se_ratio` | 0.9770 | SE calibration with cross-fitting |
| `properties[crossfit_overfitting/in_sample_control]:coverage` | 0.5225 | coverage with the deliberately in-sample tree |
| `properties[crossfit_overfitting/in_sample_control]:se_ratio` | 0.4916 | SE calibration of that control |
| `properties[crossfit_overfitting/stacked_cvtmle]:coverage_gain_ci_lower` | 0.3575 | paired 99% lower bound for coverage gained over the control |
| `properties[fold_policy/unstratified]:coverage` | 0.9425 | reported coverage of the split the package generates |
| `properties[fold_policy/treatment_stratified]:coverage` | 0.9425 | reported coverage of the treatment-stratified split |
| `properties[fold_policy/treatment_stratified]:coverage_gain_ci_lower` | -0.0063 | its paired 99% lower bound against the unstratified arm |
| `properties[fold_policy/treatment_stratified]:coverage_gain_ci_upper` | 0.0063 | its paired 99% upper bound against the unstratified arm |
| `properties[fold_policy/treatment_outcome_stratified]:coverage` | 0.9487 | reported coverage of the treatment-outcome-stratified split |
| `properties[fold_policy/treatment_outcome_stratified]:coverage_gain_ci_lower` | -0.0025 | its paired 99% lower bound against the unstratified arm |
| `properties[fold_policy/treatment_outcome_stratified]:coverage_gain_ci_upper` | 0.0163 | its paired 99% upper bound against the unstratified arm |
| `margin:confidence_level` | 0.9900 | confidence level of every Monte Carlo interval below |
| `margin:alpha` | 0.0500 | nominal size of the estimator's own intervals |
| `margin:nominal_coverage` | 0.9500 | nominal coverage those intervals claim |
| `margin:bootstrap_replicates` | 10000 | resamples behind every bootstrap interval |
| `margin:standardized_bias` | 0.2500 | bias equivalence margin, in empirical standard deviations |
| `margin:coverage_floor` | 0.9000 | validity floor the exact coverage lower endpoint must clear |
| `margin:over_coverage_ceiling` | 0.9900 | above this, coverage is conservative rather than invalid |
| `margin:se_ratio_sanity_lower` | 0.8000 | SE-ratio screen, lower limit |
| `margin:se_ratio_sanity_upper` | 1.2000 | SE-ratio screen, upper limit |
| `margin:calibration_se_ratio_lower` | 0.9300 | calibration-cell SE-ratio band, lower limit |
| `margin:calibration_se_ratio_upper` | 1.0700 | calibration-cell SE-ratio band, upper limit |
| `margin:calibration_coverage_lower` | 0.9200 | calibration-cell coverage band, lower limit |
| `margin:calibration_coverage_upper` | 0.9800 | calibration-cell coverage band, upper limit |
| `margin:type_i_ceiling` | 0.1000 | largest size the one-sided type-I bound may establish |
| `margin:paired_difference` | 0.1500 | paired similarity margin, in pooled empirical standard deviations |
| `margin:rmse_noninferiority` | 1.1000 | largest RMSE ratio the one-sided upper bound may reach |
| `margin:coverage_noninferiority` | -0.0250 | smallest coverage difference the one-sided lower bound may reach |
| `margin:calibration_noninferiority` | 0.0500 | largest excess SE-calibration error the upper bound may reach |
| `margin:minimum_power` | 0.8000 | rejection lower bound the power control must clear |
| `margin:root_n_slope` | -0.5000 | the contraction rate root-n asymptotics predict |
| `margin:root_n_slope_lower` | -0.6250 | accepted slope band, lower limit |
| `margin:root_n_slope_upper` | -0.3750 | accepted slope band, upper limit |
| `margin:excluded_slope` | -0.2500 | the slower rate the interval must exclude |
| `margin:union_model_se_lower` | 0.1000 | union-model SE-ratio screen, lower limit |
| `margin:union_model_se_upper` | 10 | union-model SE-ratio screen, upper limit |
| `margin:overfit_se_floor` | 0.8500 | SE ratio the cross-fit arm must restore |
| `margin:overfit_control_ceiling` | 0.7500 | ceiling the in-sample control's upper bound must stay below |
| `margin:overfit_coverage_gain` | 0.1500 | coverage cross-fitting must buy over the in-sample control |

## Limitations

| limitation | what it means for use |
| --- | --- |
| PAF is compared on different native scales | `cleverly` reports its fraction-scale influence curve and `tmle3` transforms a negative-log-complement interval. Point performance and coverage are compared. Raw standard errors and finite-sample endpoints on those scales are not declared equivalent |
| The cross-fit overfitting cells are relative evidence | A fully grown regression tree is fitted twice on the nonlinear bounded law, whose true propensity lies in 0.05 to 0.95. Held-out predictions must restore the SE ratio to its band, the in-sample control's upper bound must stay below 0.75, and the paired coverage gain must clear its floor. The measured cross-fitted coverage is evidence of relative recovery and calibrated influence-curve scale. It is not a separate absolute coverage claim; the primary GLM study carries that gate |
| Continuous-law ATT and ATC use the most similarity margin | Both stay inside the predeclared bound, and the RMSE, coverage, and calibration bounds stay well inside their non-inferiority margins. These are the finite-sample cells that use the most evidence budget, not tuned exceptions |
| The continuous evidence needs a known outcome support | Every continuous cell here draws a proportion and declares `q_bounds` of 0 to 1. The row says nothing about a continuous outcome whose support the analyst does not know, because the package refuses that composition under cross-fitting |
| The fold-policy cells report and do not gate | They publish three coverage intervals and two paired differences on one binary law at n = 500. A difference inside its own interval separates no policy from another. The family establishes no policy as valid, and the package refuses two of the three |
| The row is bounded to one unstratified ten-fold split | It does not establish repeated or nested cross-fitting, fold-evaluated or fold-specific-epsilon CV-TMLE, simultaneous or bootstrap intervals, missing outcomes, weights, clusters, multi-valued treatment, broad learner-library selection, or severe practical-positivity behaviour |

## Reproduction

The [fixture README](https://github.com/esbraun/cleverly-tmle/blob/main/tests/canonical/tmle3_cvtmle/README.md)
gives the full and smoke regeneration commands. The
[manifest](https://github.com/esbraun/cleverly-tmle/blob/main/tests/canonical/tmle3_cvtmle/manifest.json)
records the configuration and the provenance. The
[replications](https://github.com/esbraun/cleverly-tmle/blob/main/tests/canonical/tmle3_cvtmle/replicates.csv.gz),
[paired decisions](https://github.com/esbraun/cleverly-tmle/blob/main/tests/canonical/tmle3_cvtmle/equivalence.csv)
and [property results](https://github.com/esbraun/cleverly-tmle/blob/main/tests/canonical/tmle3_cvtmle/properties.csv)
carry every published row.