# Sensitivity and validation methods

This page accounts for every instrument `cleverly` gives you to review a method. Each entry answers
three questions in the same order. Why do you use it? What does it tell you? How does it tell you
that?

The instruments fall into four layers, and the layers see different mistakes. Read them in order.
A layer does not replace the layer above it.

| layer | the question it answers | what it cannot answer |
| --- | --- | --- |
| [Diagnostics on the fit you have](#diagnostics-on-the-fit-you-have) | did this fit, on this sample, do what the estimator asked of it? | whether the estimator asked for the right thing |
| [Sensitivity to untestable assumptions](#sensitivity-to-untestable-assumptions) | how wrong would an assumption have to be to change the conclusion? | whether the assumption is in fact wrong |
| [Refutation and simulation you run](#refutation-and-simulation-you-run) | does the fitted workflow behave as it must under a known answer? | anything the law you simulated does not contain |
| [How the library certifies itself](#how-the-library-certifies-itself) | is the derivation this library implements the correct one? | how your own data behaves |

## Diagnostics on the fit you have

These read the artifacts a completed fit already holds. They are cheap, and they run without
refitting a nuisance model.

### Positivity and overlap

**Why.** Most treatment-intervention clever covariates divide by an estimated density. A small
fitted denominator can create a large covariate. Targeting cannot create support that the data does
not contain.

**What it tells you.** The report describes fitted overlap, truncation, and concentration in
inverse-probability weights. It also describes concentration in each fitted score equation's
absolute clever-covariate load. These sample diagnostics do not prove population positivity.

**How.** `result.diagnostics.support()` returns a `PositivityReport` from
[`sensitivity/positivity.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/sensitivity/positivity.py).
It reports six separate quantities, because they fail in different places.

| quantity | how it is computed | what it describes |
| --- | --- | --- |
| per-arm weight concentration | Kish-equivalent rows and top-load shares from inverse arm weights | how unevenly the fitted arm weights are distributed |
| truncation | the units whose fitted mechanism values the bound changes | where the fit uses bounded rather than raw mechanism values |
| per-arm overlap | fitted treatment probabilities, reported by arm | empirical regions with limited fitted overlap |
| maximum clever covariate | the largest reconstructed absolute covariate value | the scale of the largest reported covariate |
| per-equation load concentration | Kish-equivalent rows and top-load shares from $|w_i H_{ij}|$ | how concentrated one fitted score equation's absolute load is |
| other mechanisms | fitted observation and intermediate probabilities when applicable | concentration outside the treatment mechanism |

The report is per arm. A multi-arm fit reads its arms from the parameter's structured index rather
than assuming two.

Missing-outcome and controlled-direct-effect reports add a derived denominator row. Its raw
quantiles describe the complete factor product. Its Kish summaries use the factorwise bounded
product on residual-contributing rows. They also include observation weights. This order matches
the targeting construction. The diagnostic never applies a separate bound to the product.

A fit with a declared missing treatment adds the factor row `P(Delta_A=1|W)` and the composite
row `P(A=a,Delta_A=1,Delta=1|W)`, or `P(A=a,Delta_A=1|W)` without `delta=`. The treatment and
response factors are read on the rows whose treatment is recorded. The composite row also reports
two numbers: its minimum, and the share of unit-arm cells below the propensity floor `g_lo`. That
floor is the scale at which R `drtmle`'s `tolg` floors the same product. The fit itself tilts the
composite inside the product of the factor floors. That product is `1e-6` at
`g_bounds=(0.01, 0.99)` and `nuisance_bound=0.01`, as
[the composite contract](dr-tmle/theorem.md#observational-missing-data-the-composite-indicator)
states.

Every mechanism row reads each unit at the arm the unit received. `Propensity` names that column.
A fit with more than two arms therefore weights each unit by its own denominator.

A derived row counts a clipped cell when any factor moves. Either bound can produce that count.
The verdict therefore names both truncation curves. It reads truncation from the factor rows and
the propensity. It reports product-weight concentration separately.

The derived row covers the marginal-mean groups. These are `mean`, `regime`, and `msm`, whose
covariate divides by the product. A fit that targets only `att`, `atc`, or an incremental
intervention gets no derived row. Those covariates divide by another quantity.

The report states that exclusion rather than leaving the row out in silence. `composed_excluded`
lists every targeted group the derived row does not describe. The summary names the alternative
denominator for each group. The list is empty when every targeted group forms the product. It is
also empty when no fitted factor stands beside `g`.

The fit retains exact absolute score weights for every reported group and equation.
`group_score_load` reads that fitted artifact.

For score equation $j$, $w_i H_{ij}$ is row $i$'s multiplier on the targeted residual. The report
summarizes its magnitude, $|w_i H_{ij}|$, for each equation separately. It does not include the
residual itself. The group row reports the equation with the smallest `targeted_ratio`.

The calculation does not collapse several equations through an L1 norm. Kish ratios and top-load
shares stay unchanged when a caller rescales one equation. The raw `max_load` retains that
equation's units.

The report does not rebuild a clever covariate from nuisance predictions. Each row pairs the
retained residual-multiplier load with the bound used by that fitted group. The `att` and `atc`
groups record `g_bounds_conditional`. The `mean` and `msm` groups record `g_bounds`. An ordinary
fit also records the bound's clipping count. If targeting changed the treatment mechanism and its
exact clipping mask was not retained, the count is unavailable and `group_score_load_omissions`
records why.

The public `diagnostics.support()` route provides group rows as follows.

| fitted group | support report | group row available |
| --- | --- | --- |
| `mean`, `att`, `atc`, or `msm` | `PositivityReport` | yes |
| regime | intervention-specific regime report | one row per regime equation |
| shift | intervention-specific shift report | one row per shift equation |
| incremental | intervention-specific incremental report | one row per outcome equation |
| longitudinal | `LongitudinalDiagnostics` from `support()` | no. A longitudinal fluctuation retains no absolute score weights |

The longitudinal row is a disclosed limitation and not an oversight. A longitudinal result returns
per-node leverage through `diagnostics.support()`.

The route reaches neither `positivity_report` nor an intervention report. A longitudinal
fluctuation also retains no `absolute_score_weights`. No fitted artifact therefore exists for a
generic or intervention load row.

The intervention-specific reports keep their ratio and support fields. Their `score_load` field
adds the fitted equation load without replacing those quantities. The report matches columns to
policies in declaration order and retains the fitted equation name. It does not parse generated
equation labels.

An incremental fit solves a second equation for the treatment mechanism. `score_load` describes
only the retained outcome residual-multiplier equation. The mechanism fluctuation does not retain
an equivalent row-level absolute load, so the report does not claim to summarize it.

The combined support row reports the most concentrated intervention equation separately. It does
not add that ratio to the mechanism effective-sample-size minimum or use it to assign status.

A fluctuation built by hand might not retain exact absolute score weights. The report does not
reconstruct an approximation. `group_score_load_omissions` names each omitted generic group and its
reason. Each intervention row uses `score_load_omission` for the same purpose.

The reason strings are machine-readable, so they are part of the API. Both paths accept an artifact
through `cleverly.data.weighting.validate_score_loads` and record the constants that module
declares. A given condition therefore produces the same text whichever report holds it.

| constant | what the report means by it |
| --- | --- |
| `SCORE_LOAD_MISSING` | no artifact arrived, so there are no exact absolute score weights |
| `SCORE_LOAD_NO_EQUATION` | an artifact arrived and the fit recorded no score equation for it |
| `SCORE_LOAD_SHAPE_MISMATCH` | the artifact is not one column per recorded score equation |
| `SCORE_LOAD_NOT_FINITE` | a load is negative, infinite, or `nan` |
| `SCORE_LOAD_EMPTY_MASK` | the fitted score mask selected no rows |
| `SCORE_LOAD_MASK_TOO_LARGE` | the mask holds more rows than the fitted data |

The table order is the guard order. A caller can be in the first two states at once, because
`check_support`, `check_policy_support`, and `check_incremental_support` all default to no artifact
and no equation names. That caller reads `SCORE_LOAD_MISSING`.

Two of those rows are behavior changes. The empty-mask and oversized-mask conditions were one reason
before, and they say opposite things: a fit that targeted nothing, and a malformed block. The
generic group path also had no size refusal at all. A mask taller than the fitted data was described
rather than omitted, and the row published a `total_ratio` above one. A direct-call table in
`tests/unit/test_intervention_load_diagnostics.py` reaches each condition, under
`test_every_refused_score_artifact_names_the_one_condition_it_failed`.

Every generic group row and intervention `score_load` row carries these concentration keys.

| key | what it holds |
| --- | --- |
| `equation` | the score equation with the smallest `targeted_ratio` |
| `n_total` | all rows in the fitted data |
| `n_targeted` | rows in the fitted score mask, including structural zeros in the selected equation |
| `effective` | Kish-equivalent rows for the selected equation's absolute load |
| `targeted_ratio` | `effective` divided by `n_targeted` |
| `total_ratio` | `effective` divided by `n_total` |
| `top_1pct` | the share of the load the largest 1% of rows hold |
| `top_5pct` | the share of the load the largest 5% of rows hold |
| `max_load` | the selected equation's largest absolute residual multiplier |
| `zero_load` | score-mask rows with a zero residual multiplier in the selected equation |
| `reported_repeat`, `n_repeats` | draw 01 and the total draw count, on intervention rows only |
| `lower_bound`, `upper_bound` | the exact treatment-mechanism bound for the group |
| `clipped_count`, `clipped_fraction` | units that bound changes, or unavailable when the exact targeted-mechanism mask was not retained |

`top_1pct` and `top_5pct` count at least one row each. Each key takes the largest
`max(1, ceil(fraction * n_targeted))` loads. At 20 score-mask rows or fewer the two keys therefore
report the same number, which is the largest single row's share. Read them as separate quantities
only above that size. The rounding rule lives in `cleverly.data.weighting.top_weight_share`.

The bound and clipping fields apply only to generic group rows. The repeat fields apply only to
intervention rows. A repeated fit's intervention load describes draw 01 and not its
coordinatewise median-combined estimate.

A generic group row carries no repeat fields. Its draw count comes from
`PositivityReport.n_repeats`, which the report records once for the whole fit. Every reader renders
the draw from the row or from the report, and no reader writes the count as a fixed string. The test
`test_the_group_load_row_counts_the_same_draws_the_report_does` in
`tests/unit/test_repeated_crossfit.py` holds the combined `support` row and
`PositivityReport.summary()` to one total.

The per-equation tests recompute these fields from the fitted score artifacts. They also use
multi-column controls with different equation scales and signs.

`max_load` and the maximum clever covariate answer different questions. The first includes score
weights and uses targeted rows. The second is unweighted and uses every row. The two need not agree.

Arm, mechanism, and group concentration are different descriptive quantities. The combined
`support` row reports them separately. It does not pool them into one minimum.

The truncation load counts units and not cells. A unit counts once when the bound moves any arm of
its mechanism. `Propensity` owns the rule the count follows. It clips a two-arm mechanism through
`g1` and takes arm 0 as the complement. It clips a mechanism with more arms column by column.

**What it grades, and what it only reports.** The report grades the truncated fraction and fitted
zero support. These findings describe the fitted procedure. They do not establish a population
positivity violation. The report does not grade any Kish ratio or top-load share. No published
result supplies a universal cutoff for these descriptive quantities.

The combined `support` row is `completed` when no graded finding applies. Do not read that status
as positivity clearance. Inspect the fitted probabilities, truncation, and concentration tables.

### Truncation stability

**Why.** A truncation bound is a finite-sample choice. A conclusion that survives only at one bound
is a conclusion about the bound.

**What it tells you.** How far the estimate moves as the bound moves.

The bound regularises the fitted procedure. It does not change the requested estimand or define an
overlap-population estimand. A moving curve shows extrapolation sensitivity through the
second-order remainder.

**How.** Point-treatment `result.diagnostics.truncation_curve()` sweeps the `g_bounds` level and
**retargets** cached nuisances through `TMLE.retarget`. Ordinary, collaborative, and unguarded
doubly-robust fits refit no nuisance model, so their curves are retarget operations.

A guarded DR-TMLE fit retargets its cached primary nuisances and refits its reduced regressions at
each bound. Its row reports the `refit` cost and requires the `retarget_cached_nuisances` replay
slot, because the curve does not call `refit()`. The function `truncation_refusal` first runs the
checks of the reduction settings that the retarget reads. It also compares `guard`, `reduction`
and `reduced_crossfit` with the fitted construction. When a fit refuses those settings or they
differ from the fitted construction, the row reads `unavailable`. The facade call and the module
call then each raise `CapabilityError`.

| test | configuration | outcome |
| --- | --- | --- |
| `TestGuardedTruncationUsesCachedNuisanceReplay` in `tests/unit/test_capability_row_predicates.py` | a stratified fold policy | the curve runs, and the three surfaces agree |
| `TestTheGuardedCurveReadsItsReductionSettings` in `tests/unit/test_summary_and_message_accuracy.py` | `targeting="one_step"` with `reduced_crossfit="nested"` | the three surfaces refuse |
| `TestTheGuardedCurveReadsItsReductionSettings` | a changed `guard`, `reduction` or `reduced_crossfit` | the three surfaces refuse |

`mechanism=None` sweeps the observation mechanism on a fit that estimates no treatment law, and
`g(W)` on every other fit. `truncation_refusal` in `sensitivity/positivity.py` then applies three
rules in order. The module call raises its sentence, and the capability row quotes it for the same
request.

| order | rule | refuses |
| --- | --- | --- |
| 1 | `observation_axis` | `mechanism=True` on a fit with no missingness and no intermediate mechanism |
| 2 | `incremental` | the treatment axis of an incremental fit, because $g$ is inside its estimand |
| 3 | `natural_course` | the treatment axis of a fit that estimates no treatment mechanism |

An incremental fit with missing outcomes reads `deferred` on `mechanism`, because
`mechanism=True` runs. `TestTheTruncationRowResolvesEachRequest` in
`tests/unit/test_capability_row_predicates.py` checks each request on five kinds of fit.

The returned frame records both ends of every evaluated pair. Scalar treatment-mechanism values
remain symmetric shorthand for `[bound, 1 - bound]`; observation- and intermediate-mechanism
values use `[bound, 1]`. It also records each parameter's own fitted pair and estimate, plus the
signed difference from that estimate. The default grid includes every exact fitted pair, without
rounding, so every parameter has one fitted marker. An explicit grid is not expanded: when it omits
the fitted configuration, the fitted columns retain that configuration and every marker is false.

Longitudinal TMLE uses a different normalization and execution path. Its grid is always explicit.
A scalar `b` means the cumulative lower-only pair `(b, 1)`. An explicit `(lower, upper)` pair keeps
both limits. Point-treatment scalar treatment bounds instead mean `(b, 1 - b)`.

For each longitudinal pair, the diagnostic keeps the realized data, resolved plans, folds,
unbounded treatment and censoring predictions, weights, clusters, and estimand structure fixed. It
recomputes each bounded cumulative mechanism prefix. It then reruns the complete backward TMLE
recursion. It refits every bound-dependent outcome or pseudo-outcome regression and every targeting
update. It reuses no earlier-node outcome fit or prediction.

The diagnostic first replays the fitted pair. Every retained estimate, regimen fit, and MSM fit
must match exactly. This preflight runs even when the requested grid omits the fitted pair. A
mismatch makes the operation unavailable.

Each row is a descriptive ordinary LTMLE point estimate under one fixed bound. It is not a
cached-nuisance retarget and does not change the causal estimand. The curve reports no standard
error, confidence interval, preferred bound, selected-bound correction, or pass threshold.

The longitudinal frame carries one row per evaluated pair and reported parameter. The curve
resolves every grid entry first and drops a repeat after that, so a repeated pair is replayed once
and reported once. A scalar `b` and the pair `(b, 1)` resolve to the same entry, so a grid carrying
both collapses to one row.

| column | what it holds |
| --- | --- |
| `lower_bound`, `upper_bound` | the cumulative pair this row was evaluated at |
| `estimand` | the reported parameter this row describes |
| `psi` | the replayed point estimate under that pair |
| `fitted_lower_bound`, `fitted_upper_bound` | the pair the fit itself used |
| `fitted_psi` | the retained estimate under that fitted pair |
| `is_fitted_bound` | whether this row's pair is the fitted pair |
| `delta_from_fitted` | `psi` minus `fitted_psi` |
| `evaluated_score_cells` | the mechanism cells the contributing fits consumed |
| `truncated_score_cells` | how many of those cells the bound moved |

The longitudinal frame names its lower endpoint `lower_bound`. The point-treatment frame above
names the same endpoint `bound`. Read the endpoint under the name its own frame uses, because
indexing a longitudinal frame at `bound` raises `KeyError`.

Each count reads only the fits that contribute to that row's parameter.

| the parameter | the fits it counts |
| --- | --- |
| a level | its own regimen fit |
| a contrast | its two regimen fits |
| an MSM coefficient | every regimen and horizon cell for its cause |

`evaluated_score_cells` counts one cell for each row in each contributing node's reporting score
mask. That mask is the node's `trained_on` set, which is the rows whose clever covariate is nonzero.
Of those cells, `truncated_score_cells` counts the ones whose bounded cumulative product differs
from the raw product.

Neither count is a count of regression rows. On a single-fold fit, the node regression is fitted on
`fitted_on`, which is that score mask. On a cross-fitted fit, each fold's untargeted regression is
fitted on the score mask inside its training rows, and the pooled fluctuation solves over the whole
score mask.

A cross-fitted replay counts one out-of-fold mechanism pair, so every scored row counts once. The
pooled fluctuation at each node divides by that pair and by nothing else. The untargeted fold
regressions read no mechanism. The count therefore has one denominator for a single-fold fit and a
cross-fitted one.

`tests/unit/test_longitudinal_truncation_refit.py` pins the counts on a two-fold fixture, measured
under the package's drawn unstratified folds. At the bound 0.12 it reports 3 truncated cells of 112
for `ey_regimen[always]`, and 22 of 112 at the bound 0.3. The same module pins the movement of that
estimate at the bound 0.3. The replayed value is 0.74038, against a fitted value of 0.56304. A
longhand recomputation with its own fold recursions and Newton solves reproduces both values.

An MSM coefficient reads every regimen and horizon cell for its cause, and each cell is its own
backward pass. Its count adds one pass per cell to the same total. On a survival MSM that total can
exceed the sample size times the node count, which bounds one complete pass. A longitudinal MSM runs
at one fold, so no fold factor enters that comparison.

The combined diagnostic row reports the signed movement range for each parameter or working-model
coefficient. The retained frame keeps the fitted pair and the per-bound values.

`estimands=` restricts the emitted rows to the parameters you name. It accepts two forms.

| you name | the curve emits |
| --- | --- |
| a reported parameter, such as `ey[high]` | the row for that parameter |
| a registered target, such as `ey` | one row for each reported alias of that target, in report order |

The two forms coincide on a two-armed fit. There the target and its one reported alias share a
spelling. The curve refuses any other name, and it names the parameters the fit carries. It also
refuses an empty selection, which would emit no row. Each refusal comes before the first retarget.

The `truncated_fraction` column counts the units the bound moves. It follows the same rule
`Propensity` applies.

The count belongs to the bound pair the curve evaluated that row at. The `support()` report always
reads `config.g_bounds`, so the two agree on a row that carries that pair. An `att` or `atc` row
carries `config.g_bounds_conditional` instead, and its count then differs. A shift fit clips no arm
probability, so the column reports no value there.

Guarded DR-TMLE and longitudinal TMLE are refit exceptions. Guarded DR-TMLE refits its reduced
regressions because the swept bounds define two regression targets. Its primary outcome regression
and propensity stay cached. Longitudinal TMLE refits the complete recursion because each later
targeted prediction becomes the preceding regression's response. Their capability rows declare
`refit` and ask for `include_refits=True`. The longitudinal row also requires explicit `bounds`.

Longitudinal replay needs a stored recipe, a cloneable outcome learner, and a cloneable
pseudo-outcome learner. The recipe stores no fitted outcome model. Cross-fitted replay rebuilds
the out-of-fold mechanism pair at each bound, because the pooled fluctuation divides by that pair.
[Replay-only unavailability](scope-and-refusals.md#replay-only-unavailability) states the
`random_state` rule each learner must satisfy, and lists every omission code that makes the
diagnostic unavailable.

### Nuisance model quality

**Why.** A nuisance model can order the rows well and still predict probabilities that are too
extreme or too moderate. The clever covariate divides by the predicted probability itself, so the
spread of the predictions matters as well as their order.

**What it tells you.** The report gives prediction loss and calibration for each retained nuisance
fit. It also gives the candidate weights and risks when a Super Learner supplies them. For C-TMLE,
it retains the selector or outcome-adaptive fit. For repeated cross-fitting, it reports parameter
movement across split draws.

**How.** `result.diagnostics.nuisance_models()` returns a report for the fitted result family.
Point-treatment results return `NuisanceDiagnostics` from
[`validation/nuisance.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/nuisance.py).
That report contains propensity AUC, the calibration slope and its standard error, outcome fit
metrics, and Super Learner candidate details. The metrics are out-of-fold when the fit cross-fitted
its nuisances. They are in-sample when it used one fold, and `evaluation` records which. The
[calibration-slope rule](#calibration-slope-rule) below reads the slope.

Longitudinal results return `LongitudinalNuisanceDiagnostics`. The report includes only models the
estimator fitted. Treatment and censoring models appear once per node because each model serves all
regimens. Outcome and pseudo-outcome models appear for each fitted regimen, cause, horizon, and
node.

| role | evaluation target | reported loss |
| --- | --- | --- |
| treatment | observed arm under the observed history | weighted negative log likelihood |
| censoring | observed retention under the observed treatment history | weighted negative log likelihood |
| outcome | final target in each fitted recursion | weighted Brier loss for a binary target, or mean squared error otherwise |
| pseudo-outcome | each earlier target in the fitted recursion | weighted mean squared error |

The treatment and censoring predictions come from the fitted observed-law pass. The report does not
refit those learners or reconstruct observed histories from regimen predictions. Each row uses
`evaluation` to label its loss as `out_of_fold` or `in_sample`.

`loss_name` is `log_loss`, `brier`, or `mse`. `loss` holds that role-specific value. The nested
`model` is a `NuisanceModelReport`. It retains calibration and learner-library details.

The `mse` field answers about node regressions alone. An outcome row or a pseudo-outcome row
reports its square loss there. A treatment row or a censoring row reports `nan`, because a
mechanism fit has no square loss. On a weighted fit, the `mse` value averages under the
observation weights.

The frame admits an empty value in `regimen`, `cause`, and `horizon`, because a mechanism row
carries no regimen identity. That row stores `None`, which a dataframe renders as `None` or as
`NaN` by column type.

For a categorical treatment, `log_loss` is the observed-class multinomial negative log likelihood.
The nested report uses `kind="multinomial probability"` and retains no binary calibration table.
Armwise calibration requires a separate report design because no treatment arm is privileged.

`LongitudinalNuisanceDiagnostics.omissions` holds typed `LongitudinalNuisanceOmission` records. Each
record names `role`, `time`, and `reason`. An outcome or pseudo-outcome record also names
`regimen`, `cause`, and `horizon`, and a fold-level record names its `folds`. The reasons are in
the table.

| reason | role and time | why the row is absent |
| --- | --- | --- |
| `LONGITUDINAL_CENSORING_NOT_FITTED` | `censoring`, no time | the design has no censoring columns, so the fit made no censoring fit |
| `LONGITUDINAL_NO_CENSORING` | `censoring` at one node | no eligible unit was censored at the node, so its factor is exactly one and no learner ran |
| `LONGITUDINAL_NO_CENSORING_IN_FOLD` | `censoring` at one node | one training fold had no censored unit and predicts one. Its held-out fold holds every censored unit at the node. The node's model row stays |
| `LONGITUDINAL_HELD_DECISION` | `treatment` at a later node | the node holds the baseline decision, so its factor is exactly one and no model ran |
| `LONGITUDINAL_CLASSIFIER_RATIO` | `treatment` at a continuous node | the classifier ratio route fits no density |
| `LONGITUDINAL_KNOWN_MECHANISM` | `treatment` or `censoring` at one node | the data declares the node's factor known, so the fit read the declaration and no learner ran |
| `LONGITUDINAL_CONSTANT_TARGET` | `outcome` or `pseudo_outcome` at one node | every follower held 0, such as no event at the node, or every follower held 1. The regression is that value and no learner ran |
| `LONGITUDINAL_CONSTANT_TARGET_IN_FOLD` | `outcome` or `pseudo_outcome` at one node | one or more training folds held 0, or held 1, while the sample did not. Those folds fitted no learner, and `folds` names them. The node's model row stays |

`to_frame()` starts with the row identity, evaluation, loss, model name, and model kind. It then
adds the union of metrics that the nested model reports. A binary row reports
`calibration_slope` and `calibration_slope_se`, from the fit's folds and cluster codes. An outcome
or pseudo-outcome row with a continuous target reports its linear slope as `regression_slope`. The
longitudinal report applies no calibration rule.

Read the propensity AUC as a positivity signal and not as a score. A higher AUC means the treatment
is more predictable, which means the arms overlap less. Higher is not better here.

One exception changes the interpretation. A C-TMLE propensity is a selected working mechanism, not
the estimated treatment law given the complete adjustment set. The report records this role in
`treatment_role` and retains the exact method artifact in `selection`. `TMLEResult.ctmle_selection`
returns that same artifact, typed. The report keeps every AUC and calibration value.

The role drops exactly two claims. Both claims read a treatment law that a collaborative fit never
estimates.

| claim the report can make about the propensity | on a collaborative fit | why |
| --- | --- | --- |
| an AUC below 0.55 means overlap is excellent and confounding by these covariates is limited | dropped | an intercept-only candidate gives an AUC near chance by construction, so the claim describes a model nobody fitted |
| a calibration-slope finding says the predictions are more extreme or more moderate than the observed rates | dropped | the same reason. A selected working mechanism has no calibration target. An intercept-only candidate has no slope at all |
| an AUC above 0.9 signals a positivity problem | kept | `CTMLE._nuisances` puts the selected mechanism on `nuisance.propensity`, so it is the denominator the clever covariate divides by. An AUC near one there means the fitted weights are near-degenerate |
| a super learner weight above 0.8 on the marginal mean means the model contributes little | kept | the note states a fact about a learner library, not an interpretation of the treatment law |

`tests/e2e/test_ctmle.py` holds both halves of this rule. Its
`test_the_role_suppresses_two_claims_and_no_others` mutates the retained report and asserts that
each dropped claim returns under `estimated_treatment_law`. Its
`test_a_mean_only_learner_library_is_reported_for_a_working_model_too` raises the mean weight on a
working model and requires the note. Use `support()` to inspect the denominator the selected
mechanism creates.

The selector artifact retains every candidate, risk, fitted fold, and the selected index. The
outcome-adaptive artifact retains its design, its outcome-prediction features and its treatment
risk, and under the per-arm design the risk of each arm. Each artifact
renders one sentence through `describe()`. The nuisance summary and the combined assessment row
both print that one sentence.

| artifact | what `describe()` renders |
| --- | --- |
| `CTMLESelection` | `C-TMLE greedy selected candidate 2 of 4 for ate` |
| `CTMLEOutcomeAdaptiveFit`, `oat_design="per_arm"` | `C-TMLE outcome-adaptive fit used one Qbar feature per arm in 2 binary mechanisms` |
| `CTMLEOutcomeAdaptiveFit`, `oat_design="shared"` | `C-TMLE outcome-adaptive fit used 2 Qbar features in one categorical mechanism` |

On a repeated fit these objects describe draw 1, because the result retains method-specific state
for that draw. The nuisance summary and the combined assessment row render the scope through
`cleverly.utils.text.format_draw`, as `draw 1 of 3`. The score-load rows above print the
zero-padded `draw 01 of 03` instead. Those rows pad every style, so a table cell and the
sentence that describes it agree on one width.

`repeat_spread` contains one `RepeatSpreadRow` per reported parameter when the fit uses two or more
draws. The row pairs the split standard deviation across every retained draw with the
median-combined result's standard error.

Both quantities are on the inference scale of the estimand. That scale is the log scale for a ratio,
and the outcome scale otherwise, so `ratio_to_standard_error` divides two like quantities.
`test_a_ratio_takes_its_spread_on_the_scale_its_standard_error_lives_on` in
`tests/unit/test_repeated_crossfit.py` reads `ate`, `rr`, and `or` off one fit. The ratio is
descriptive and has no pass threshold.

A selector-path collaborative fit supplies no standard error, so its row pairs the spread with the
plug-in diagnostic. `repeat_spread_frame()` then names the last two columns
`plugin_standard_error` and `ratio_to_plugin_standard_error`. The printed table heads them
`plugin se` and `sd/plugin se`. The `RepeatSpreadRow` fields keep their names.
`cleverly.inference.influence.spread_name` holds each renamed column.

A one-draw fit retains an empty tuple instead of reporting zero. A table cell with no finite value
prints `-`. `repeat_spread_frame()` follows the input dataframe backend.

`repeat_spread_omission` names the cause of every missing spread, and `selection_omission` does the
same for an absent artifact. Both read a constant that `cleverly.validation.nuisance` declares, so
one condition always produces one text.

| constant | what the report means by it |
| --- | --- |
| `SPREAD_SINGLE_DRAW` | the fit drew the split once, so there is no between-draw spread |
| `SPREAD_UNAVAILABLE_DRAWS` | some draw reported no estimate for the named parameters |
| `SPREAD_NOT_FINITE` | the spread exists and is not finite for the named parameters |
| `SPREAD_NO_PARAMETERS` | the result reports no parameter to take a spread of. No fitted result reaches this state, because every fit reports at least one estimand |
| `NUISANCE_SELECTION_MISSING` | the fit declares itself collaborative and retains no `ctmle` artifact |

The summary and the combined row print a spread reason only above one draw. `SPREAD_SINGLE_DRAW` is
the ordinary state of an ordinary fit, and printing it would put an "unavailable" line under every
report this package produces.

### Calibration-slope rule

**Why.** A learner can predict probabilities that are too extreme or too moderate for the observed
rates. The calibration slope measures that spread. A finding asks the analyst to review the
learner. It is not evidence that the estimate is biased.

**What it tells you.** The table defines the statistic and the rule.
`cleverly.validation.nuisance` implements both.

| term | definition |
| --- | --- |
| calibration slope | the last coefficient of a weighted logistic regression of the label on one indicator for each validation fold and on $\operatorname{logit} \hat p$, with no other intercept. A slope below 1 means the predictions are more extreme than the observed rates. A slope above 1 means they are more moderate. An in-sample fit has one fold, which gives the pooled recalibration of Cox (1958) |
| standard error | the sandwich. With $B = X^\top \operatorname{diag}(w \mu (1 - \mu)) X$ and the score $s_i = w_i (y_i - \mu_i) x_i$, the curve of the slope is $n (B^{-1} s_i)_{\mathrm{slope}}$. `influence_variance` reads the curve with the fit's cluster codes |
| omission | `NuisanceModelReport.calibration_omission` names why a report has no slope, and `summary()` prints it under the table. `to_frame()` has no column for it. `CALIBRATION_CONSTANT_WITHIN_FOLDS` is the state of a mean-only learner or an intercept-only mechanism. `CALIBRATION_SEPARATED` means that in every fold one threshold on $\operatorname{logit} \hat p$ splits the labels, the same way in each fold. The likelihood then rises without bound in the slope. A Newton fit that does not converge, a singular information matrix, or a standard error that is not finite also removes the slope |
| tested models | every probability report with a finite slope and standard error, except a C-TMLE working mechanism, on an out-of-fold fit. An in-sample fit tests none |
| level | with $K$ tested models, the interval is the slope plus or minus $z_{1 - 0.05 / (2K)}$ standard errors. That is a Bonferroni split of `CALIBRATION_FAMILY_ALPHA` over the report |
| finding | the interval lies above 0 and excludes 1 |
| no finding | the interval contains 1, reaches 0, or lies below 0 |
| message | the model, the direction, the slope, the interval, the AUC, and for `propensity`, `propensity[<arm>]` and `missingness` the largest untruncated inverse weight. That weight is the largest $1 / \hat p$ of the observed label over the rows whose weight uses it, before any truncation bound applies |

The fold intercepts matter. Cross-fitting moves the level of each fold's predictions against that
fold's labels. With one pooled intercept that movement drives the slope, and a prediction that is
constant in each fold reads close to $-(V - 1)$. At three folds,
`test_w4_a_fold_constant_prediction_has_no_slope` pins that pooled slope within 0.05 of $-2$.

Four facts shape the rule. Each has its source or its witness.

| fact | source or witness |
| --- | --- |
| the slope measures spread, not specification. The limit $\eta^*$ of a logistic model with an intercept and main effects solves $E[(A - \operatorname{expit} \eta^*)(1, \eta^*)] = 0$, so its population slope is 1 whether the model is correct or not | the score equations of the model. [Riley et al. (2021)](../references.md#calibration-of-prediction-models), Section 2.1.2, state the in-sample case |
| estimation noise can put the out-of-fold slope of a correct model below 1 when the signal is weak or the covariates are many | Riley et al. (2021), Section 2.1.2. The [calibration-slope study](method-evidence/calibration-slope-warning.md) measures the mean slope of a correct model on four covariates |
| an unpenalized logistic model with an intercept has an in-sample slope of exactly 1 | Riley et al. (2021), Section 2.1.2. `test_w10_an_in_sample_fit_carries_no_test` |
| the slope measures the spread of the predictions, and it is read with the calibration intercept, not alone | [Stevens and Poppe (2020)](../references.md#calibration-of-prediction-models), Sections 3 and 4, and [Van Calster et al. (2019)](../references.md#calibration-of-prediction-models), section "How to assess calibration?" |

No read source gives a fixed band or a rule for cross-fitted nuisance predictions. The interval, the
Bonferroni split and the gate at 0 are therefore design decisions of this package. The gate at 0
refuses to read a spread where the slope shows no association between the predictions and the
label.

**Limits.** A prediction at 0 or 1 has a logit near $\pm 27.6$ after the clip, and a few such rows
can move the slope toward 0. When the interval reaches 0, the rule gives no finding. Read the
calibration table and the AUC beside the slope.

| witness | what it checks |
| --- | --- |
| `tests/unit/test_calibration_slope_rule.py` | the slope and its standard error on weighted data with three folds equal an independent fit to 1e-10. The weights, the clusters and the fold intercepts each move the result. The file also checks the truth table of the rule, the Bonferroni divisor, the gate at 0 and the in-sample gate. A correct weak-signal fit at a seed where the old band fired gives no finding, and a doubled or halved logit gives one |
| [calibration-slope warning](method-evidence/calibration-slope-warning.md) | the standard error of the slope for a known propensity, the rule's false-warning rate on five laws, the old band's rate on the same fits, and detection on two tempered learners, over 10,000 samples each |

### Score equations

**Why.** TMLE is defined by the equation its fluctuation solves. A fit that stopped early solves it
approximately, and the reported influence curve is then not mean zero.

**What it tells you.** Whether the fluctuation reached the root of the equation the library posed.

**How.** `result.diagnostics.score_equations()` recomputes $P_n \hat{D}^*(O)$ from the fitted
influence curves and compares it against a tolerance scaled to the score's own units
([`validation/score.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/score.py)).
A point-treatment fit compares the score in the outcome's units against
`tolerance * se / sqrt(n)`. A longitudinal fit bounds each node's relative score.

**What it does not tell you.** `score_check` is necessary and not sufficient. A clever covariate
that is wrong in the same way in both the targeting step and the reported curve solves its own
equation exactly. That the equation is the right one is a claim about the library, and
[how the library certifies itself](#how-the-library-certifies-itself) is where that claim is
tested.

### Correction identities

**Why.** `DRTMLE` reports a curve assembled from three separate score equations. The curve is only
mean zero if each correction the curve subtracts is the correction whose equation the fit solved.

**What it tells you.** Two different failures, told apart. An **identity residual** means the
software solved one expression and reported another. A **correction score** means the fit did not
converge.

**How.** `result.diagnostics.corrections()` recomputes each correction's empirical mean from the
exact returned state and compares it with the score the solver recorded
([`validation/drtmle.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/drtmle.py)).
It reports the two residuals and a clipping bias term separately.

This instrument found a real defect. One clipped row in six hundred left the reported curve
uncentred at `2e-04` while all three fluctuation rows reported `1e-11`.

### Intervention support

**Why.** Arm positivity, regime support, shift support, and incremental support are four different
questions. One propensity histogram answers none of them.

**What it tells you.** Whether the declared intervention puts mass where the data has none.

**How.** Each intervention class exposes its own report through
[`interventions/support.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/interventions/support.py).
A shift fit asks whether the density *ratio* stays bounded. A per-arm propensity table has no rows
on a continuous dose, so `diagnostics.support()` dispatches to the question that applies rather
than returning an empty table.

### Design weights

**Why.** Observation weights tilt the population. Their cost to precision is separate from the
clever covariate's cost, and adding the two together hides both.

**What it tells you.** The effective sample size the declared weights leave, before any positivity
cost.

**How.** `WeightReport` in
[`data/weighting.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/data/weighting.py)
reports the weighted effective sample size and warns when the weights concentrate.

### The status contract

Every assessment row returns one of seven states, and the states are part of the contract rather than a
presentation choice.

| status | what it means |
| --- | --- |
| `passed` | the check ran and its condition holds |
| `failed` | the check ran and its condition does not hold |
| `warning` | the check ran and an explicit diagnostic rule requires qualification |
| `completed` | a descriptive analysis ran and defines no pass or fail rule |
| `deferred` | the operation can run after the caller supplies a required choice or cost opt-in |
| `not_applicable` | no such analysis exists for this scientific question |
| `unavailable` | the analysis is meaningful, but its method, derivation, fitted artifact, or requested variant cannot run it |

`AssessmentReport.summary()` prints three sections under these headings, in this order.

| section | the rows it holds | its shape |
| --- | --- | --- |
| `Returned results` | every `completed` row | one row per operation, which gives the surface, the operation, and the returned result in full text |
| `Checks` | `failed`, then `warning`, then `passed` | the first row of a status group states the status and the count. Each row names one surface-qualified operation |
| `Not run` | `deferred`, then `unavailable`, then `not_applicable` | the same compact shape as `Checks` |

An empty section states `none`. The summary ends with two footer lines. They name `to_frame()`,
`next_steps()`, and `report(...)`.

`_SUMMARY_CHECK_ORDER` and `_SUMMARY_OMISSION_ORDER` in `src/cleverly/assessment.py` pin the two
status orders above. The test
`test_the_battery_summary_expands_results_before_compact_checks_and_omissions` in
`tests/unit/test_post_fit_assessment_battery.py` builds one report that carries all seven
statuses. It reads the three section headings, the status order, and the count on each group. It
also reads the surface qualification of a duplicated name, and it asserts that the long check
detail stays out of the summary.

**The summary leads with returned results, and that order is a recorded decision.** A completed
analysis carries the numbers a reader interprets, so it expands first. A `failed` check therefore
prints below several full-width result rows.
[Pull request 198](https://github.com/esbraun/cleverly-tmle/pull/198) records the decision. Two
independent reviews found that its first draft over-prioritized warnings, and the result-first
revision resolved their findings.

The reader still finds every actionable row in the `Checks` section. That section gives one line to
each failure and each warning. Read `AssessmentReport.attention` for those rows as objects. Call
`next_steps()` for the follow-up action on each row that carries one.

`AssessmentReport.to_frame()` remains the complete row ledger, including warning and refusal
details. The three report surfaces retain the original rows. `next_steps()` retains follow-up
actions. `report(...)` returns a retained payload from an operation that ran.

The summary prints no next step for any row, and `next_steps()` returns them. Deferred rows retain
the required argument or flag in their next step. They also retain the request the caller made, and
the combined run's seed when the operation accepts one.

Known unsupported omissions carry the capability's reason and remain `unavailable` or
`not_applicable`. Such a row retains no arguments, because this fit refuses it before any request
is considered. An expected refusal after invocation becomes `unavailable` and retains the bound
invocation arguments. The report continues with other accepted operations. Structural errors still
propagate.

#### The ambiguous default estimand

Each operation below answers for one parameter, and takes that parameter from a default. A fit that
reports a bare `ate` settles the `"ate"` default. A multi-arm fit reports `ate[high vs low]` and
`ate[medium vs low]`, so it settles nothing. The caller then owns the choice. The row reports
`deferred`, and its next step names `estimand`.

| operation | facade | default estimand |
| --- | --- | --- |
| `omitted_confounding` | sensitivity | `"ate"` |
| `robustness_value` | sensitivity | `"ate"` |
| `elements` | sensitivity | `"ate"` |
| `benchmark` | sensitivity | `"ate"` |
| `contour` | sensitivity | `"ate"` |
| `tipping_gamma` | sensitivity | `"ate"` |
| `refute` | diagnostics | `"ate"` |
| `evalue` | sensitivity | `None` |

The library reads that default from the routed signature. `benchmark` takes `covariates` first and
`estimand` by keyword, and the keyword default is as ambiguous as a positional one. `evalue`
selects its own contrast from a `None` default, and defers when two contrasts qualify.

The count of eligible reported parameters selects the answer.

| eligible parameters | the combined report |
| --- | --- |
| the fit reports a bare `ate` | runs the operation |
| exactly one, on a sensitivity route | supplies that name, then runs the operation |
| exactly one, on `refute` | reports `deferred` and names `estimand` |
| two or more | reports `deferred` and names `estimand` |
| none | keeps the fit's own refusal, which is `not_applicable` or `unavailable` |

The sensitivity facade fills in a sole eligible name. `refute` fills in nothing, because it refits
under the name it is given. Its row therefore defers for one eligible alias, and the caller names
that alias to run it.

`simulated_confounding` shares the same default and is not in the table. It declares `estimand` in
`requires_arguments` beside `grid`, so the required-argument gate defers its row and names both.

An estimand the caller names is never a deferral. A name the fit never reported, an unsupported
contrast, a missing derivation, and a missing replay artifact all report `unavailable`. The facade
resolves the deferral for each request, so a supplied `estimand` makes the row available again.

`ASSESSMENT_CAPABILITIES` in
[`assessment.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/assessment.py)
declares, for each operation and each result family, the answer, the required artifacts, the cost,
and the execution class. The two costly classes are disjoint and are named separately.

A **refit**
operation fits new nuisance models. A **retarget** operation re-solves the fluctuation against
cached ones. `run_all` includes cheap retargets, such as an E-value derivation, by default.
It excludes refits and moderate retargets. Each deferred row names the flag that runs it.
The `arguments` mapping supplies required analyst choices for named operations.

An operation can decline caller-supplied arguments after its capability precheck. The combined
report records that refusal as an omission and continues with later operations. A direct call still
raises `CapabilityError` with the full refusal. Structural exceptions still stop the report.

The top-level `random_state` reaches `refute`, `benchmark`, and `simulated_confounding` only.
A seeded fit supplies its fit seed when this value is absent. An unseeded fit draws and records a
seed. Supplying the seed both places is an error.

## Sensitivity to untestable assumptions

Identification rests on assumptions that no diagnostic can test. These instruments do not test
them either. Some derive a formal scale. The simulated surface reports estimate movement
under a declared perturbation instead. Its `movement_scale` field names the scale of that
movement, which is additive for a difference and logarithmic for a ratio.

| instrument | the assumption it stresses | the number it reports | what it assumes to report it |
| --- | --- | --- | --- |
| omitted-variable bounds | no unmeasured confounding | the largest bias an unmeasured confounder of declared strength can produce | the confounder acts through the outcome regression and the Riesz representer. `cf_y` is its share of the residual outcome variation, and `cf_d` is its share of the representer's second moment |
| robustness value | no unmeasured confounding | the single strength at which the conclusion flips | that the two strengths are equal |
| benchmark | no unmeasured confounding | the strength of a confounder "as strong as" a named observed covariate | that dropping the covariate and refitting calibrates the scale |
| simulated common cause | no unmeasured confounding | estimate displacement across a declared strength grid, on the additive scale or the log scale | a supported latent perturbation family and plausible declared strengths |
| E-value | no unmeasured confounding | the equal-strength threshold for treatment and outcome confounder associations that explains away the effect | a risk-ratio scale |
| missingness tilt | outcomes missing at random | how the estimate moves as the unobserved outcomes are tilted away from the observed ones | the tilt is a constant on the logit scale |
| tipping gamma | outcomes missing at random | the tilt at which the conclusion changes | as above |

### Omitted-variable bounds, robustness value, benchmark, and contours

**How.** [`sensitivity/omitted_variable.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/sensitivity/omitted_variable.py)
implements the bounds of Equation (14) in Section 4 of Chernozhukov, Cinelli, Newey, Sharma and
Syrgkanis (2026). Their Theorem 2 gives the bias of a linear functional. The bound is

$$
|\text{bias}| \le |\rho| \sqrt{\frac{c_D^2}{1-c_D^2}}\; c_Y \sqrt{\sigma^2 \nu^2},
$$

with $\sigma^2 = E[(Y - \bar{Q})^2]$ and $\nu^2$ the second moment of the Riesz representer. The
two primitives are exposed as `elements()`.

The estimate of $\nu^2$ reads the fitted treatment mechanism, so it needs a consistent assignment
model. On a wrong model, the doubly robust estimate of $\nu^2$ falls below $\nu_0^2$ by
$\lVert \hat\alpha - \alpha_0 \rVert^2$. The bounds then read too narrow, and the robustness value
reads too large. The plug-in estimate can fall on either side of $\nu_0^2$. Witness 5 in
`tests/unit/test_omitted_variable_refusals.py` gives the plug-in 5.32 against $\nu_0^2 = 4.4$.

A plain `TMLE` fit with `treatment_adjustment=` is admitted. The fitted representer uses the
mechanism $g_n(W_g)$, which is the fit of the restricted model. So the bound is claimed only under
the declared [statistical-model restriction](point-treatment-tmle.md#treatment-mechanism-covariates).
The fit's summary prints that restriction. `tests/unit/test_treatment_covariates_collaborative.py` fits the
bound on such a fit and asserts the restriction in its summary. A collaborative fit stays refused,
including a fit that selects nothing.

`cleverly` refuses every operation in this group on the fits below. The
refusal runs in `sensitivity_elements()` before any computation. Every entry point and the
matching capability row therefore carry one reason.

| refused fit | the result the package does not have |
| --- | --- |
| a `drtmle` fit | an estimate of $\nu^2$ for an estimator that does not assume a consistent treatment mechanism. $E[2 m(\hat\alpha) - \hat\alpha^2]$ equals $\nu_0^2 - \lVert \hat\alpha - \alpha_0 \rVert^2$, so it falls where that mechanism is wrong |
| a `collaborative_tmle` fit | the same estimate. The working mechanism conditions on a function $V$ of the covariates. $V$ is the selected set $W_S$ on the greedy, ordered, and discrete paths, and the fitted outcome regression under `strategy="oat"`. Where the working mechanism is $P(A \mid V)$ in the limit, the representer is $E[\alpha_W \mid A, V]$. Its second moment cannot exceed that of $\alpha_W$, while $\sigma^2$ reads every declared covariate. `CTMLE(strategy="derivative")` is not refused. Its mechanism reads every declared covariate, as a `TMLE` mechanism does |
| a fit with a response mechanism | an implementation. The identified mean is a linear functional of the regression of $\Delta Y$ on $(A, \Delta, W)$, so Theorem 2 of the cited paper covers it. The implemented representer omits $\Delta$. $\sigma^2$ averages the respondents, and the theorem needs $E[\Delta (Y - \bar{Q})^2]$. $c_D$ would combine the treatment and response mechanisms. The missingness tilt remains the sensitivity analysis for response |
| a fit with a declared missing treatment | a derivation. The representer of the implemented bound is derived for a recorded treatment. This fit divides by the composite mechanism $P(A = a, \Delta_A = 1 \mid W)$, or $P(A = a, \Delta_A = 1, \Delta = 1 \mid W)$ with `delta=`, and no representer or confounding strength is derived for it. See [X9](../roadmap.md#x9-omitted-variable-bounds-on-the-other-linear-functionals) |
| a fit with an intermediate variable | an implementation. Each estimand at the level $z$ is a linear functional of the regression of $Y$ on $(A, Z, W)$, so Theorem 2 covers it. The representer carries the weight $1\{Z = z\} / P(Z = z \mid A, W)$, so $c_D$ would combine the treatment and intermediate mechanisms |
| a longitudinal fit | any registered longitudinal derivation. See [F16](../roadmap.md#f16-longitudinal-sensitivity-bound-estimation) |
| a `regime`, `shift`, or `msm` parameter axis | an implementation. Each of these parameters is a linear functional of the outcome regression and has a Riesz representer, so the bound is well posed here |
| a `learned_rule` parameter axis | a bound for the frozen fold rules. A benchmark refit would relearn the rules and change the target. See [X11](../roadmap.md#x11-learned-policy-follow-ups) part (n), which waits on X9 |
| an `ipsi` parameter axis | a bound that covers it. An incremental intervention tilts the treatment mechanism, so the mechanism is part of the estimand rather than a nuisance |
| an arm-indexed fit that reports no counterfactual mean and no linear contrast, such as a ratio-only, PAR, PAF, or complete-outcome `ey_obs` fit | a parameter to bound. One bound is the second moment of one contrast's own Riesz representer. On a fit that reports `rr` or `or`, the sentence names `sensitivity.evalue()` |
| a median-combined fit from `repeats=` | an influence function for the median bound. A coordinatewise median of per-draw influence terms is not one. Fit one split for this analysis |

The table rule for `repeats=` runs last. A repeated fit that another rule also refuses reports that
other rule, because one split would not lift it.

`nu2_estimator=` accepts `"auto"`, `"doubly_robust"`, and `"plugin"`. `"auto"` resolves to
`"doubly_robust"`. A value outside those three raises `ValueError` before the table above refuses a
fit. `benchmark()` checks two things before it, as the order below states. A
`"doubly_robust"` estimate that is not positive raises `CapabilityError` and names the estimator.
The package returns no plug-in value in its place, because the plug-in squares the same fitted
representer. The plug-in estimator reports no confidence limits. The
[standard error of the omitted-variable bound](#standard-error-of-the-omitted-variable-bound) gives the reason.

For the ATT and the ATC, the doubly robust score weights the contrast by $1\{A = c\} / P(A = c)$.
Here $c$ is the arm the estimand conditions on. This weight is the observed arm indicator of the
ATT and the ATU in Example 2 of Online Appendix A of the cited paper, not the fitted $\hat g_c$.
Item (2) of Theorem 5 there writes $m$ and the representer. Only the observed
indicator keeps the Riesz identity above. `TestTheConditionalEffectScoreReadsTheObservedArm` in
`tests/unit/test_omitted_variable_refusals.py` checks both conditional effects on an exact law.

`robustness_value()` inverts the bound for the single strength that flips the conclusion.
`benchmark()` drops each named observed covariate, refits, and calibrates the strength scale
against what that covariate was worth. `contour()` returns the grid a contour plot needs.

`benchmark()` reports two gain statistics. The table defines them and the share that each
stands for. Here "short" is the refit without the named covariates.

| strength | `BenchmarkResult` field | bound parameter it stands for |
| --- | --- | --- |
| `cf_y` | `gain_y` $= \sigma^2_{\text{short}} / \sigma^2 - 1$, clipped to $[0, 1]$ | $\eta^2_{Y \sim U \mid D, X}$, in $[0, 1]$ |
| `cf_d` | `gain_d` $= \nu^2 / \nu^2_{\text{short}} - 1$, clipped to $[0, 1]$ | $1 - R^2_{\alpha \sim \alpha_s}$, in $[0, 1)$ |

Pass the gains to the bound unchanged. Remark 8 of Appendix E.5 of the cited paper states that the
gains serve as proxies for the two shares when $k_Y = k_D = 1$. DoubleML uses the same
convention. Do not convert a gain to the form $G / (1 + G)$. That form is a narrower calibration
that the cited paper does not give.

`omitted_variable_bounds(result, benchmark=record)` reads `cf_y` and `cf_d` from the record. It
refuses a record of another estimand or another fit, and it refuses explicit `cf_y=` or `cf_d=`
beside the record. The benchmark does not set `rho`. Pass `rho=record.rho` to use its alignment.

| benchmark | what the bound does |
| --- | --- |
| `gain_y` above 1 | uses `cf_y = 1`. The bound is finite at $\eta^2 = 1$, which is the largest share. The summary of the record prints the unclipped gain |
| `gain_d` of 1 or more | raises `ValueError`. The bound is unbounded at `cf_d = 1`, so this record calibrates no finite bound |

`tests/unit/test_omitted_variable_benchmark_handoff.py` checks the hand-off against explicit
strengths and against the share form.

`benchmark()` is the only member of this group that refits. It reads the refusals above before it
refits, so a refused fit pays for no second fit.

`benchmark()` checks a request in this order, before the refit. The function and
`result.sensitivity.benchmark()` both call `_benchmark_names` first, so a malformed name comes
before every refusal.

| order | request | what happens |
| --- | --- | --- |
| 1 | an empty `covariates`, an indicator column of an encoded covariate, or a name that the fit does not adjust for | `DataError`. The message for an unknown name lists the covariates of the fit, with an encoded covariate once by its column name |
| 2 | a result that holds no estimator | `CapabilityError` |
| 3 | a `nu2_estimator` outside the three accepted values | `ValueError` |
| 4 | a fit that the table above refuses | `CapabilityError` with that table's sentence |
| 5 | an estimand that the bound does not cover, or that the fit did not report | `CapabilityError` from `resolve_parameter` |
| 6 | every covariate of the fit | `CapabilityError` from `benchmark_refusal`. The short model would adjust for nothing, and no estimator here fits without a covariate |

Name an encoded categorical covariate by its column. The refit then drops the whole block of
indicator columns, because one indicator alone does not define a covariate. The class
`TestBenchmarkNamesAreLogicalCovariates` in `tests/unit/test_summary_and_message_accuracy.py`
checks the dropped block. A boolean covariate is one column under its own name, and a benchmark
names it directly. The facade and a combined report read a one-shot iterator argument once, into
a tuple, and the assessment cache keys the answer by that tuple. A generator of names therefore
gives the same result as a list.

The `benchmark` capability row reads the same predicate for each request. On a fit with one
covariate, the bare row reads `unavailable`, because no value runs. On a wider fit, the bare row
keeps its `covariates` argument. `TestTheBenchmarkRowResolvesEachRequest` in
`tests/unit/test_capability_row_predicates.py` checks both fits.

### Standard error of the omitted-variable bound

Each bound has a one-sided confidence limit on each end. Theorem 4 in Section 4 of Chernozhukov,
Cinelli, Newey, Sharma and Syrgkanis (2026) gives them. The curve of a limit is the curve of the
estimate plus or minus the strength times the curve of the bias scale. The standard error is the
square root of `influence_variance` of that curve, which sums the rows of each cluster on a
clustered fit. The table gives each curve per row, with weights $w$ of mean one.

Theorem 4 inherits Lemma 3's DML conditions. They include cross-fitting, the cited DML
assumptions, and $o_P(n^{-1/4})$ $L^2$ rates for the outcome regression and representer. The
bias scale $S = \sqrt{\sigma^2\nu^2}$ must be positive. $S$ is a scale and not the largest bias,
because the strength that multiplies it can exceed 1. An in-sample TMLE also needs
conditions that control the empirical-process remainder. The registered study checks correct GLMs
on its declared law; it does not establish coverage for arbitrary learners.

The theorem's DML result uses independent rows. The cluster sum and fixed-weight calculations use
the same row curve, but the registered study does not measure their coverage.

When $\sigma^2=0$, the point bias bound is zero. The ordinary curve in the table divides by
$S=0$, so the package keeps the point bounds and refuses their limits. A robustness threshold is
`None` if no equal strength below one reaches the null. If the baseline confidence limit already
includes the null, its robustness threshold is zero.

| quantity | curve per row | source |
| --- | --- | --- |
| estimate | the estimate's own influence curve. For the ATT and the ATC it includes the share term of $\theta_s$ | Online Appendix A, "Statistical Inference", Equation (15) |
| $\sigma^2$ | $w ((Y - \bar Q^*)^2 - \sigma^2)$ | Lemma 3 |
| $\nu^2$, doubly robust | $w (2 m(O, \alpha) - \alpha^2 - \nu^2) - 2 \nu^2 w (1\{A = c\} - p) / p$ | Lemma 3 for the first term. The second term is this package's derivation |
| bias scale $S$ | $(\sigma^2 \psi_{\nu^2} + \nu^2 \psi_{\sigma^2}) / (2 \sqrt{\sigma^2 \nu^2})$ | Theorem 4 |

The second term of the $\nu^2$ curve applies to the ATT and the ATC only. Here $c$ is the arm the
parameter conditions on: the contrast arm for an ATT and the reference for an ATC. With more than
two arms each contrast has its own $c$, and $p$ is the weighted share $P_w(A = c)$ of that arm.

The representer and $m(O, \alpha)$ both divide by $p$, so the estimate is $\hat F / \hat p^2$. The
Riesz identity makes $\hat F$ insensitive to the fitted mechanism at first order, so its curve is
the Lemma 3 score. The curve of $\hat p$ is $w (1\{A = c\} - p)$, and the derivative of $F / p^2$ in $p$
is $-2 \nu^2 / p$. The delta method gives the second term.

The paper gives no such term for $\nu^2$. Equation (15) of Online Appendix A gives the term for the
estimate $\theta_s$ only. The R package `dml.sensemakr` carries the same $\nu^2$ term since commit
`d5293ecb`, and DoubleML omits it. The term sums to zero over the weighted rows, so a check of a
mean or of a point estimate cannot see it.

| witness | what it checks |
| --- | --- |
| `tests/unit/test_omitted_variable_standard_error.py` | on the two exact laws, weighted and unweighted, each curve of $\hat\nu^2$ equals the Gateaux derivative of $\nu^2$ row by row. The whole lower bound and its clustered standard error match the exact curve. Seven committed mutations fail it, and an unchanged control (M0) passes |
| [omitted-variable bound standard error](method-evidence/omitted-variable-bound-standard-error.md) | the ratio of the reported standard error to the sampling spread of each bound, over 10,000 samples, and two controls that read the curve without the share term |

Omitting the term does not always widen the limits. The table gives the lower-bound standard
error on the two-arm exact law, with the curve before RM22 and with the exact curve.

| parameter and sample | without the term | exact | effect of the omission |
| --- | ---: | ---: | --- |
| ATT, iid | 0.03882 | 0.03544 | wider |
| ATT, 50 clusters of 20 rows | 0.02129 | 0.02303 | narrower |
| ATC, 50 clusters of 20 rows | 0.02572 | 0.02413 | wider |

The direction depends on the parameter and on the sampling design. This is one exact law, so the
table shows that either direction occurs and not how often.

`nu2_estimator="plugin"` reports no confidence limits. The plug-in $E_n[\hat\alpha^2]$ moves at
first order with the fitted treatment mechanism, and its curve $\hat\alpha^2 - \nu^2$ has no term
for that fit. The cited sources give no curve that covers every mechanism learner this API accepts.
For a specified smooth, finite-dimensional mechanism model, stacked M-estimation can include its
score and Jacobian. [X26](../roadmap.md#x26-plug-in-omitted-variable-limits-with-a-declared-parametric-mechanism)
holds that specialization. `nu2_estimator="doubly_robust"` gives the limits for any mechanism
learner, and the refusal names it.

| under the plug-in | result |
| --- | --- |
| `lower`, `upper`, `max_bias`, `bias`, `robustness_value` | reported |
| `benchmark()`, `contour()`, and `nu2` and `sigma2` of `elements()` | reported |
| `ci_lower`, `ci_upper`, `robustness_value_ci`, and the three `plugin_interval_*` names | `CapabilityError`. The inference status is checked first |
| `to_dict()` | carries `nu2_estimator` and omits the three limit keys |
| `robustness_value()` | carries `nu2_estimator` and no `rva` under any name |
| `psi_nu2` and `psi_max_bias` of `elements()` | `None` |

[F26](../roadmap.md#f26-confidence-limits-of-the-plug-in-omitted-variable-bound)
holds the missing result.

The table gives each source this section cites, and the version read.

| source, and the version read | what it gives |
| --- | --- |
| Chernozhukov, Cinelli, Newey, Sharma and Syrgkanis (2026), arXiv 2112.13398v6 of 21 September 2026 | Theorem 2 in Section 3.2 gives the bias of a linear functional. Equation (14) in Section 4 gives the bounds. Lemma 3 gives the $\nu^2$ score, and Theorem 4 the curve of each bound and its one-sided limits |
| the same paper, the published Online Appendix A, read through the Internet Archive | Example 2 gives the ATT and the ATU weights $\ell(W_s)$. Item (2) of Theorem 5 writes $m$ and the representer. "Statistical Inference", Equation (15), gives the share term of $\theta_s$ |
| `dml.sensemakr`, `R/short-parameters.R`, commit `d5293ecb` | the same $\nu^2$ share term, in code, with no derivation cited |
| DoubleML, `doubleml/irm/irm.py`, commit `b69f86ef` | `_sensitivity_element_est` builds the $\nu^2$ curve without the share term |

The publisher's site refused access to the published main text, so this package did not read it.
The main-text locators follow v6. The published appendix cites "C.7. Proof of Theorem 2" and
"C.10. Proof of Lemma 3 and Theorem 4", which match the numbering of v6.

### Simulated common-cause stress surface

**Why.** An analyst can inspect whether a fitted parameter moves under a plausible latent
common cause. Sharma and Kiciman (2020) name this procedure. Sharma et al. (2021) state its
qualitative limits.

**What it tells you.** `simulated_confounding()` reports the point estimate and its displacement
at every declared strength pair. `movement_scale` states how to read that displacement.
The frame from `to_frame()` carries it as a column on every surface, additive and ratio alike.
Additive parameters and attributable fractions use `estimate_difference`. Risk and odds ratios
use `log_ratio` and the signed displacement

$$
\log(\widehat\psi_{\mathrm{refit}})-\log(\widehat\psi_{\mathrm{original}}).
$$

Its exponential is the refitted ratio divided by the original ratio. Each ratio cell keeps its
point estimate on the ratio scale. Gruber and van der Laan (2012), Section 2.1 and Section 2.7,
define the ratios and their log-scale intervals. Appendix A gives their log-scale influence
curves. The pinned `tmle3` `R/delta_functions.R` source uses the same contrasts.

Neither source defines a simulated-confounding displacement. The log difference is this package's
descriptive movement convention, not a paper-derived sensitivity measure.

Each cell also reports `induced_treatment_association`. This value is the realised correlation
between the latent vector and the treatment of that cell. The operation gives no corrected
estimate, bound, p-value, confidence interval, robustness value, threshold, or pass/fail result.

`target_measure` records how the operation reads the analysis rows. It is `unweighted` when the
fit declares no weights. It is `fixed_empirical_tilt` when the fit declares fixed weights.
`weight_report` carries the weight kind, column, scale, effective size, and concentration measures.
The label follows the declared weight column, so read `weight_report.is_weighted` to learn whether
the realized tilt is nonconstant.

**How.** The operation draws one row-level standard-normal latent vector. It reuses that vector
and one refit seed across the complete grid. Each cell starts from the original data.

For binary treatment strength $k_A$, the operation flips treatment when
$U \geq \Phi^{-1}(1-k_A)$. A continuous treatment uses $A' = A+k_AU$. Gaussian outcomes use
$Y' = Y-k_YU$. Binomial outcomes use the same tail-flip construction as binary treatment.
Flip strengths range from zero through 0.5. Continuous treatment strengths are signed finite
coefficients.

**Target populations.** The surface accepts one exact reported alias. Structured parameter metadata
identifies its arms, policy, and baseline stratum. The following contracts distinguish its populations.

| reported field | population contract |
| --- | --- |
| `stratum`, `strata_names` | The selected baseline values and their column names. A marginal parameter has no selected stratum |
| `population="baseline"` | Arm, regime, incremental, attributable, modified-policy, and MSM parameters use fixed baseline membership |
| `population="perturbed_treatment_group"` | ATT and ATC use observed-treatment membership after the cell's perturbation |
| `conditioning_arm` | The observed arm defining ATT or ATC. Other targets report `None` |
| `target_population_fraction` in each cell | The conditioning group's weighted share within the baseline population. Other targets report one |
| `association_population` | Correlation uses the selected baseline stratum, or all fitted rows for a marginal parameter. Both arms remain eligible |
| `calibration_population` | Numeric calibration uses the complete original fitted population |
| `refit_population` | Every non-anchor cell fits the complete perturbed dataset |

For baseline strata, the operation holds $S=f(W)$ fixed and changes only treatment and outcome.
The existing perturbation therefore acts on the conditional empirical law within each stratum.
One latent vector spans all rows and all strata. The operation never subsets rows before nuisance
fitting and never draws a new latent vector for each stratum. Two calls with the same seed and
different stratum aliases use the same perturbed datasets.

Ordinary TMLE supports conditional binary arm means, ATE, ratios, PAR, PAF, ATT, ATC, and regime
parameters. It also supports conditional incremental means and contrasts, MSM coefficients under
every built-in link, and modified-policy means and contrasts. Binary C-TMLE and DR-TMLE support
conditional arm means, ATE, and ratios. These paths retain fixed observation weights and estimator-owned repeat aggregation.

A DR-TMLE fit replays a requested stratum at every `guard`, through a complete refit.

ATT and ATC condition on the arm that the structured key names. Each cell recomputes that group's
membership from its perturbed treatment. Displacement combines changes in the fitted outcome
contrast and changes in population composition. It does not assess the effect among the original
treated or control rows held fixed. `target_population_fraction` makes the changing group share
visible. `CTMLE` and `DRTMLE` refuse ATT and ATC when they estimate, so this surface receives
ordinary-TMLE results only.

The DoWhy refit at `2116d5c` preserves effect modifiers and the target-unit selector. Its
propensity-weighting estimator recomputes ATT and ATC indicators from the perturbed data. This
supplies a finite-sample population convention, not a new sensitivity theorem. See the
[DoWhy source locators](../references.md#sensitivity-analysis). van der Laan (2010),
[Part I, Section 4](../references.md#point-treatment-and-stochastic-interventions), describes the
observed-treatment conditional target.

The post-fit assessment battery reads `target_population_fraction`. It reports `warning` when
`population` is `"perturbed_treatment_group"` and the smallest cell keeps less than half the anchor
cell's fraction. The anchor is the cell whose treatment strength and outcome strength are both
zero, so its fraction is the unperturbed share of the conditioning arm. The rule compares each
surface against its own anchor, so a small treated arm alone raises no warning.

The minimum covers every cell that recorded a fraction, and not the successful cells alone. A cell
whose conditioning arm empties fails its refit with `DataError`, and that cell holds the hardest
collapse on the surface. The surface records the fraction before it refits, so the failed cell
still carries the fraction.

A surface whose `population` is `"baseline"` never warns under this rule, because every cell
reports a fraction of one. The detail line carries the minimum fraction and the anchor fraction on
every surface. One test witnesses each part of the rule.

| claim | witness |
| --- | --- |
| the detail line carries both fractions | `test_post_fit_assessment_battery.py::test_descriptive_interpreters_complete_without_inventing_a_verdict` |
| the one-half threshold, from both sides | `test_simulated_confounding_populations.py::test_the_assessment_row_warns_when_the_conditioning_group_halves` |
| a failed collapsing cell still reports the collapse | `test_simulated_confounding_populations.py::test_empty_perturbed_conditioning_population_retains_a_failed_cell` |
| a `"baseline"` surface never warns | `test_post_fit_assessment_battery.py::test_a_baseline_surface_reports_no_population_collapse` |

Every file in that table sits under `tests/unit/`.

The diagnostic population differs from the ATT or ATC conditioning group. Treatment is constant
inside that group, so its correlation with the latent variable is undefined there. The surface
instead measures association across both arms within the selected baseline population. Numeric
calibration stays global; a conditional surface does not introduce a conditional calibration formula.

`tests/unit/test_simulated_confounding_populations.py` checks these contracts with complete refits
and nonzero population witnesses. Existing target instruments in the
[evidence manifest](evidence.md) establish the reused estimands. This wrapper adds no interval or
repeated-sampling claim and changes no estimator equations.

**The binary treatment axis.** The misclassification analysis below, its closed form, and the
treated-fraction table describe the binary tail flip only. The continuous law follows in its own
block.

The treatment law is non-differential misclassification. It flips a treated row and an untreated
row in the same latent tail. The association it induces between $U$ and the treatment therefore
depends on the treated fraction $\pi$. Write $q = \pi + (1-2\pi)k_A$ for the perturbed treated
fraction. When $A$ is drawn independently of $U$, the induced correlation is

$$
\operatorname{corr}(A', U) = \frac{(1-2\pi)\,\phi(\Phi^{-1}(1-k_A))}{\sqrt{q(1-q)}}.
$$

| treated fraction $\pi$ | $k_A = 0.1$ | $k_A = 0.3$ | $k_A = 0.5$ |
| --- | --- | --- | --- |
| 0.2 | +0.240 | +0.430 | +0.479 |
| 0.35 | +0.108 | +0.210 | +0.239 |
| 0.5 | +0.000 | +0.000 | +0.000 |
| 0.65 | -0.108 | -0.210 | -0.239 |
| 0.8 | -0.240 | -0.430 | -0.479 |

A 500,000-row simulation reproduces every entry to within 0.005. That bound is the Monte Carlo
error at that sample size. On a balanced design the treatment axis induces no association with
$U$. It moves the estimate through misclassification of the treatment alone. Above a treated
fraction of one half the sign reverses.

Sharma and Kiciman (2020) prescribe this construction, so the law does not change. The surface
instead reports what the law achieved on your data. Every cell carries the realised correlation
between the latent vector and its own treatment, in `induced_treatment_association`. Read that
value before you read a movement along the treatment axis as confounding. A cell near the anchor
value moved the estimate by misclassification of the treatment alone. The table above gives the
population value each cell approaches.

**The continuous treatment axis.** Under the continuous law, $U$ changes the dose by construction,
so $\operatorname{corr}(U, A')$ grows with $k_A$. A confounding path also needs $U$ to enter the
outcome. That happens only when the outcome strength is nonzero. A cell in the zero
outcome-strength column therefore carries no confounding path, whatever its association. Its
movement reports dose perturbation alone.

**The zero treatment-strength column.** A cell at $k_A=0$ leaves $U$ out of the treatment, so it
carries no confounding path either. Its movement reports the outcome perturbation alone. The
Gaussian law $Y'=Y-k_YU$ is a level shift, and the surface draws $U$ uncentred. An
`ate_policy[...]` contrast subtracts one policy mean from the other, so it removes most of that
level and the column stays small. An `ey_policy[...]` policy mean keeps it, so read the $k_A=0$
column of a policy-mean surface as an artifact of the outcome law.
`tests/unit/test_simulated_confounding.py::test_continuous_policy_mean_runs_a_real_ordinary_tmle_refit`
measures both columns on one fit.

**The reported association.** The surface measures the correlation on the analysis data. For binary
treatment, the value carries the treated fraction of your own fit. For continuous treatment, the
value reports what $A'=A+k_AU$ achieved on your dose distribution. The `(0, 0)` anchor cell
measures the original treatment, which gives the null level of the same data. A cell with constant
treatment reports no association, because its correlation is undefined. A failed cell keeps the
association of the treatment the surface built for it.

The `(0, 0)` cell returns the original estimate without a refit. A failed replacement or refit
remains visible as a `ReplicationFailure`. Successful cells report their displacement from the
original estimate, on the scale `movement_scale` names.

The surface supports these compositions with one or more cross-fitting draws.
Risk ratios, odds ratios, and PAF require a binomial outcome. The other rows support Gaussian and binomial outcomes.

| treatment | parameter | replayed estimator |
| --- | --- | --- |
| binary | backdoor-identified marginal ATE | ordinary TMLE, collaborative TMLE, or complete-outcome DR-TMLE |
| binary | one explicitly named marginal `ey1`, `ey0`, or `ey[...]` counterfactual mean | ordinary TMLE, collaborative TMLE, or complete-outcome DR-TMLE |
| binary | marginal `rr` risk ratio or `or` odds ratio | ordinary TMLE, collaborative TMLE, or complete-outcome DR-TMLE |
| binary | marginal or baseline-stratum `par` population attributable risk or `paf` population attributable fraction | exact ordinary TMLE |
| binary | baseline-stratum arm mean, ATE, risk ratio, or odds ratio | ordinary TMLE, collaborative TMLE, or complete-outcome DR-TMLE |
| binary | marginal or baseline-stratum ATT or ATC | exact ordinary TMLE |
| binary | marginal or baseline-stratum `ey_regime[...]` mean or `ate_regime[...]` contrast | exact ordinary TMLE; fixed `Static`, `Rule`, or `Stochastic` intervention |
| binary or continuous | marginal or baseline-stratum `msm[...]` coefficient | exact ordinary TMLE; built-in identity, log, or logit link with fixed projection measure |
| binary | marginal or baseline-stratum `ey_ipsi[...]` mean or `ate_ipsi[...]` contrast | exact ordinary TMLE; fixed odds multipliers with a refitted mechanism |
| continuous | one explicitly named marginal or baseline-stratum `ey_policy[...]` policy mean | exact ordinary TMLE |
| continuous | one explicitly named marginal or baseline-stratum `ate_policy[...]` contrast | exact ordinary TMLE |

Every row accepts fixed probability weights under its listed estimators.

**Fixed policies and projections.** A regime surface holds its declared density $q(a\mid W)$
fixed. It refits the observed treatment mechanism and outcome regression after each perturbation.
It then targets the same functional, $E_W\sum_a q(a\mid W)Q(a,W)$, on that cell's empirical law.
Static assignments and baseline rules are degenerate densities under this definition.

An MSM surface holds its grid design, link, and projection measure fixed.
The estimator recomputes the projection from the perturbed data. A coefficient uses additive
movement on its original scale. Projection weights define the working approximation; observation
weights define the empirical population. The surface preserves both without combining their roles.

The operation checks typed identification, parameter keys, replay configuration, and every fitted
repeat before it draws the latent vector. It evaluates the original policy or projection callbacks
through `RegimeSet.evaluate` or `MSMSet.evaluate`. Each evaluator checks the declaration of each
`Stochastic` density, or of the `MSM` design and weight, before it runs a callback. The operation
compares the callback output with the stored arrays. A refused declaration or a disagreement raises
`CapabilityError`.

The replay freezes baseline policy and grid arrays after validation.
Continuous observed-dose evaluations use the declared deterministic callbacks on each new dose vector.
This also prevents an assessment from changing the original estimator's configuration.

| input | what stays fixed |
| --- | --- |
| `Static` | the assigned treatment label |
| `Rule` | the assignment at each original baseline row |
| `Stochastic` | the complete probability vector at each original baseline row |
| finite-arm `MSM` | terms, link, arm-specific design, and known projection weights |
| continuous-dose `MSM` | terms, link, integration grid, grid design, and raw grid weights |
| `Incremental` | odds multiplier, intervention name, and reference label |
| baseline strata and observation weights | original row membership and normalized weight |

The regime functional and influence curve follow the existing
[known-regime contract](point-treatment-tmle.md#known-regimes), which cites Díaz and van der Laan (2013).
Van der Laan and Gruber (2010), Section 6, supplies the working projection.
The pinned DoWhy refuter preserves the target specification when it refits perturbed data.
Pinned `lmtp` and `tmle3` supply policy and projection implementation provenance.
The [source locators](../references.md#sensitivity-analysis) distinguish these roles.
None of these sources supplies a sensitivity-adjusted interval for this composition.

`tests/unit/test_simulated_confounding_policies.py` compares nonzero cells with independent complete
refits. It also checks static-arm identities, fixed densities, projection weights, callback drift,
metadata mutations, assessment routing, and persistence. The registered deterministic-regime,
stochastic-regime, and MSM studies validate the unchanged estimator paths.

**Incremental interventions.** Each cell preserves the declared multiplier $\delta$ and refits the treatment mechanism $g$.
The estimator rebuilds $q_\delta=\delta g/(\delta g+1-g)$ and retains its mechanism targeting contribution.
The surface therefore compares the same multiplier rule under different fitted laws.
It does not preserve the original intervention probabilities.
Kennedy (2019), equation (1) and Corollaries 1–2, supplies the parameter and influence curve.
The [source audit](../references.md#sensitivity-analysis) separates that result from the qualitative perturbation.

A multiplier-one mean equals the natural-course mean and is refused before a random draw.
A contrast against that reference remains supported. Means and contrasts report additive movement.
The operation validates every stored repeat against its own initial, untruncated propensity predictions.
It checks intervention names, multipliers, densities, clever weights, derivatives, and the reference.

**Nonlinear and continuous MSMs.** The surface preserves built-in identity, log, and logit links.
It invokes the existing complete projection and targeting solver in each cell.
Movement remains a coefficient difference, including coefficients on log or logit scales.
It never substitutes a ratio-scale coefficient view.

For continuous treatment, the declared integration grid stays fixed after $A'=A+k_AU$.
The replay freezes raw grid weights. The estimator applies its existing trapezoid rule once.
The observed-dose design, raw weights, and support mask are recomputed at $A'$.
Freezing these observed arrays would evaluate the score at the original treatment.

Deterministic callbacks must accept the new dose vector. Callback and outcome-support failures remain recorded in their cells.
Custom MSM links require their own replay audit and remain refused.

A continuous MSM has no implemented dose-grid support diagnostic.
[X65](../roadmap.md#x65-sensitivity-analyses-beyond-the-shipped-scope) part (j) holds it.
Its combined assessment records that row as unavailable and continues to the requested surface.
Outcome diagnostics remain available; integration-grid positions never become treatment-arm calibration rows.
`tests/unit/test_continuous_msm_assessment.py` checks these reporting boundaries.

| witness | what it checks |
| --- | --- |
| `tests/unit/test_simulated_confounding_incremental.py` | nonzero complete refits, mechanism rebuilding, natural-course refusals, repeated provenance, and persisted assessment |
| `tests/unit/test_simulated_confounding_msm.py` | built-in links, fixed quadrature, observed-dose rebuilding, support masks, and corrupted cached arrays |
| existing incremental and MSM estimator tests | unchanged targeting, influence curves, and projections |

These are deterministic diagnostic checks, not new repeated-sampling or interval claims.
The registered incremental and point-MSM artifacts remain unchanged.
Stratified incremental, linked-MSM and continuous-MSM fits replay too, as do stratified
DR-TMLE fits at every `guard`. Every replay is a complete refit, which targets the stratum blocks
as the fit did. `tests/unit/test_simulated_confounding_strata.py` checks that the zero-strength
cell of a stratum alias equals the fit's own stratum estimate, and that every other cell equals
the manual refit of its replaced data.
The point-MSM study covers identity-link finite-arm estimation; it supplies no nonlinear or continuous-MSM interval validation.

For `repeats > 1`, each non-anchor cell calls the replay estimator once. The estimator owns all
repeat draws and reports their coordinatewise median. Additive displacement compares the refitted
median estimate with the original median estimate. Ratio displacement compares their median log
estimates. The result records `n_repeats` and `repeat_aggregation="coordinatewise_median"`.

The root seed gives every non-anchor cell the same repeat seed sequence. The perturbation cannot
move a fold. The draw reads the row count, the cluster labels and the seed, and it balances
neither the treatment nor the outcome, so the realized partition is the same under the same seed.

The root seed reproduces the folds of the original fit only when it equals that fit's seed. The
helper `resolve_assessment_seed` returns an explicit `random_state` first, the fit's own seed
second, and a fresh seed for an unseeded fit. A refit through `TMLE.refit` reuses the estimator's own seed
sequence only for a seed equal to its `random_state`. Under any other seed each non-anchor cell
rebuilds its folds, and the anchor keeps the original ones. Movement near the anchor can therefore
carry a fold artifact.

The binary mean path keeps the named counterfactual arm fixed. Each cell replaces only the
observed treatment and outcome before the complete refit. A sole reported mean needs only the
grid. A fit that reports several means also needs one explicit alias.

The binary ratio path keeps the numerator and denominator arms fixed. It validates their direction
and the stored log estimate before the latent draw. A ratio-only result needs only the grid.

**Population attributable parameters.** Let $\mu$ be the natural-course outcome mean and $\mu_a$
the counterfactual mean at the declared reference arm. Both means describe the selected baseline
population and use its fixed weights when declared.

| parameter | cell estimate | displacement scale |
| --- | --- | --- |
| `par` | $\widehat\mu_{\mathrm{cell}}-\widehat\mu_{a,\mathrm{cell}}$ | estimate difference |
| `paf` | $1-\widehat\mu_{a,\mathrm{cell}}/\widehat\mu_{\mathrm{cell}}$ | estimate difference |

Each non-anchor cell recomputes both means from its complete refit. Its natural-course mean uses
the perturbed outcome, rather than the original outcome. The reference arm and baseline membership
remain fixed. A sole PAR or PAF alias needs only the grid. A stratified result needs an explicit alias.

Hubbard and van der Laan (2008), Sections 1 and 3, define population-intervention contrasts and
their observed-outcome contribution. PAR reverses their intervention-minus-observed difference.
PAF takes the complement of their intervention-to-observed ratio. The pinned `tmle3`
`tmle3_Spec_PAR.R` composes the same two means, and `delta_functions.R` supplies both contrasts.
The [source locators](../references.md#point-treatment-and-stochastic-interventions) distinguish
that parameter construction from the descriptive stress surface.

PAF uses `movement_scale="estimate_difference"`, even though its definition contains a ratio.
A negative PAF remains valid and is not clipped or logged. A cell with zero observed outcome risk
cannot define PAF; the surface retains its failure. These conventions do not add sensitivity-adjusted inference.

The identified effect's method catalog refuses PAR and PAF under C-TMLE and DR-TMLE before fitting.
This includes outcome-adaptive C-TMLE. Their joint observed-law and intervention construction
needs separate estimator evidence. Low-level mean-group arithmetic does not remove that gate.
The surface preserves the same refusal and replays exact ordinary TMLE only for these targets.

The tests in `tests/unit/test_simulated_confounding_attributable.py` check the composition directly.

| instrument | test |
| --- | --- |
| weighted marginal and conditional cells equal a complete refit | `test_attributable_cells_equal_complete_weighted_refits` |
| observed and reference components remain distinct from frozen-mean, sign, denominator, and arm errors | `test_observed_and_reference_components_move_with_the_cell` |
| negative fractions retain identity movement, repeated fitting, and persistence | `test_negative_fraction_uses_identity_movement_and_repeat_aggregation` |
| zero observed risk retains a failed cell | `test_zero_observed_risk_retains_a_failed_fraction_cell` |
| the pandas and polars backends report equal cells, and a sole PAR or PAF alias reaches the surface with no named estimand | `test_attributable_backend_parity_and_sole_alias_facade` |
| C-TMLE and DR-TMLE refuse a PAR or PAF fit, and the surface withholds and refuses a forged result of either kind | `test_unevidenced_attributable_fit_refuses_upstream`, `test_unreplayable_attributable_estimators_are_withheld` |
| the surface refuses the natural-course mean, and no suggested alias offers it | `test_natural_course_refuses_and_is_absent_from_suggestions` |
| every registered point target is supported or refused for a stated reason | `test_registered_point_targets_have_an_explicit_surface_disposition` |
| inconsistent result, identity, or registry metadata fails before randomness | `test_attributable_metadata_refuses_before_randomness`, `test_attributable_identity_and_registry_metadata_refuse_before_draws` |
| PAF refuses a Gaussian outcome before the first draw | `test_fraction_family_refuses_before_draws` |

The continuous path keeps the fitted modified treatment policies fixed. Each cell replaces only
the observed dose and outcome before the complete refit.

**Fixed observation weights.** Ordinary-TMLE surfaces accept declared fixed probability weights
for every row in the support table. Binary complete-outcome collaborative-TMLE and DR-TMLE
surfaces also accept them. The weights define $dP_w=w\,dP/E_P[w]$. The operation keeps each
normalized weight on its original row during both replacements and every complete refit.

The latent value stays independent and standard normal. Each weight depends only on its own
observed row, so the joint law factorizes:

$$
d(P_w\times\Phi)(o,u)=\frac{w(o)}{E_P[w]}\,dP(o)\,d\Phi(u).
$$

Benkeser et al. (2017), Theorem 1, supplies the complete-outcome DR-TMLE corrected curve and its
remainder conditions.
`tests/unit/test_remainder_drtmle.py::TestAWeightedFitTransportsToTheTiltedLaw`
transports its conditional expectations, mechanism, marginal means, and scores to $P_w$. It keeps
a wrong-transport control with a nonzero remainder.
`tests/unit/test_simulated_confounding.py::test_fixed_weight_drtmle_witnesses_the_weight_on_the_reduced_regressions`
adds the applied witness. It removes the weight from the reduced regressions alone, and it
requires the cell estimate to move.

Van der Laan and Gruber (2010), Sections 2, 5.1, and 6, define C-TMLE for a generic law and its
empirical distribution. Replacing that law by $P_w$ makes $P_{n,w}$ the empirical measure.
Selector strategies use the same normalized row mass in nuisance fits, targeting, outcome loss,
the influence-curve penalty, cross-validated risk, and the plug-in. The outcome-adaptive strategy
uses it in the outcome fits, the mechanism fits, targeting, and plug-in.

The fixed-weight C-TMLE tests cover greedy selection, both data-adaptive ordered preorders, an
explicit ordering, discrete selection, and outcome-adaptive fitting. They reconstruct a selected
path's weighted squared loss, weighted log-likelihood loss, influence-curve penalty, and nested
cross-validated risk term by term. They do not read the value that a production scoring method
returns. The nested reconstruction does call the production path search inside each training fold.
That search runs the production propensity fits, the targeting step, the loss, and the penalty. The
reconstruction discards the risk each candidate carries, and scores that candidate again on the
held-out rows.

The same test runs the production fold splitter on the surface's root seed. It compares the result
against the fold assignment the refit stored. Component mutations strip the weights from the loss,
the influence curve, the targeting step, the fold nuisance fit, the intercept mechanism, or the
candidate mechanism. Each mutation moves the stored nested-risk array.

Three tests ask whether a search decision reads the observation weights. The candidate risk is the
loss plus the penalty. A greedy search on a binomial fit and a logistic preorder on a Gaussian fit
each lose the weights from the loss. Each one moves the risk it is scored on. The
partial-correlation preorder never reaches the loss, so a third test removes the weights from that
ranking and asserts the order it produces. An explicit ordering carries no witness, because the
reader declares that order.

A fourth test pins the selected adjustment set, and not only the order the search visited. A moved
order that leaves the selection alone leaves the fitted mechanism identical. The fixture carries
two candidates that serve opposite weight blocks, in a mass ratio of nine to one. The weighted
search selects one candidate and the unweighted search selects the other, while the visited order
stays equal.

End-to-end mutations drop every selector weight or drop the shared outcome-adaptive
mechanism weights. Each mutation moves a manual refit that runs on the surface's seeds at
strengths 0.2 and 0.3. Neither control reads a surface cell, because its grid holds the anchor
alone and the anchor runs no refit. Each control requires a move above one part in a thousand of
the unmutated refit's estimate.

The per-arm outcome-adaptive design has its own weight controls in
`tests/unit/test_outcome_adaptive_per_arm.py`. Integer weights must equal duplicated rows, and a
cross-fitted longhand with fixed weights must agree to `1e-8`. Dropping the weights from the
per-arm mechanism, in sample or in each fold, fails these checks.

The tests rebuild the vector-target penalty and the ratio penalty once each. The `ey1`, `ey0`, and
`or` estimands reuse the same arm-curve and delta-method arithmetic, and they carry no separate
dropped-weight mutation.

The pinned R `drtmle` 1.1.2 implementation accepts no observation-weight argument. It supplies no
weighted numerical comparator. This surface therefore claims no weighted R parity and broadens no
interval claim.

The canonical R `ctmle` 0.1.2 source at commit `18de559` also accepts no observation-weight
argument. The `ctmleDiscrete` entry point, which the pinned selector-parity study calls, takes
none, and neither do its `stage2` and `cv` helpers. The `ctmleGeneral`, `stage2_general`, and
`cv_general` paths use unweighted operations. The archived `ctmle3` source supplies
outcome-adaptive control flow but no weighted comparison. The fixed-weight C-TMLE surface
therefore claims parity with neither implementation.

Conditional on the observed rows and fixed weights, the simulation draws each latent value
independently from a standard normal law without using either. After that draw, the realized
weighted empirical distribution is discrete. Its latent marginal need not have weighted mean zero
or variance one, and chance association with the original treatment need not vanish. The anchor's
reported association records that finite-sample imbalance. Each cell reports the same parameter
functional on its perturbed weighted empirical law. This statement does not claim that the
operation reproduces the sampling or selection mechanism.

Hartman and Huang (2024) treat sensitivity to a confounder that the weighting model omits. They
report a bound, a robustness value, and a benchmark. This surface reports none of those, so their
method does not govern it.

The induced treatment association uses weighted means, variances, and covariance. Numeric
calibration also uses weighted scaling, model fitting, prediction-change fractions, and moments.
Constant weights use the exact unweighted calculation path. A common weight scale leaves the cell
estimates, the displacements, the induced associations, and the calibration strengths numerically
unchanged. The agreement is numerical rather than bitwise, because `check_weights` normalizes each
weight vector to mean one.
`tests/unit/test_simulated_confounding.py::test_fixed_weight_surface_is_invariant_to_a_common_weight_scale`
pins ordinary TMLE to 1e-12.
`tests/unit/test_simulated_confounding.py::test_fixed_weight_drtmle_surface_is_invariant_to_a_common_weight_scale`
applies the same tolerance to the DR-TMLE cells. The supplied scale is itself a descriptive
measurement, so
`weight_report.scale` does change with it.
`tests/unit/test_simulated_confounding.py::test_fixed_weight_ctmle_surface_is_invariant_to_a_common_weight_scale`
applies the same tolerance to selector-based and outcome-adaptive C-TMLE cells.

**Refusals.** `_FIT_WIDE_RULES`, in
`src/cleverly/sensitivity/_simulated_confounding_request.py`, is one ordered table. It names every
boundary that neither the requested estimand nor the strength grid can move. `_fit_wide_refusal`
walks that table and returns the first reason that applies. Eight missing-science stops apply
in this order: longitudinal, learned-rule, multi-arm, missing-outcome, missing-treatment,
intermediate, estimated-weight, and clustered fits. Each one has a row in the refusal table below,
and each one carries a roadmap item. F12 tracks the missing-treatment stop with the
missing-outcome stop.

Twelve provenance and shape stops read the object you hold rather than the state of the science.
The `result_type` stop runs second, before the learned-rule stop, because every later rule reads a
field that only a `TMLEResult` declares. The other eleven follow the clustered stop. The table
below gives all twelve in contract order.

| rule | refuses |
| --- | --- |
| `result_type` | an artifact that is not exactly a point-treatment `TMLEResult` |
| `missing_estimator` | a result that stores no replay estimator |
| `repeat_provenance` | a stored draw count, a `config.crossfit.repeats`, and a replay estimator `repeats` that disagree |
| `binary_estimator` | a binary fit made by an estimator other than `TMLE`, `CTMLE`, or `DRTMLE` |
| `continuous_estimator` | a continuous fit made by anything but exact ordinary `TMLE` |
| `outcome_family` | an outcome family other than `gaussian` or `binomial` |
| `weight_kind` | a weight kind other than `probability` |
| `weight_provenance` | a `weights_name` and a `WeightSpec` name that disagree |
| `undeclared_weights` | nonconstant observation weights with no declared weight column |
| `identification` | a fit that records no identification metadata |
| `functional` | an identified functional other than a backdoor mean contrast |
| `provider` | backdoor provenance from anything but a registered explicit adjustment set |

`_validate_request` applies the first reason before calibration, a random draw, or a refit. The
assessment capability walks the same table, so `available`, `status`, `reason`, and direct
execution name the same stop for all twenty. A fit that reaches the `provider` stop reports
`reason="simulated_confounding needs registered explicit-adjustment backdoor provenance"` on its
capability row.

The rows below record the eight missing-science stops, plus the estimands and compositions the
surface refuses. The `code` column holds the reason code of the raise site. The codes are those of
[how to read a refusal](scope-and-refusals.md#how-to-read-a-refusal). A row that cites an `F` item
of the roadmap also has a part that waits on published theory, as the
[roadmap's eligibility rules](../roadmap.md#eligibility) define the term.

| refused | code | why |
| --- | --- | --- |
| a longitudinal result | `implementation_gap` | no time-indexed latent law covers censoring, survival outcomes, a time-varying covariate after the first treatment node, or a time-varying latent. See [F13](../roadmap.md#f13-longitudinal-simulated-confounding-replay). One time-fixed latent on an uncensored terminal-outcome fit is not written yet. See [X48](../roadmap.md#x48-longitudinal-simulated-confounding-replay-with-a-time-fixed-latent) |
| a learned-rule fit | `implementation_gap` | a replay relearns the rules, so each draw has a different target. A replay that freezes the fold rules needs a new fold-evaluated path. See [X11](../roadmap.md#x11-learned-policy-follow-ups) part (o) |
| multi-arm treatment | `implementation_gap` | a swap of the two arms of a declared pairwise contrast is the declared law. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (a) |
| a missing outcome | `implementation_gap` | a response mechanism that reads the covariates only keeps missing at random given the perturbed treatment. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (e). A mechanism that reads the treatment breaks it. See [F12](../roadmap.md#f12-missing-outcome-simulated-confounding-replay) |
| a declared missing treatment | `implementation_gap` | as the missing-outcome row. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (e) and [F12](../roadmap.md#f12-missing-outcome-simulated-confounding-replay) |
| a controlled direct effect, or any fit that carries an intermediate variable | `implementation_gap` | the outcome axis and an intermediate axis that holds the treatment are declared laws. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (f). A flip of the treatment makes the latent confound the intermediate and the outcome by an unnamed path. See [F15](../roadmap.md#f15-controlled-direct-effect-simulated-confounding-replay) |
| estimated observation weights | `implementation_gap` | a replay that holds the supplied weights fixed targets the fit's weighted population. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (d) |
| a clustered fit | `implementation_gap` | the caller declares a row-level, cluster-level, or mixed latent cause. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (b) |
| identification other than a backdoor mean contrast with explicit adjustment | `invalid_declaration` | the surface reads registered explicit-adjustment provenance |
| ATT or ATC under C-TMLE or DR-TMLE | `implementation_gap` | `CTMLE` and `DRTMLE` refuse these functionals when they estimate, so no such fitted result exists |
| a custom MSM link | `implementation_gap` | only built-in identity, log, and logit links have a replay audit. See [X66](../roadmap.md#x66-simulated-confounding-beyond-x47-and-x48) part (a) |
| a custom intervention type | `implementation_gap` | only exact `Static`, `Rule`, `Stochastic`, and `Incremental` declarations have an audited baseline-input contract. See [X66](../roadmap.md#x66-simulated-confounding-beyond-x47-and-x48) part (b) |
| a regime, incremental target, or MSM under C-TMLE or DR-TMLE | `implementation_gap` | the method catalog lacks the corresponding collaborative score or reduced-dimension correction. See [X50](../roadmap.md#x50-further-dr-tmle-compositions) and [X51](../roadmap.md#x51-c-tmle-point-estimates-for-further-targets) for regimes and MSMs, and [F5](../roadmap.md#f5-other-refused-c-tmle-and-dr-tmle-compositions) for incremental targets |
| continuous-treatment C-TMLE and DR-TMLE | `implementation_gap` | both estimators refuse a modified-policy functional. See [F5](../roadmap.md#f5-other-refused-c-tmle-and-dr-tmle-compositions) |
| PAR or PAF under C-TMLE or DR-TMLE | `implementation_gap` | the method catalog lacks a score or correction for these observed-law contrasts. See [X66](../roadmap.md#x66-simulated-confounding-beyond-x47-and-x48) part (h), which depends on [X50](../roadmap.md#x50-further-dr-tmle-compositions), [X51](../roadmap.md#x51-c-tmle-point-estimates-for-further-targets) and [X62](../roadmap.md#x62-other-dr-tmle-and-c-tmle-compositions) |
| `NaturalCourseMean`, reported as `ey_obs` | `invalid_declaration` | $E[Y]$ has no counterfactual treatment term. Outcome perturbation alone cannot make it a confounding diagnostic |
| the policy mean of a zero-delta shift | `invalid_declaration` | $d_0(a, w) = a$ on both branches, so the policy is the natural course. Its mean is $E[Y]$, and no counterfactual treatment dependence remains for a common cause to move. The `ate_policy[...]` contrast that uses this policy as its reference is still accepted |
| an incremental mean at multiplier one | `invalid_declaration` | its mean is $E[Y]$. Contrasts with this reference remain supported |
| a categorical benchmark covariate | `implementation_gap` | a block calibration over every encoded column of the covariate, fitted unpenalized. See [X47](../roadmap.md#x47-simulated-confounding-laws-for-point-treatment-fits) part (c) |

Some rows above record an upstream limit rather than a surface limit. The identified effect's
method catalog refuses unsupported variant targets at `estimate()`, before it builds an estimator.
`DRTMLE` itself refuses a baseline stratum at a non-empty `guard` before any learner. No fitted
result reaches the surface guard in these cases, and `_replay_refusal` keeps that guard as defence
in depth.

A `DRTMLE` fit at `guard=()` is the exception. It fits strata, so a request for one of its
stratum-specific aliases reaches `_replay_refusal`. The stratum guard is the only refusal of that
request. The surface covers marginal DR-TMLE targets only, and the message tells the caller to fit
the ordinary TMLE, which `guard=()` is.

| composition | layer that refuses | when | message |
| --- | --- | --- | --- |
| ATT or ATC under C-TMLE or DR-TMLE | the identified effect's method catalog | `estimate()` | `method 'collaborative_tmle' cannot estimate ATT: no collaborative score is evidenced for this functional`. `DRTMLE` names its reduced-dimension correction instead |
| a modified-treatment policy under C-TMLE or DR-TMLE | the identified effect's method catalog | `estimate()` | `method 'drtmle' cannot estimate ModifiedTreatmentPolicy: no reduced-dimension correction is evidenced for this functional` |
| PAR or PAF under C-TMLE or DR-TMLE | the identified effect's method catalog | `estimate()` | `method 'collaborative_tmle' cannot estimate PopulationAttributableRisk: no collaborative score is evidenced for this functional`. DR-TMLE names its reduced-dimension correction; PAF names its own type |
| regime or MSM under C-TMLE | the identified effect's method catalog | `estimate()` | `method 'collaborative_tmle' cannot estimate RegimeMean: no collaborative score is evidenced for this functional`. Other targets name their own type |
| regime or MSM under DR-TMLE | the identified effect's method catalog | `estimate()` | `method 'drtmle' cannot estimate RegimeMean: no reduced-dimension correction is evidenced for this functional`. Other targets name their own type |

The surface also refuses a constant benchmark covariate. That check reads the requested covariate
rather than the fitted result, so it sits outside the fit-wide table.

`simulated_confounding_refusal` returns the refusal of one request before any draw. It reads the
fit-wide table and then the parameter and calibration checks that the call runs. The capability
row reads it for each request, with the request's `benchmark_covariates` held fixed.

| request | row |
| --- | --- |
| a refused parameter, such as `ey_obs` or a zero-delta policy mean | `unavailable`, with the call's sentence |
| a categorical or a constant benchmark covariate | `unavailable`, with the call's sentence |
| no `estimand` on a natural-course fit | `unavailable`, because no reported parameter runs |
| a malformed argument | unchanged. The call raises `ValueError` or `TypeError` |

A malformed argument is a duplicate, non-string, or unknown covariate name, an unknown estimand, or
a grid strength out of range. A request on a continuous fit that names no policy parameter is
malformed too. The call reports a malformed name before any parameter or calibration refusal.
`TestTheSimulatedConfoundingRowResolvesEachRequest` in
`tests/unit/test_capability_row_predicates.py` checks four study fits.

`test_fit_wide_refusals_have_one_order_and_match_assessment` checks the shared order and exact
capability reasons. `test_real_mar_fit_refuses_before_calibration_draw_or_refit` uses a fitted
ordinary MAR estimator. `test_randomized_missing_outcome_fit_refuses_simulated_confounding_before_work`
uses the separate randomized DR-TMLE construction. Both tests fail if refusal reaches calibration,
the latent draw, or a refit.

`tests/unit/test_simulated_confounding.py::test_weight_refusals_and_provenance_tampering_precede_draws_and_refits`
pins the three weight stops before the first draw and refit. It also tampers with stored
provenance, so a passing run witnesses each refusal on a real fitted result.

Numeric calibration follows the maintained DoWhy source as secondary implementation provenance.
For a binary variable, it reports the class-prediction change after one standardized column is set
to zero. For a Gaussian outcome or continuous dose, it reports `corr(W_j, V) * sd(V)`.
All terms use $P_w$ when the fit declares fixed probability weights.

The Gaussian calibration is signed. It carries the covariate's own direction of association with
the outcome or with the dose. Each axis converts a calibrated value into a declared strength by its
own rule, because the two perturbation laws carry opposite signs.

| axis | law | conversion |
| --- | --- | --- |
| outcome | $Y'=Y-k_YU$ | the law subtracts, so an outcome strength of $k_Y$ calibrates at $-k_Y$. To match a covariate that calibrates at $+c$, declare an outcome strength of $-c$ |
| treatment, continuous dose | $A'=A+k_AU$ | the law adds, so a treatment strength of $k_A$ calibrates at $+k_A$. To match a covariate that calibrates at $+c$, declare a treatment strength of $+c$ |

A binary treatment or a binomial outcome has no such conversion. Its calibration is the
class-prediction change fraction, which carries no sign.

Calibration does not select or modify the grid. It is not partial R-squared and does not reuse the
omitted-variable `benchmark()` scale.

The maintained source is pinned at revision `2116d5c`. The implementation does not copy its
cumulative cell mutations, schedule-dependent random draws, non-exact zero refit, automatic
ranges, categorical encoded-column deletion, or unstructured failure behavior.

### E-value

**How.** [`sensitivity/evalue.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/sensitivity/evalue.py)
implements VanderWeele and Ding (2017): $E = RR + \sqrt{RR(RR-1)}$. It computes the E-value for the
point estimate and, separately, for the confidence limit, because the second is the one an
adversarial reader asks for.

The E-value is the threshold when the two confounder associations have equal strength.
Unequal strengths can trade off under the bounding factor of
[Ding and VanderWeele (2016)](https://arxiv.org/html/1507.03984).
For $RR=2$, the E-value is $3.414$. Associations of $3$ and $4$ give a bounding factor of
$3\times4/(3+4-1)=2$, so both associations need not reach $3.414$.
The E-value does not decide whether a confounder of either strength is plausible.

The selected path depends on the reported contrast and retained artifacts.

| request | E-value path |
| --- | --- |
| reported risk ratio | use it directly and mark the result exact |
| unambiguous default binary marginal ATE or odds ratio from ordinary TMLE | retarget cached nuisances to the matching risk ratio and mark the result exact; combined runs include this cheap retarget by default |
| explicit reported odds ratio, or default odds ratio without exact retarget support | use the common-outcome approximation $\sqrt{OR}$ and mark the result approximate |
| binary ATE without exact retarget support, with a usable reported reference-arm mean | hold the baseline risk fixed and mark the result approximate; includes DR-TMLE and CV evaluation |
| Gaussian ATE, ATT, or ATC with every outcome observed | standardize by the observed outcome standard deviation, weighted on a weighted fit, and mark the result approximate |
| Gaussian ATE, ATT, or ATC on a fit with a response mechanism | report `unavailable`. The conversion divides by `sd(Y)` from the observed rows alone, and under missing at random the respondents' standard deviation estimates a different quantity from the population standard deviation. The refusal is on this path only, so a ratio E-value on the same fit stays available |
| binomial ATT or ATC | refuse because the conditional baseline risk and conditional ratio target are absent |
| level or non-arm parameter | report `not_applicable` because no supported two-arm contrast exists |
| two-arm contrast on a fit whose status supplies no inference: a collaborative fit at any `strategy`, a `DRTMLE` fit with a `guard` and varying weights declared estimated, or a clustered fit under `"few_cluster_plugin"` | report `unavailable`. Each branch reads the estimate's interval or the reference arm's standard error, and this fit supplies neither. The reason of the status follows |
| any request on a learned-rule fit | report `unavailable` before the estimand is resolved. No E-value of a regime contrast ships, and a learned-rule contrast is not a contrast of two exposure levels. See [F27](../roadmap.md#f27-learned-policy-value-outside-the-published-conditions) |
| any request on a controlled-direct-effect fit with a discrete treatment, which is such a fit with `intermediate=` | report `unavailable` before the estimand is resolved. This package has no implemented E-value bound for the fitted controlled direct effect and its confounding model. [X49](../roadmap.md#x49-e-value-for-a-controlled-direct-effect) holds the per-arm bound |
| binomial ATE without exact retarget support or a usable reported baseline | report `unavailable` and name the missing evidence, artifact, or target |
| several eligible contrasts and no explicit estimand | report `deferred` and name `estimand` in the next step |

`_select_evalue` in `cleverly.sensitivity.evalue` runs three fit-wide checks before it resolves
the estimand. A longitudinal fit and a controlled-direct-effect fit report `unavailable`. A fit
with a continuous treatment reports `not_applicable`. The checks run in this order. So a
continuous-treatment fit with an intermediate variable reports `not_applicable`. A request for
`ey1` on a controlled-direct-effect fit reports `unavailable`, and a multi-arm
controlled-direct-effect fit does not defer.

The status refusal follows the `not_applicable` check of the request. A request for `ey1` on a
collaborative fit therefore still reports `not_applicable`. Several eligible contrasts require an
explicit alias, which is the `deferred` row above.
[The status contract](#the-status-contract) states that rule for every operation that shares it.
Combined runs select availability and cost from the alias before they apply the cost flags.

Raw results compose arm identities forward from fitted treatment metadata.
Explicit structured keys remain authoritative. No routing step parses display aliases.
The fixed-baseline approximation ignores baseline sampling error and requires the contrast's matching reference arm.
Its availability does not depend on persistence or estimator retention.

A usable reference-arm mean is finite, positive, and separated from zero by its own standard error.
The conversion divides by that mean, so a mean at zero leaves the ratio without a stable denominator.
The library also refuses a risk difference at or below the negative of that mean.
Such a difference implies a nonpositive risk in the contrast arm, and no risk ratio describes one.
Both refusals report `unavailable` and name the two reported numbers.

The conversion is affine, so the lower interval bound can leave the parameter space while the point ratio stays inside it.
Only the lower bound can leave it. `wald_ci` gives `high >= psi`, and the refusals above force `baseline.psi > 0` and `baseline.psi + psi > 0`.
The report truncates the lower bound at 0, records the untruncated value in `truncated_bound`, and repeats it in the note.
The `to_dict` mapping and the battery row both carry that value, so no surface presents the 0 as a converted confidence limit.

What the truncation means for the confidence limit depends on the side of the null the point ratio is on.

| point ratio | bound the E-value reads | effect of the truncation |
| --- | --- | --- |
| at or above the null | the truncated lower bound | the interval covers the null, so the confidence-limit E-value is 1 |
| below the null | the untruncated upper bound | none. The interval can still exclude the null, and the confidence-limit E-value can exceed 1 |

For rare outcomes, the odds ratio itself approximates the risk ratio. The square-root
transformation addresses common outcomes and can lie above or below the risk ratio.
See [VanderWeele's analysis](https://pmc.ncbi.nlm.nih.gov/articles/PMC5617805/).

The Gaussian path first applies Chinn's odds-ratio to standardized-mean-difference relation.
It then applies the common-outcome square-root approximation before calculating the E-value.
The resulting conversion is $RR \approx \exp((1.81 / 2)d) = \exp(0.905d)$.
This retains an approximate analysis, not an exact continuous-outcome risk ratio.

This package has no implemented E-value bound for its fitted controlled direct effect and
confounding model. That fit invokes assumptions about treatment-outcome and
intermediate-outcome confounding (`estimators/direct_effect.py`, assumptions 2 and 3).
The ordinary conversion does not by itself quantify sensitivity for that target.
[X49](../roadmap.md#x49-e-value-for-a-controlled-direct-effect) holds the specialization: the bias
splits into one bounding factor per arm, and the E-value inverts their product. `tests/unit/test_evalue_direct_effect_refusals.py` pins the refusal.

| source, and the version read | what it covers | what it does not cover |
| --- | --- | --- |
| VanderWeele (2010), [PMC4231822](https://pmc.ncbi.nlm.nih.gov/articles/PMC4231822/) | Theorem 1 and Appendix 1 give controlled-direct-effect bias formulas when the latent confounds only the intermediate and the outcome | a latent that also confounds the treatment. The [hard-stop review](hard-stop-review.md) records it as context for X49 |
| Smith and VanderWeele (2019), [author PDF](https://www.louisahsmith.com/publications/smith2019mediational.pdf), pages 835–837, and [supplement](https://download.lww.com/wolterskluwer_vitalstream_com/PermaLink/EDE/B/EDE_2019_07_22_SMITH_EDE18-0696R2_SDC1.pdf), Sections 1–2 | an approximate mediational E-value for natural direct and indirect effects under its stated confounding model | no mapping in this source to the fitted population controlled direct effect at fixed $z$ |
| Ding and VanderWeele (2016), *Biometrika*, [arXiv 1601.05155](https://arxiv.org/html/1601.05155), Sections 2–4 and Appendix D.4 | bounds for natural direct and indirect effects; Appendix D.4 drops treatment-confounder independence and changes the target to the unexposed population | no stated mapping to this package's population controlled direct effect at fixed $z$ |
| Ding and VanderWeele (2016), *Epidemiology*, [arXiv 1507.03984](https://arxiv.org/html/1507.03984), Sections 2–3 and Appendix 2.5 | a bounding factor for an exposure contrast with general, possibly multivariate confounders and averaging over measured covariates | a composite exposure such as $(A,Z)$ needs an explicit confounding and standardization mapping before the factor describes the fitted target |
| Gilbert, Fong, Kenny and Carone (2023), [*Biostatistics*](https://academic.oup.com/biostatistics/article/24/4/850/7320953), Sections 2.1 and 4.1 | a controlled direct effect and an E-value for a marker contrast within a fixed vaccination arm | its E-value compares marker interventions within one vaccination arm, not treatment arms at fixed marker value |
| Mathur, Ding, Riddell and VanderWeele (2018), [PMC6066405](https://pmc.ncbi.nlm.nih.gov/articles/PMC6066405/) | software for ordinary E-values and effect-scale conversions | no mapping of that software's ordinary conversion to this fitted controlled direct effect |
| Tingley, Yamamoto, Hirose, Keele and Imai (2014), [*Journal of Statistical Software*](https://www.jstatsoft.org/article/view/v059i05) | the `mediation` package and its mediation sensitivity analysis | no E-value bound for this fitted controlled direct effect is supplied by its software account |
| the R package `EValue` 4.1.4, [CRAN manual](https://cran.r-project.org/web/packages/EValue/EValue.pdf) | software for confounding, selection bias, and measurement error | its lack of a mediation API is a software limit, not a mathematical impossibility |

The search did not read these sources.

| source | title |
| --- | --- |
| VanderWeele and Ding (2017), appendix | the online appendix of *Sensitivity analysis in observational research: introducing the E-value* |
| VanderWeele, Ding and Mathur (2019) | [*Technical considerations in the use of the E-value*](https://biostats.bepress.com/harvardbiostat/paper215/), *Journal of Causal Inference*. The abstract was available; the full text was not |
| VanderWeele (2015) | *Explanation in Causal Inference: Methods for Mediation and Interaction*, a book |

The Smith and VanderWeele supplement was read in the present review. The 2017 appendix and
the full text of the 2019 technical discussion remain unread. A search that did not locate a
target-specific theorem does not establish that none exists.

A weighted fit standardizes by the weighted outcome standard deviation.
The estimate targets the population the observation weights describe, and the standardizing scale describes the same population.
The weighted form divides by the reliability-weight correction $\sum w - \sum w^2 / \sum w$.
That correction is $n - 1$ when every weight is one, so an unweighted fit reports the plain sample standard deviation.

### Missingness tilt and tipping gamma

**How.** [`sensitivity/missingness.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/sensitivity/missingness.py)
implements the Scharfstein, Rotnitzky and Robins (1999) tilt. It sets

$$
\bar{Q}^{\text{miss}}_\gamma = \operatorname{expit}\{\operatorname{logit} \bar{Q}^* + \gamma\}
$$

for the unobserved outcomes, and mixes it with $\bar{Q}^*$ by the estimated missingness
probability. At $\gamma = 0$ it reproduces the missing-at-random estimate by construction, which is
the control that says the tilt is wired in. `arm_gamma=` gives per-arm tilt directions and must
name every arm. `tipping_gamma()` inverts the tilt for the value at which the conclusion changes.

The tilt applies $\gamma$ on the logit of the outcome scale that the fit uses.
A binary outcome uses that scale directly.
A continuous outcome maps onto $[0, 1]$ through the declared `q_bounds`.
Without `q_bounds`, `OutcomeScaler.from_outcome` takes the observed range and widens it by 10% on each side.
Declare `q_bounds` to fix what one unit of $\gamma$ means for a continuous outcome.

Away from $\gamma = 0$, the curve is a plug-in value.
`_tilted_psi` reuses the targeted $\bar{Q}^*$ of the $\gamma = 0$ fit and does not target again at each $\gamma$.
The accuracy of the curve away from $\gamma = 0$ therefore depends on the outcome model.

This is a retarget operation and not a refit.

`fit_wide_tilt_refusal` holds every refusal that no argument lifts. `missingness_tilt()` and
`tipping_gamma()` raise its sentence first, and both capability rows quote it. The table is
ordered, and each rule assumes that the rules above it passed.

| order | rule | refuses | row status |
| --- | --- | --- | --- |
| 1 | `longitudinal` | a longitudinal result | `unavailable` |
| 2 | `learned_rule` | a learned-rule fit, which refuses missing outcomes before any learner. [F27](../roadmap.md#f27-learned-policy-value-outside-the-published-conditions) holds that refusal | `unavailable` |
| 3 | `natural_course` | a missing-outcome `NaturalCourseMean` fit that reports `ey_obs` alone | `unavailable` |
| 4 | `missing_outcome` | a fit with no missing outcome | `not_applicable` |
| 5 | `continuous` | a continuous dose | `unavailable` |
| 6 | `incremental` | an incremental fit | `unavailable` |
| 7 | `tiltable_parameters` | a fit that reports no arm-indexed mean and no linear contrast, such as a regime, MSM, ratio-only, or PAR-and-PAF-only fit. On the last kind the sentence names the natural-course targets | `unavailable` |

On a composite fit with a missing treatment and a missing outcome, the tilt re-mixes the targeted
regression with the response mechanism $\pi(a, W) = P(\Delta = 1 \mid A = a, \Delta_A = 1, W)$.
Both are defined there, so the formula does not change. The tilt holds the treatment-missingness
condition fixed: it moves the outcome's recording away from missing at random and leaves the
treatment's recording at it.

The tilt is arm-specific. A fit that reports `ey_obs`, PAR or PAF beside arm means passes the
table, and three rules keep the natural-course targets out of the tilt:

| request | result |
| --- | --- |
| `missingness_tilt(estimands=...)` that names `ey_obs`, `par` or `paf` | `CapabilityError` that names those targets and the arm-mean remedy |
| the default sweep | the arm means and contrasts only. The capability row's interpretation names the skipped targets |
| a bare `tipping_gamma()` whose default would fall to the one reported arm mean | the row reads `deferred` and requires `estimand`. The call raises the row's reason, which names the natural-course targets and the arm mean to choose, rather than a silent switch to the arm mean |

A fit that reports PAR or PAF and no arm mean meets rule 7. `tests/unit/test_attributable_mar_capabilities.py`
checks each row.

The omitted-variable bound refuses a fit with a response mechanism. Its sentence points at the tilt
only when this table admits the fit. On a fit that reports `ey_obs`, PAR or PAF, the pointer names
the arm means, and a bound request for PAR or PAF carries no pointer. `TestTheTiltRowsReadTheCallsPredicate` and
`TestTheBoundPointsAtTheTiltOnlyWhereItRuns` in `tests/unit/test_capability_row_predicates.py`
check the rows and the pointer.

The curve is a point-estimate sweep, so a selector-based C-TMLE fit receives all of it. Only the
three spread columns read an interval that those paths refuse. Such a fit therefore receives
`plugin_std_err`, `plugin_interval_lower`, and `plugin_interval_upper` in place of `std_err`,
`ci_lower`, and `ci_upper`, which is the swap the truncation curve above already makes.
`tipping_gamma()` keeps its default point-estimate search on that fit. It refuses `use_ci=True`,
because it returns one float and a float carries no column name.
[Collaborative TMLE](collaborative-tmle.md) states the refusal these columns come from.

### The scope rule

A point-treatment sensitivity formula is not reused on longitudinal data. `LTMLE` reports these
operations `unavailable` with the reason its own capability row declares, rather than borrowing a
derivation. Stagewise support, scores, and nuisance loss are supported longitudinally, because each
has its own derivation.

Tan (2025) derives population sensitivity bounds for binary, static longitudinal strategies, but
no sample estimator for them. [F16](../roadmap.md#f16-longitudinal-sensitivity-bound-estimation)
states the contracts that source leaves open.

## Refutation and simulation you run

These fit new models. They cost what a fit costs, multiplied by the number of draws.

### Refutation operations

**Why.** A diagnostic reads the fit you have. A refuter constructs a case whose answer is known and
checks that the workflow returns it.

**What each tells you.** [`validation/refute.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/refute.py)
ships seven operations. Six test the implementation. The negative-control outcome tests a design,
and the paragraph below states its boundary.

| refuter | what it does | what must happen | what it tests |
| --- | --- | --- | --- |
| `placebo` | permutes the treatment column and refits | an additive contrast approaches zero; a risk or odds ratio approaches one | the pipeline, not the data |
| `random_common_cause` | adds an irrelevant covariate and refits | the estimate does not move | the adjustment set is not sensitive to noise |
| `subset` | refits on random subsamples | the scatter is about one standard error | the reported standard error is the right size |
| `negative_control_outcome` | refits on an outcome the treatment cannot affect | an additive contrast approaches zero; a risk or odds ratio approaches one | the design, under the control assumptions the paragraph below states |
| `dummy_outcome` | draws independent Gaussian noise and refits | the empirical draws include zero | outcome replacement and the full estimator pipeline |
| `simulated_outcome` | draws `f(W) + effect * A + epsilon` and refits | the empirical draws include the declared effect | adjustment and treatment terms in the full pipeline |
| `bootstrap_measurement_error` | bootstraps, perturbs declared adjustment variables, and refits | the empirical draws include the original estimate | stability under the declared measurement error |

A refuter refits the nuisance models once for each replication. The three default operations use
five replications each, so `refute()` costs about 15 fits. Empirical refuters use 100 draws by
default because their rule reads a distribution. `run_all(include_refits=True)` runs `refute()`.

`random_common_cause` adds one column of independent noise, and the refit must accept that column.

| estimator | what the refit runs |
| --- | --- |
| `TMLE` | the estimator |
| `CTMLE` with an explicit `ordering=` | a copy whose ordering is the declared ordering, then each added covariate |
| `DRTMLE` with an `evaluation=` companion that lacks the added column | a copy without the companion |

An explicit ordering ranks covariates by their prior relevance. Independent noise has none, so the
noise goes last. `CausalData.with_extra_covariate` also appends the column last. The companion
enters no fit, fold, or score, so the refit reports the same estimate without it.
`tests/unit/test_drtmle_companion.py` checks that bit for bit. Neither copy changes the fitted
estimator. `TestAnAddedCovariateGoesAfterTheDeclaredOrdering` and
`TestARefitDropsACompanionThatLacksACovariate` in `tests/unit/test_capability_row_predicates.py`
check both cases.

`refute_refusal` in `validation/refute.py` holds every refusal of a request before any refit.
`refute()` first checks its argument values with `_validated_tests`. An unknown test name and an
invalid `n_replicates` raise `ValueError`. `result.diagnostics.refute()` runs the same check before
it reads its capability row. The call then raises the first refusal of this table as
`CapabilityError`.

| order | rule | refuses |
| --- | --- | --- |
| 1 | `estimator` | a result with no estimator |
| 2 | `learned_rule` | a learned-rule fit, whose refit relearns the rules and so changes the target. The `refute` row reads `unavailable` for the whole fit |
| 3 | `estimand` | an estimand that the fit did not report |
| 4 | `no_effect_null` | `placebo` or `negative_control_outcome` on an outcome or policy mean, an MSM coefficient, or a target with no registered null |
| 5 | `row_set_under_plan` | `subset` or `bootstrap_measurement_error` on a fit given `split_plan=` |
| 6 | `generated_rule` | a generated-outcome test under a rule that is not exactly `EmpiricalInclusionRule` |
| 7 | `generated_eligibility` | a generated-outcome test whose fit, estimand, or process has no derivation, such as a direct fit that records no identification |
| 8 | `measurement_declaration` | `bootstrap_measurement_error` without exactly a `BootstrapMeasurementError` declaration |
| 9 | `measurement_rule` | `bootstrap_measurement_error` under a rule that is not exactly `EmpiricalInclusionRule` |
| 10 | `measurement_eligibility` | a measurement-error declaration that the data of the fit cannot carry |
| 11 | `measurement_budget` | a `bootstrap_measurement_error` draw count below the floor of its rule |
| 12 | `generated_budget` | a generated-outcome draw count below the floor of its rule |

Rules 7 and 10 call the validators that the tests run, so the table copies no check. The argument
check also requires an outcome array for `negative_control_outcome`. It raises `ValueError` before
any refit when that array is absent.

The `refute` capability row reads the same table for each request. An omitted `tests` means the
default tests. When the default tests meet a refusal and a single default test runs, the row reads
`deferred` on `tests`, with the call's sentence. A request that names only tests that run lifts the
deferral. `TestTheRefuteRowResolvesEachRequest` in the same file checks both fits below.

A refutation also runs on a selector-path collaborative fit. The `placebo`, `random_common_cause`,
`subset`, and `negative_control_outcome` rules scale their tolerance by a standard error. Each rule
reads that scale through `plugin_std_error`, so the tolerance makes no coverage claim. On a fit
that supplies no inference, each `RefutationTest` carries `inference`. `RefutationTest.to_frame()`
and `RefutationResult.draws_frame()` then name the standard-error column `plugin_std_error` in
place of `std_error`.

A fit that declared `split_plan=` refuses two of these operations. A supplied plan labels the rows
it was realized on, by position. It cannot label a refit that drops rows or draws them with
replacement. So `refute()` raises `CapabilityError` for `subset` and for
`bootstrap_measurement_error` before it refits anything, and it names both the plan and the test.

A `bootstrap_measurement_error` draw holds the declared number of rows, so only the request
identifies it. The `placebo` and `random_common_cause` operations still run, because each one
replaces a column and leaves every row in its position. The default tests include `subset`, so the
`refute` row reads `deferred` on `tests`. A `run_all(include_refits=True)` battery records that
row as `deferred`, with the refusal's sentence, and continues to the rest of the battery. A request
with `tests=("placebo", "random_common_cause")` runs. See
[reusable outer split plans](cv-tmle.md#reusable-outer-split-plans) for the table.

The two no-effect operations read the registered target through the reported parameter key.
Differences and the attributable fraction use zero. Risk ratios and odds ratios use one on their
reported scale. Outcome and policy means have no fixed no-effect value. An arbitrary MSM
coefficient has none either.

Both operations refuse these parameters before any refit. The `subset` and
`random_common_cause` operations keep their stability interpretations. All ratio verdicts use
log ratios because their standard errors use the log scale. Reports keep the refit ratios on their
reported scale.

A ratio placebo or noise test reports the geometric mean that its log-scale verdict tests.
Its `values` remain the individual reported ratios. A ratio subset reports log-scale scatter in
`refuted_sd` and its expectation. Its `values` also remain reported ratios.

The default tests include `placebo`, so `estimand="ey_obs"` defers on `tests`. A request with
`tests=("random_common_cause", "subset")` runs. `TestReportedScaleNoEffectNull` checks the
reported null, refusal, and capability row.

`refute()` draws its randomization from the seed of the fit, unless the caller passes
`random_state`. A fit that carries a seed gives the same refutation on every call. A fit that
carries no seed gives a different refutation on every call.

The report records the seed under `random_state`. Pass that value back to `refute()` to obtain
the report again. The seed governs the perturbations and the refits they feed, so it repeats
the report of a fit that carries no seed of its own. The seed applies to a copy of the
estimator, so a refutation never changes the fit it examines.

Each empirical draw derives a child seed from the recorded root seed. The perturbation and its
full refit use that child seed. `report.draws_frame(name)` reports every child seed, estimate,
standard error, family, and failure. The report stores each draw as an
`EmpiricalRefitRecord`.

`EmpiricalInclusionRule` uses a two-sided empirical rank and inclusive half-ties. It passes only
when the empirical probability strictly exceeds alpha, matching the maintained DoWhy convention
that a probability at or below alpha fails. The default rule uses alpha 0.05, requires 40
successful draws, and fails when any refit fails. A failure stays in the report as the shared
`ReplicationFailure` record, so the operation never reports only the conditional distribution of
successful fits.

The rule count and the draw budget are separate defaults. `DEFAULT_OUTCOME_REPLICATES` is 100 and
sets `n_replicates`. `EmpiricalInclusionRule()` requires 40 successful draws and sets no budget.
The operation refuses a budget below the rule's minimum before it refits anything.

Bootstrap measurement error uses the same rule against the original estimate. It draws an iid or
whole-cluster sample through the inference bootstrap design. It perturbs numeric variables with
mean-zero Gaussian noise after sampling. The noise scale is the declared multiplier times the
selected variable's bootstrap-sample standard deviation.

Each sampled cluster occurrence receives a distinct code before perturbation. Repeated draws of
one source cluster therefore remain separate during the refit.

The categorical path recovers original logical levels from `CausalData.encodings`. Each changed
row draws uniformly from the other levels. The operation then rebuilds the complete drop-first
indicator block. Boolean covariates use the same two-level path.

The operation checks every condition in the table below before the first refit. Each row is one
`CapabilityError` branch of `_validate_measurement_error_eligibility` in
[`validation/refute.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/refute.py),
and each message names the variable it rejects.

| what the operation refuses | what it requires instead |
| --- | --- |
| a result whose data is not point-treatment `CausalData` | a point-treatment `CausalData` result |
| `resampling="cluster"` on data that carries no cluster ids | declared cluster ids, or iid resampling |
| a selected strata variable | a variable outside the strata, because the operation cannot perturb target metadata coherently |
| a generated indicator name | the original categorical variable that the indicator encodes |
| a name that is not an original adjustment variable | one of the original names the message lists |
| a categorical variable whose encoded block lost an indicator to duplicate-column removal | a retained indicator for every level the encoding generated |
| a selected variable whose values are all 0 or 1 and that carries no `CategoricalEncoding` | the same variable declared to `CausalData.from_frame` as a boolean or categorical column, because relative Gaussian noise makes an undeclared indicator real-valued |
| a selected numeric variable that is constant | a variable with nonzero spread, because a relative noise scale of zero leaves every draw unperturbed |

The report records the declaration, requested draw count, resolved mode, rule, estimates,
standard errors, child seeds, and failures.

**Alpha is a width here, and not a false-alarm rate.** The rule decides one question. Does the
declared effect lie inside the central `1 - alpha` of the refit estimates? At the default alpha
that is the central 95%.

A correct estimator centres the refit distribution on the declared effect, so each draw falls
above it with probability about one half. Under 100 draws the rule then fails only when at most
two draws fall on one side. That probability is `2 * (1 + 100 + 4950) / 2**100`, which is about
`8e-27`. More draws do not buy power. They stabilise the quantile the rule reads.

The default minimum of 40 draws is the smallest budget at alpha 0.05 that can fail on anything
short of a one-sided sweep. With 40 draws, one estimate on the minority side gives `2 / 40`, which
equals alpha and fails. With 39 draws, one estimate on the minority side gives `2 / 39`, which
exceeds alpha and passes. The rule therefore refuses a declaration whose `minimum_draws * alpha`
is below 2, because such a rule fails only when every draw falls on one side.

The first process catalog covers Gaussian outcomes and additive `ate`, `att`, or `atc` contrasts.
`refute()` and its `_validate_generated_eligibility` helper in
[`validation/refute.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/refute.py)
check every condition in the table before the operation refits anything. Each row is one
`CapabilityError` branch, and each message names what is missing.

| what the operation refuses | what it requires instead |
| --- | --- |
| an inclusion rule of any other type | the exact registered `EmpiricalInclusionRule` declaration |
| a process declaration of any other type | the exact registered `GaussianIndependentOutcome` or `GaussianAdjustmentOutcome` |
| a process whose family is not `"gaussian"` | the Gaussian family, which is the one with an implemented effect derivation |
| a fit that carries no identification metadata | identification metadata on the result |
| a functional that is not `BackdoorMeanContrast` | a backdoor-identified additive mean contrast |
| a provider that is not `ExplicitAdjustmentProvider` | registered backdoor provider provenance |
| an original outcome family that is not `"gaussian"` | an original family equal to the process family, so a binomial fit is refused |
| an estimator configured with a family other than `"auto"` or `"gaussian"` | a configured family that accepts the declared process |
| a saved outcome learner with no regression-capable route | a regression-capable outcome learner |
| a longitudinal functional | a point-treatment functional |
| a treatment that is not binary | code one against code zero |
| an intermediate node or a controlled direct effect | no intermediate node |
| a fit with missing outcomes | complete outcomes |
| an MSM functional or the `msm` axis | the `arm` axis |
| an intervention-indexed functional | the `arm` axis with no declared interventions |
| a parameter key that is not a structured `ParameterKey` | a structured key for the selected estimand |
| a ratio estimand, or any other non-additive contrast | `ate`, `att`, or `atc` |
| disagreement between the functional target, the identified estimand, and the parameter key | one estimand named by all three |
| a missing registered identification artifact | the artifact `TARGETS` records for that target |
| an arm contrast that does not resolve to two distinct arms | a resolvable contrast of two distinct arms |
| a draw budget below the rule's minimum | `n_replicates` of at least `minimum_draws` |

```python
from sklearn.linear_model import LinearRegression, LogisticRegression

from cleverly import ATE, CausalStudy, PointTreatment
from cleverly.datasets import make_linear_ate
from cleverly.validation import GaussianAdjustmentOutcome, GaussianNoise

frame, _ = make_linear_ate(n=200, seed=21)
study = CausalStudy(
    frame,
    design=PointTreatment(outcome="Y", treatment="A", adjustment=("W1", "W2", "W3", "W4")),
)
result = study.estimate(
    ATE(),
    outcome_learner=LinearRegression(),
    treatment_learner=LogisticRegression(max_iter=1000),
    cross_fit=False,
    random_state=21,
)
process = GaussianAdjustmentOutcome(effect=0.5, noise=GaussianNoise())
report = result.diagnostics.refute(
    tests=("simulated_outcome",),
    simulated_outcome=process,
    random_state=21,
)
print(report.summary())
```

Sharma and Kiciman (2020) define the refutation framework. The maintained DoWhy dummy refuter,
pinned at
[`2116d5c`](https://github.com/py-why/dowhy/blob/2116d5cbace5a057937e03b2efba95c13140cc4c/dowhy/causal_refuters/dummy_outcome_refuter.py),
supplies secondary evidence for independent noise and `f(W) + h(A)`. That implementation uses a
normal rule below 100 draws, which `perform_normal_distribution_test` in
`dowhy/causal_refuter.py` applies. It declares no failure policy at all. The pinned file contains
no `try` block, and its refits run under `joblib.Parallel`, so one failed refit aborts the whole
refutation. `cleverly` keeps each failed refit as a `ReplicationFailure` record and fails the
refutation under the rule stated above.

The maintained DoWhy bootstrap refuter at the same revision supplies secondary control-flow
evidence for measurement-error sampling and refitting. `cleverly` does not copy three defects in
that source. It uses logical metadata for numeric and categorical variables. It does not reuse a
Boolean probability array for another categorical variable. It derives a distinct child seed for
each draw.

A negative-control outcome must have no causal path from treatment. It must also share the relevant
confounding structure with the primary outcome. A non-null result flags residual bias or a bad
control, and the refuter cannot tell you which. A null result does not establish that unmeasured
confounding is absent. See [negative controls](../references.md#negative-controls).

### Coverage studies

**Why.** Coverage, bias, and standard-error calibration are claims about repeated sampling. One fit
contains no information about any of them.

**What it tells you.** Three numbers, from
[`validation/simulation.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/validation/simulation.py).

| number | how it is computed | how to read it |
| --- | --- | --- |
| coverage | the share of replications whose interval contains the truth | sustained undercoverage beyond Monte Carlo uncertainty indicates invalid intervals on that law |
| root-n bias | $\sqrt{n}$ times the mean error | bounded values support a negligible first-order bias claim. They do not establish efficiency |
| SE ratio | mean reported standard error over the empirical standard deviation of the estimates | one means the reported uncertainty matches the real spread |

**How.** `CoverageStudy` draws from a generator with a known truth, runs the complete estimator on
each draw, and summarises through `summarize_replications`. A failed draw is retained as a
`ReplicationFailure` record carrying its index, its seed, and its exception. A study that silently
replaced failed draws would report the distribution of the draws that happened to work.

A study of a selector-path collaborative estimator measures the plug-in diagnostic. Each
replication reads `plugin_interval`, `plugin_std_error`, and the same p-value arithmetic that
`pvalue` refuses on that fit. The records carry the inference status, and the table changes as
follows.

| output | ordinary estimator | selector-path collaborative estimator |
| --- | --- | --- |
| `to_frame()` columns | `mean_std_error` | `inference` and `mean_plugin_std_error` |
| `summary()` | the table and the verdict | a line above the table that says the columns describe the plug-in diagnostic |
| `verdict()` with no finding | coverage and bias are consistent with a correctly working estimator | bias is consistent, and the coverage column certifies no confidence interval |

A clustered status depends on the draw, so the replicates of one study can take different
statuses. A cluster count that straddles 4 gives that mix. Every record measures the plug-in
numbers, and on an inferential fit those numbers equal `ci` and `std_error`. So
`summarize_replications` summarizes a mix under the status that `precedent_status` gives. The
table gives what the summary adds.

| output | a study whose replicates took more than one status |
| --- | --- |
| `EstimandSummary.status_counts` | each status with its replicate count. It is empty when every replicate took one status |
| `to_dict()` and `to_frame()` | a `mixed_statuses` column, for example `few_cluster_plugin in 4, influence_curve in 4` |
| `summary()` | the line above the table says that the estimator "supplies no interval on some replicates". One more line per mixed estimand names its statuses and counts |

`tests/unit/test_simulation_summary.py` runs a study whose cluster count straddles 4. It holds
the mixed witness, a mutation that labels the mix `influence_curve`, and a uniform control.

The generators live in
[`datasets/`](https://github.com/esbraun/cleverly-tmle/tree/main/src/cleverly/datasets) and each
one carries an exact `truth`.

**A simulated law is an instrument, and it can be wrong.** A coverage study is evidence only if the
number it calls the truth is the number an adjusted fit estimates. Two shipped clustered generators
once failed that test, and
[evidence.md](evidence.md#a-simulated-law-is-an-instrument-too-and-it-can-be-wrong-the-same-way)
records what went wrong and what each generator now asserts about itself.

### Variable importance

`variable_importance` gives each candidate covariate the treatment role in its own fit, and
reports the target-relevant change with multiplicity-adjusted p-values
([`variable_importance.py`](https://github.com/esbraun/cleverly-tmle/blob/main/src/cleverly/variable_importance.py)).
It is an assessment of the fitted causal workflow. It is not a predictive feature-importance score,
and it introduces no new influence function.

An estimator whose status supplies no inference is refused with `CapabilityError` before the
first fit. That includes a collaborative estimator at any `strategy`, and a `DRTMLE` with a
`guard` and varying weights declared estimated. It also includes a clustered `TMLE` or `DRTMLE`
under `"few_cluster_plugin"`. The procedure adjusts one p-value per candidate with
Benjamini and Hochberg, and those fits report no p-value. The check prepares each candidate's data
and asks the estimator's own `_inference_status()`, which is the status its estimates carry.

## How the library certifies itself

The three layers above review *your fit*. This layer reviews *the implementation*. It is the
evidence that the equation `score_check` solved is the right equation.

The instruments go blind in different places, and the differences are the reason there are six of
them. [evidence.md](evidence.md) records which instrument covers which registered estimand, in both
directions, and it is a test rather than a note.

| instrument | why it exists | what it tells you | how it tells you | what it cannot see |
| --- | --- | --- | --- | --- |
| **exact oracle law** | an estimator has to recover a parameter that was computed rather than estimated | the reported number is the parameter, exactly | a finite-support law whose every cell probability is a multiple of $1/N$, so an $N$-row frame **is** the law. Handed oracle nuisances, the fit is exactly right and $\epsilon$ is zero | nothing about a term that is zero at the truth |
| **Gateaux comparison** | the influence curve is what every interval is built from | the reported curve is the pathwise derivative of the parameter | complex-step differentiation of an independently written functional, compared at about `1e-14` absolute with `rtol=0` | a sign on any block that vanishes at correct nuisances, and any counterfactual block, because $\epsilon$ is zero there |
| **second-order remainder** | double robustness *is* the remainder carrying both nuisance errors | one wrong nuisance still leaves the remainder second order | the von Mises expansion evaluated at nuisances that are wrong on purpose, against a longhand form of the exact remainder | a first-order error that cancels inside the remainder |
| **exact identity** | some mistakes are algebraic and cheap to catch | a relation that holds by definition holds bit for bit | relabelling the arms, a null outcome model giving zero, weights scaling out, the one-step and iterative solvers agreeing | anything symmetric in whatever the identity is symmetric in |
| **theorem check** | the anchor the others need | the implementation agrees with the source's own theorem | evaluation at values where the quantity does **not** vanish | nothing the theorem does not state |
| **deliberate mutation** | a passing test proves nothing unless a wrong version fails it | each plausible way of building the thing wrong is shown to fail | the component is broken on purpose and the suite is required to go red | a mistake nobody thought to make |

Three supporting rules make the table mean what it says.

- **The oracle laws share no code with the library.** `tests/unit/test_oracle_independence.py`
  asserts that the oracle modules never import `cleverly`. A shared helper would move both sides of
  the comparison equally.
- **A heading is not enough.** `tests/unit/test_registry.py::TestEvidenceManifest` checks the
  evidence table against the target registry in both directions, checks that every module named
  there exists, and checks that the oracle-law column names the law whose functional really has the
  branch.
- **Cross-fitting is checked without a tolerance.**
  `tests/unit/test_crossfit_leakage.py` rigs a law in which one covariate is constant within a
  cluster and the outcome *is* that covariate with no noise. A nearest-neighbour learner then
  reproduces a held-out row bit for bit if and only if a same-cluster row was in its training set.
  The assertions are array equality, so leakage is not a matter of degree.

### The oracle-law gate

Registering a target whose reported parameters have no branch in an oracle law's `functional` is a
test failure rather than an oversight caught in review. The evidence this package offers that an
influence curve is correct is that it agrees with one obtained by complex-step differentiation of
an independently written functional on an exactly representable law. An estimand without that has
no such evidence.

The gate walks the *parameter* names a target reports rather than the target name, so a per-arm
target needs an oracle for each arm. A target intended for more than two arms needs one on the
three-armed law, because two arms cannot distinguish code that keys by arm from code that has two
columns and calls them 0 and 1. The gate runs in both directions. An oracle branch that no target
reports is dead code, so a law and the registry must cover each other exactly.

### Registered repeated-sampling studies

The instruments above ask whether each parameter is implemented correctly. A registered study asks
the complementary question. Apply a complete estimator to samples from a known law, and does its
bias and its uncertainty behave as its source theory predicts?

The design rules are in
[method benchmarking strategy](../development/method-benchmarking.md). The grid is in
[the technical reference index](method-evidence/validation-grid.md). The test-by-test results
are in [the implementation validation studies](method-evidence/index.md).

Three properties of the harness are worth stating here, because they are what make a green study
mean something.

- **A verdict is bounded by a margin declared before the run.** No rule tests whether a discrepancy
  is exactly zero.
  [The verdict rules](method-evidence/how-to-read.md#the-verdict-rules) give the argument and list
  every rule with its own control.
- **Every positive claim carries a control that must fail.** Double robustness carries a
  both-wrong-nuisance control. A type-I error cell carries a power cell, so an inert test cannot
  pass by never firing. An interval-calibration cell carries deliberately invalid inference.
- **A replication is a fixed sample.** Seeds spawn on the study's own record and on the replication
  index, so replication *k* is the same draw whatever the study's budget. A two-replication probe
  redraws exactly the published first two.

Matching a canonical R implementation is a separate and weaker claim. Two implementations
descended from one source share transcription errors, so agreement localises a discrepancy and does
not certify either one. Every study therefore tests each implementation against known truth first,
and separately.