Results, inference, and assessment#

Run the assessment battery#

Call assess() to collect validation, diagnostics, and sensitivity in one report.

battery = result.assess()
print(battery.summary())
support = battery.report("support")

The summary prints three sections. Returned results expands each completed analysis with its returned result. Checks and Not run inventory every other row. Each group states its status and count on its first row. It then gives one surface-qualified operation on each line.

Call battery.to_frame() for the complete row ledger. It carries the surface, check, status, detail, and next_steps columns. The summary prints no next step, so call battery.next_steps() for the follow-up actions.

ledger = battery.to_frame()
deferred = ledger[ledger["status"] == "deferred"]
print(deferred[["surface", "check", "next_steps"]])

for step in battery.next_steps():
    print(step)

battery.attention contains failures and warnings. battery.omissions contains deferred, not-applicable, and unavailable analyses. Use battery.report(...) to retrieve the detailed payload from an operation that ran.

The default call reads stored artifacts and runs the cheap E-value retarget when applicable. It does not refit nuisance models.

Pass analyst choices through arguments. Opt in to each expensive work class by name.

battery = result.assess(
    include_refits=True,
    include_retargets=True,
    random_state=21,
    arguments={
        "benchmark": {"covariates": ("W1", "W2")},
        "simulated_confounding": {"grid": grid},
    },
)

The common seed reaches refute, benchmark, and simulated_confounding. A seeded fit reuses its fit seed when you omit this argument. An unseeded fit draws a seed and records it on the returned row.

assess() refuses arguments for score_equations. The validation battery owns that name and runs it argument-free. The battery presents the validation row for it, so it would compute your answer and then hide it. Call result.diagnostics.run_all(arguments=...), or call the operation itself.

The battery also owns support and nuisance_models, but neither one accepts an argument. An argument for either is a TypeError from the signature. An empty mapping applies nothing, so assess() accepts it for all three names.

Read the support label#

Every estimate carries a support label. The label says whether a registered study covers the configuration that produced the estimate. You do not request it, and no argument turns it off.

label	meaning
`supported`	a support row covers the configuration, and its study cells passed. The row names the study and states its limits
`experimental`	the package computes the estimate, but no registered study covers this configuration, or a stated condition applies. The label makes no coverage claim

An experimental label does not mean the estimate is wrong. It means the package has no registered evidence for the interval, the band, or the derived value you read. A configuration that no row covers also reads experimental. A derived estimate, such as a ratio, a contrast, or a restricted mean, is supported only when its row lists the derived kind and every input is supported. Where a row, a status, or a condition makes the estimate experimental, the label carries the reason.

surface	where the label appears
`repr(estimate)`	the marker `[experimental]` after the estimate
`result.summary()`	a `support` column, only when an estimate is experimental, and one `experimental:` note per distinct reason below the table
`estimate.support`	a `Support` record with `label`, `reason`, `evidence`, `row` and `limits`
`estimate.to_dict()` and `result.to_frame()`	`support` on every row. An experimental row adds `support_reason` and `support_evidence`. A row with limits adds `support_limits`
`estimate.bootstrap.support` and `bootstrap_support`	the label of the percentile interval, which a row can license apart from the Wald interval
`result.simultaneous.support`	the label of the band
`result.config.support_key`	the key the fit was classified under. `result.save()` and `cleverly.load()` keep it with every label

The evidence of a supported estimate holds study slugs. The evidence of an experimental estimate holds roadmap ids, such as X54, that name the missing evidence.

A fit with fewer clusters than the interval floor shows its interval under the experimental label. Its bootstrap prints as bootstrap sd and a percentile range. A fit that withholds its interval, such as a collaborative fit, keeps its point estimate and refuses ci. The inference status table lists each case.

Result structure#

CausalResult is a read-only mapping from stable aliases to ParameterEstimate objects.

names = list(result.estimates)
table = result.to_frame()
covariance = result.covariance(names)
point = result[names[0]]

print(point.psi, point.std_error, point.ci, point.pvalue)

A collaborative fit refuses std_error, ci, and pvalue with CapabilityError at every strategy. A DRTMLE fit with a guard and varying weights declared estimated (weights_estimated=True) reports them under the experimental label. A clustered fit refuses them below 3 clusters with positive weight mass, and reports them under the experimental label from 3 to 9 clusters, or to 19 for longitudinal TMLE. The property result.inference_status gives the status of the fit, and point.supplies_inference gives it for one estimate.

Read point.plugin_std_error and point.plugin_interval on such a fit. Each is a diagnostic of the reported curve, and neither is a confidence statement. Inference status lists each status, the fit that declares it, its reason, and the renamed columns. A fit reports an interval only when no non-inferential row of that table applies to it.

result.parameter_keys maps each alias to a structured ParameterKey. Use those fields for programmatic selection. Display aliases are not a serialization format.

result.estimate is the sole ParameterEstimate only when the result holds one parameter, and result.psi() is its numeric point value. With several parameters, index the alias or pass a name to psi.

Contrasts and simultaneous inference#

The result retains influence curves and their joint covariance, so smooth contrasts and simultaneous bands do not refit nuisance models.

if len(names) >= 2:
    difference = result.contrast(lambda values: values[0] - values[1], names[:2])

Use cluster roles in the study design for cluster-robust variance. Below 40 clusters with positive weight mass, a TMLE, DR-TMLE, or longitudinal TMLE fit reports Student t intervals, and the summary prints a df column. Below 10 such clusters, or 20 for longitudinal TMLE, it reports no interval. For TMLE and DR-TMLE, both rules also apply to one reported stratum. Clusters gives the rules. Use Inference(simultaneous=True) when the reported family, rather than each interval separately, needs error control.

A contrast inherits the inference status of its inputs. On a fit whose status supplies no inference, the contrast refuses ci as its inputs do. The fit builds no simultaneous band.

Use ratio for the risk ratio or the odds ratio of two levels. The interval is on the log scale. Use wald_test(null=...) to test an estimate against a value other than the default null. Pass transform= to contrast to compute the interval on the scale of a monotone map.

from cleverly.inference import Transform

risk_ratio = result.ratio("ey[1]", "ey[0]")
test = risk_ratio.wald_test(null=1.5)
logged = result.contrast(
    lambda values: values[0] / values[1],
    ["ey[1]", "ey[0]"],
    transform=Transform.log(),
)
print(risk_ratio.ci, test.statistic, test.pvalue, logged.ci)

call	what it reports	the null it tests by default
`result.ratio(a, b)`	\(\psi_a / \psi_b\), interval on the log scale	1
`result.ratio(a, b, kind="or")`	the ratio of the two odds. The outcome must be binary	1
`estimate.wald_test(null=c)`	the Wald statistic and its two-sided p-value at \(c\)	the scale’s default: 0, or 1 for a ratio
`result.contrast(fn, names, transform=f)`	\(fn\) of the estimates, with the interval \(f^{-1}(f(\hat h) \pm z\,se)\)	\(f(0)\), so pass `wald_test(null=...)` when 0 is outside the domain of \(f\)

Transformed parameters gives the arithmetic.

Diagnostics#

validation = result.validate()
support = result.diagnostics.support()
nuisance = result.diagnostics.nuisance_models()
scores = result.diagnostics.score_equations()
influence = result.diagnostics.influence()
corrections = (
    result.diagnostics.corrections()
    if result.diagnostics.capability("corrections").available
    else None
)
stability = result.diagnostics.truncation_curve()
all_cached = result.diagnostics.run_all()

influence() reports how few rows carry each estimate’s influence curve. It gives the Kish count, the skewness, the kurtosis, the relative standard deviation of the variance estimate, and the coverage loss of the Wald interval that the Edgeworth term predicts. This operation is a direct call. run_all() and the assessment battery do not include it.

The correction diagnostic is available only for a DR-TMLE fit whose guard actually subtracts a correction term. Ordinary and collaborative TMLE report it as not_applicable.

For a longitudinal result, nuisance.rows covers every retained nuisance fit. Each fitted treatment and censoring model appears once per time node. Outcome and pseudo-outcome rows also identify their regimen, cause, and horizon when those fields apply.

The evaluation field is out_of_fold when the fit used more than one outer fold. It is in_sample for a one-fold fit. The loss_name field is log_loss, brier, or mse. The loss field holds the named value.

Treatment and censoring rows report weighted negative log likelihood. An outcome row reports weighted Brier loss for a binary target, or mean squared error otherwise. A pseudo-outcome row reports weighted mean squared error.

The mse field answers about node regressions alone. A treatment row and a censoring row report nan there. On a weighted fit the value averages under the observation weights.

The nested model retains calibration and Super Learner details when the learner supplies them. nuisance.omissions retains typed reasons for unavailable mechanism rows. A complete-data fit records LONGITUDINAL_CENSORING_NOT_FITTED. A censoring node with no censored unit records LONGITUDINAL_NO_CENSORING, and each later node of a held baseline treatment records LONGITUDINAL_HELD_DECISION. A node regression whose followers all hold 0, such as a node with no event, or all hold 1, records LONGITUDINAL_CONSTANT_TARGET. Validation methods lists every reason.

A categorical treatment row uses kind="multinomial probability" and an empty calibration table. The report does not convert top-class confidence into binary calibration for a privileged arm.

A binary row reports calibration_slope and calibration_slope_se. An outcome or pseudo-outcome row with a continuous target reports its linear slope as regression_slope. The longitudinal report applies no calibration rule.

nuisance.selection retains the complete C-TMLE selector or outcome-adaptive artifact. For these fits, nuisance.treatment_role is collaborative_working_model. The propensity values describe the selected denominator and not treatment given the complete adjustment set.

The role drops exactly two claims. The report keeps every other claim it makes about the propensity.

claim	on a collaborative fit	why
an AUC below 0.55 means overlap is excellent and confounding by these covariates is limited	dropped	the claim reads a treatment law that the fit never estimated. An intercept-only candidate gives an AUC near chance by construction
a calibration-slope finding says the predictions are more extreme or more moderate than the observed rates	dropped	the same reason. A selected working mechanism has no calibration target
an AUC above 0.9 signals a positivity problem	kept	`CTMLE._nuisances` puts the selected mechanism on `nuisance.propensity`, so it is the denominator the clever covariate divides by
a super learner that puts over 80% of its weight on the marginal mean contributes little	kept	the note describes a learner library and not a treatment law

Two tests in tests/e2e/test_ctmle.py hold this rule. They are test_the_role_suppresses_two_claims_and_no_others and test_a_mean_only_learner_library_is_reported_for_a_working_model_too. Each one mutates the retained report and reads the claim under both roles.

On a repeated fit, nuisance.repeat_spread contains one typed row per reported parameter. Each row gives standard_deviation, reported_standard_error, and ratio_to_standard_error.

Both standard_deviation and reported_standard_error are on the inference scale of the estimand. That scale is the log scale for rr and or, and the outcome scale otherwise. The two are therefore comparable, and test_a_ratio_takes_its_spread_on_the_scale_its_standard_error_lives_on in tests/unit/test_repeated_crossfit.py checks both scales on one fit. The ratio stays descriptive and carries no pass threshold.

Use nuisance.repeat_spread_frame() for a dataframe in the input backend. A one-draw fit retains no spread row. nuisance.repeat_spread_omission then names the cause, and every absent or non-finite spread gets its own machine-readable reason. A table cell with no finite value prints -.

Nuisance models and C-TMLE selection describe draw 1. Split-spread rows read every retained draw. The nuisance summary states the scope on its own line, as describing draw 1 of 3.

The combined assessment row states the draw beside the C-TMLE selection fact alone. An ordinary repeated fit therefore carries no draw text in that row. test_ctmle_repeats_its_selection_per_draw in tests/unit/test_repeated_crossfit.py checks the collaborative case.

Missing-outcome and controlled-direct-effect support reports add a derived product row. This row is the complete denominator used by the clever covariate. Its quantiles describe the raw product. Its Kish summaries use the bounded product on residual-contributing rows. The calculation also includes observation weights.

The estimator bounds each fitted factor before it multiplies them. The product row follows that order and counts a cell when any factor changes. It does not apply a separate product bound. The factor rows remain available, so an analyst can locate which mechanism creates the joint load.

Read the product row rather than the factor rows alone. Each factor can look acceptable while the product concentrates the weight, because the two costs multiply. The clever covariate divides by the product.

The product row describes the marginal-mean estimands. These are ey and its contrasts, a regime, and a marginal structural model. A fit that targets only att or atc reports its factor rows and no product row. Those covariates normalize by P(A=a) and reweight through propensity odds. The product row does not describe that construction.

The report names each estimand the product row does not describe. It also names what that estimand divides by instead. composed_excluded carries the same list for a caller that must branch on it. Read that note before you read the absence of a row. An absent row and a healthy denominator look the same, and the note is what separates them.

The group_score_load table reads each fitted score equation separately. For row \(i\) and equation \(j\), \(w_i H_{ij}\) multiplies the targeted residual. The table summarizes its magnitude, not the residual contribution. The group row selects the equation with the smallest Kish load ratio among score-mask rows. Its equation field identifies that equation.

mean_load = support.group_score_load["mean"]
print("equation:", mean_load["equation"])
print("Kish load ratio among targeted rows:", mean_load["targeted_ratio"])
print("top 5% load share:", mean_load["top_5pct"])
print("fitted bound:", (mean_load["lower_bound"], mean_load["upper_bound"]))

Read this row as a concentration diagnostic. It is not an effective sample size for the estimate. It does not include the targeted residual, so it is not an influence contribution. n_targeted counts score-mask rows, including structural zeros in the selected equation. The row also does not prove or disprove population positivity.

The public support workflow provides these group rows for mean, att, atc, and msm fits. Regime, shift, and incremental fits return intervention-specific support reports instead. Each policy row carries the matching fitted equation in score_load. Read its equation, targeted_ratio, and top_5pct entries together. A longitudinal fit returns per-node support diagnostics and carries no load row of either kind.

For a longitudinal fit, support() is the aggregate report name.

The policy row keeps the intervention ratio and support measures separate from score_load. Unequal observation weights or a further mechanism can make the two concentrations differ. An incremental row describes its outcome score only. The separate treatment-mechanism equation has no retained row-level load.

For repeated cross-fitting, reported_repeat and n_repeats make explicit that these retained loads describe draw 01. Every printed draw on a policy row is rendered from those two fields, so a summary and the combined support row cannot disagree about the total. The loads do not describe the coordinatewise median-combined estimate.

A report can lack exact score weights, for example when the fit recorded no score equation for the target. The report then keeps every existing policy field, sets score_load to None, and records the reason in score_load_omission. Do not replace an intervention-specific report with the generic positivity function.

Read the group row for the fitted clever covariate. Read the product row for a composed mechanism denominator. The report keeps these quantities separate because they answer different questions.

Combined reports carry one status per row. The status contract names every status. It is the canonical list. passed and failed belong to checks with an explicit verdict. completed means a descriptive analysis ran without an inferential verdict. warning uses an existing diagnostic rule.

deferred means that the same operation can run after the caller supplies a required choice or cost opt-in. These rows keep their next step and remain in omissions. They also keep the request you made, with the seed a combined run supplies to the operations that accept one. Pass that mapping back to the direct call to run the row you deferred.

An ambiguous default estimand is one such choice. Each operation below answers for one parameter. Seven of them default to estimand="ate", and a multi-arm fit reports no bare ate. evalue() defaults to None and selects the contrast itself, and a multi-arm fit gives it two. The caller then owns the choice. The report defers each of these rows and names estimand in the next step.

operation	facade	default estimand
`omitted_confounding()`	`sensitivity`	`"ate"`
`robustness_value()`	`sensitivity`	`"ate"`
`elements()`	`sensitivity`	`"ate"`
`benchmark()`	`sensitivity`	`"ate"`
`contour()`	`sensitivity`	`"ate"`
`tipping_gamma()`	`sensitivity`	`"ate"`
`refute()`	`diagnostics`	`"ate"`
`evalue()`	`sensitivity`	`None`

Pass the parameter through arguments= to run the row. The row is available again for that request. A sensitivity analysis needs no argument when the fit reports exactly one eligible parameter, because it supplies that name for you. refute() supplies no name, so its row defers for one eligible parameter as well.

Four rows can also defer on another argument. The row defers when the call refuses the default value and another value runs. It reads unavailable when no value runs. The row quotes the sentence that the call raises.

operation	argument	the row defers when
`refute()`	`tests`	the fit refuses one default test and admits another, such as `subset` on a fit given `split_plan=`
`truncation_curve()`	`mechanism`	the fit refuses the default axis and admits the other, such as the treatment axis of an incremental fit with missing outcomes
`benchmark()`	`covariates`	always, because the call needs the argument. A fit with one covariate reads `unavailable`, because the call cannot drop every covariate
`simulated_confounding()`	`estimand`	the call refuses the default estimand, and another estimand that the fit reports runs

An expected refusal after invocation becomes unavailable. An estimand you name that the fit never reported is such a refusal. The aggregate run then continues with other accepted diagnostics. Direct calls still raise the precise refusal. not_applicable and unavailable also appear in omissions.

Known omissions carry the capability’s reason. Examples include an E-value without a supported contrast and a missingness analysis without missing outcomes. Such a row records no arguments, because the fit refuses it before the report considers your request. A row that your request refuses is different. It records your arguments with the signature defaults filled in, and its reason names the refused value. An example is truncation_curve with mechanism=False on an incremental fit with missing outcomes.

An operation can also refuse after invocation, such as omitted-confounding sensitivity when the doubly robust \(\nu^2\) is not positive. That row becomes an unavailable omission, retains its invocation arguments, and names the direct call. Other accepted diagnostics still run.

Structural errors still stop the report. Examples are an invalid argument name, an unknown refute test, and a malformed benchmark request. An empty covariates, an indicator column of an encoded covariate, and an unknown covariate are malformed. A direct call raises the same error.

A combined report runs summaries and cheap retargets by default. The two costlier classes are named separately because they are disjoint. refute() and benchmark() refit nuisance models. Point-treatment truncation_curve(), missingness(), and tipping_gamma() retarget cached nuisances. The report marks costlier rows as deferred until the caller passes the matching flag. The E-value retarget is cheap and runs by default.

Two truncation curves move into the refit class. Guarded DR-TMLE refits its reduced regressions at every bound. Longitudinal TMLE refits its complete backward recursion. Both rows ask for include_refits=True. Read the class from the capability row rather than from the operation name.

The operation varies a finite-sample regularisation choice. It does not change the requested estimand. Movement shows sensitivity to extrapolation in regions where estimated support is limited.

The curve records upper_bound, the parameter-specific fitted bound pair and estimate, and delta_from_fitted. Its default grid contains each fitted pair exactly. An explicit bounds= grid keeps the requested grid cardinality and can therefore have no fitted marker. The fitted columns still name the configuration the grid omitted.

A longitudinal curve has no default grid. Supply bounds= explicitly. A scalar b means the cumulative lower-only pair (b, 1). An explicit (lower, upper) entry keeps both limits. This differs from point treatment, where scalar treatment bounds mean (b, 1 - b).

Each longitudinal row is a descriptive ordinary LTMLE point estimate under one fixed cumulative mechanism bound. The replay keeps the data, resolved plans, folds, raw mechanism predictions, weights, clusters, and estimand structure fixed. It recomputes bounded cumulative probabilities. It then refits every bound-dependent outcome or pseudo-outcome regression and targeting update. It never reuses an earlier-node outcome fit or prediction.

Before the grid runs, a fitted-bound preflight must exactly reproduce every retained estimate, regimen fit, and MSM fit. This check also runs when the requested grid omits the fitted pair. A mismatch makes the operation unavailable.

Its frame carries one row per evaluated pair and reported parameter. The row records both endpoints, the parameter, the replayed psi, the fitted pair, fitted_psi, is_fitted_bound, delta_from_fitted, and two score-cell counts. Read the lower endpoint as lower_bound here, because the point-treatment frame above names the same endpoint bound. Truncation stability holds the column table and the two counting rules.

The operation reports no standard error, interval, preferred bound, selected-bound correction, or pass threshold. It does not validate positivity or account for uncertainty from choosing a bound. It refuses mechanism=True, which is a point-treatment option. It also refuses a result without its stored replay recipe, and a learner it cannot replay. Replay-only unavailability states the random_state rule each learner must satisfy.

estimands= restricts the rows to the parameters you name. Name a reported parameter, such as ey[high], to select its own row. Name a registered target, such as ey, to select one row for each arm of that target the fit reports. The two forms coincide on a two-armed fit. The curve refuses any other name, and it refuses an empty selection.

A combined report gives one signed movement range for each reported parameter or working-model coefficient. It measures that movement from the parameter’s own fitted estimate. It also counts the parameters whose fitted pair the grid skipped. Read the fitted estimate and the fitted pair from the retained curve.

Explicit odds-ratio and reported risk-ratio E-values summarize existing estimates:

everything = result.diagnostics.run_all(include_refits=True, include_retargets=True)

Pass required choices with arguments={"operation": {...}}. The report row retains the effective arguments and the returned object. Use report.report(name) to retrieve that object.

A refusal names the first thing that is wrong. An operation that needs a choice you did not pass names that choice, and not a cost flag. No flag supplies the covariates for benchmark, so the row says so under every flag.

A completed tipping_gamma search can return None when its interval contains no tipping point. Use battery.report(name, surface="diagnostics") to retrieve a diagnostic that also appears under validation.

The E-value row records its source alias, such as estimand="ate", for replay. A derived ratio can have a different output alias, such as rr. For a default odds-ratio source, the row keeps estimand=None because an explicit alias chooses the approximation. The returned E-value records that source in source_estimand.

The omitted-confounding row says “at the default strengths” when you omit both strength arguments. It names the omitted strength when you supply only one. The effective arguments retain both numeric values.

Use generated outcomes to test the full pipeline against a declared effect. The dummy operation uses independent Gaussian noise and declares zero. The simulated operation uses a standardized adjustment function, an additive treatment term, and Gaussian noise.

from sklearn.linear_model import LinearRegression, LogisticRegression

from cleverly import ATE, CausalStudy, PointTreatment
from cleverly.datasets import make_linear_ate
from cleverly.validation import GaussianAdjustmentOutcome

gaussian_frame, _ = make_linear_ate(n=200, seed=21)
gaussian_study = CausalStudy(
    gaussian_frame,
    design=PointTreatment(outcome="Y", treatment="A", adjustment=("W1", "W2", "W3", "W4")),
)
ate_result = gaussian_study.estimate(
    ATE(),
    outcome_learner=LinearRegression(),
    treatment_learner=LogisticRegression(max_iter=1000),
    cross_fit=False,
    random_state=21,
)
generated = ate_result.diagnostics.refute(
    tests=("simulated_outcome",),
    simulated_outcome=GaussianAdjustmentOutcome(effect=0.5),
    n_replicates=40,
    random_state=21,
)
draws = generated.draws_frame("simulated_outcome")

That call reports includes 0.5 yes on 40 successful refits. Each row in draws records the child seed, estimate, standard error, family, or refit failure.

Use bootstrap measurement error to assess estimate stability under declared covariate error. The operation samples rows first. It then perturbs original adjustment variables and refits the complete estimator.

from cleverly.validation import BootstrapMeasurementError, RelativeGaussianNoise

measurement = ate_result.diagnostics.refute(
    tests=("bootstrap_measurement_error",),
    bootstrap_measurement_error=BootstrapMeasurementError(
        variables=("W1", "W2"),
        numeric_noise=RelativeGaussianNoise(standard_deviation=0.1),
    ),
    n_replicates=40,
    random_state=21,
)
measurement_draws = measurement.draws_frame("bootstrap_measurement_error")

The numeric multiplier uses each variable’s standard deviation in its bootstrap sample. Categorical changes act on the original variable and rebuild its complete indicator block. Set categorical_change_probability on the declaration to control those changes.

The operation compares the original estimate with the empirical refit distribution. It can pass or fail because it measures stability under the declared error. It stays outside DEFAULT_TESTS because every draw resamples, perturbs, and refits. Plain bootstrap inference does not add measurement error.

The placebo refutation permutes treatment and interprets movement toward zero as evidence for an effect claim. NaturalCourseMean is an outcome level, not an effect, so that interpretation does not apply and the operation refuses it before refitting. The random_common_cause and subset refutations retain their usual stability interpretations for this target.

Two defaults govern generated-outcome calls, and they are separate objects. n_replicates defaults to DEFAULT_OUTCOME_REPLICATES, which is 100 draws. outcome_rule defaults to EmpiricalInclusionRule(), which needs 40 successful draws, uses alpha 0.05, and fails when any refit fails. The simulated-outcome example asks for 40 draws, which is the smallest budget the default rule accepts.

Read alpha as a width and not as a false-alarm rate. The rule passes when the declared effect lies inside the central 95% of the refit estimates, which the technical reference derives. Generated outcomes apply only to identified additive contrasts for binary point treatments.

score_equations(tolerance=...) gates both result families, but on the scale each one’s score lives on. A point-treatment fit compares the score in the outcome’s own units against tolerance * se / sqrt(n); a longitudinal fit bounds each node’s relative score, the quantity the sequential targeting loop itself gates on, and can only tighten the fit’s own convergence verdict rather than overturn it.

Read a calibration-slope finding#

The point-treatment report tests the calibration slope of each probability model. The slope comes from a logistic recalibration of the out-of-fold predictions with one intercept per fold. A slope below 1 means the predictions are more extreme than the observed rates. A slope above 1 means they are more moderate. The calibration-slope rule gives the statistic and its standard error.

The report warns only when the slope’s interval lies above 0 and excludes 1. The interval splits a level of 0.05 over the tested models. A finding reads like this line from a learner whose logit is doubled:

propensity: the out-of-fold predictions are more extreme than the observed rates (calibration
slope 0.50, 97.5% interval 0.44 to 0.56, AUC 0.756, largest untruncated inverse weight 220).
Estimation noise can cause this even in a correctly specified model when the signal is weak or
the model has many covariates. Review the learner before you change the model

question	answer
what does a finding ask you to do?	review the learner. For the `propensity` model of a binary point treatment, `result.diagnostics.support()` describes the weights that model creates
can a correct model warn?	yes. With a weak signal or many covariates, estimation noise can make out-of-fold predictions more extreme than the truth
does a finding mean that the model is misspecified?	no. A misspecified logistic model with an intercept also has a population slope of 1, so the slope cannot detect it
does a finding mean that the estimate is biased?	no. The rule measures the spread of the predictions and does not measure the estimate
why is there no finding on an in-sample fit?	the report tests out-of-fold predictions only. A fit without cross-fitting has `nuisance.evaluation` equal to `in_sample`, and the report tests nothing there. An unpenalized logistic model with an intercept has an in-sample slope of exactly 1, so that slope says nothing about new rows
why is a slope missing?	`nuisance["propensity"].calibration_omission` names the cause for that model, for example predictions that are constant within every fold. `nuisance.summary()` prints each cause under the table. `nuisance.to_frame()` has no column for it

The calibration-slope study measures the rule’s false-warning rate and its detection rate for a binary propensity.

Sensitivity analysis#

all_sensitivity = result.sensitivity.run_all()
print(all_sensitivity.summary())

Support and truncation stability live under result.diagnostics. Point-treatment sensitivity methods are explicit: omitted_confounding(), robustness_value(), elements(), benchmark(), contour(), evalue(), missingness(), tipping_gamma(), and simulated_confounding(). A simulated surface needs an explicit strength grid, so run_all() never starts it.

The missingness() tilt and the tipping_gamma() search perturb the arm-indexed means and their linear contrasts. The table gives each missing-outcome fit where both are unavailable.

fit	reason
`NaturalCourseMean()`	the natural-course target needs its own sensitivity parameter
a shift fit	whether the tilted parameter is still the shift parameter has not been derived
an incremental fit	its targeting also solves a score in the treatment mechanism, and no derivation covers the tilt of that fixed point
a regime, MSM, or ratio-only fit	the fit reports no arm-indexed mean and no linear contrast

A per-arm outcome-adaptive CTMLE fit regresses its observation factor on the outcome regression only. Both methods read the full-covariate model nuisance.tilt_missingness on that fit, and no targeting step reads it.

Both capability rows read unavailable before any call, with the sentence that the call raises. A fit with no missing outcome reads not_applicable. The combined assessment reports the same row instead of substituting an armwise analysis.

from cleverly.sensitivity import ConfounderStrengthGrid

grid = ConfounderStrengthGrid(
    treatment=(0.0, 0.05, 0.10),
    outcome=(0.0, 0.25, 0.50),
)
surface = ate_result.sensitivity.simulated_confounding(
    grid=grid,
    benchmark_covariates=("W1", "W2"),
    random_state=21,
)
movements = surface.to_frame()
calibration = surface.calibration_frame()

An ordinary-TMLE fit can declare fixed probability weights. Binary complete-outcome collaborative TMLE and DR-TMLE fits can also declare them. The surface keeps each normalized weight on its row during every treatment replacement, outcome replacement, and complete refit. Read surface.target_measure to distinguish unweighted from fixed_empirical_tilt. surface.weight_report records the weight column, kind, supplied scale, and concentration. The label follows the declared weight column, so read surface.weight_report.is_weighted to learn whether the realized tilt is nonconstant.

Under fixed_empirical_tilt, each cell evaluates its parameter on the perturbed weighted empirical law. This operation does not reproduce the original sampling or selection mechanism. It conditions on the fixed weights that the fit stores.

The surface keeps the fitted estimator’s repeats setting. Each non-anchor cell runs one complete refit, and the estimator combines its draws with a coordinatewise median. Read surface.n_repeats and surface.repeat_aggregation to confirm that provenance. repeat_aggregation names the rule for more than one draw. A single draw needs no aggregation. The shared root seed gives every non-anchor cell the same repeat seed sequence.

The folds themselves do not move. The draw reads the row count, the cluster labels and the seed, so a treatment or outcome perturbation leaves the realized partition alone.

Pass the seed of the original fit, or pass no random_state at all. The surface reproduces the folds of that fit under the seed of that fit alone. A different seed gives every non-anchor cell new folds, and a fit that declared no seed has the same effect. The anchor keeps the original folds, so movement near the anchor can carry a fold artifact.

The operation draws one shared latent variable. For binary treatment, it flips the treatment in the upper latent tail. A Gaussian outcome subtracts the outcome strength times the latent value. A binomial outcome flips in the same tail.

A binary CounterfactualMean(treatment=...) fit reports one ey1 or ey0 mean. The facade selects that sole mean when you pass the grid. A CounterfactualMean() fit reports ey[...] for both arms. Pass one exact alias through estimand= for that multi-mean result. The intervention arm stays fixed while the operation perturbs the observed treatment and outcome.

Binary ordinary-TMLE fits also support fixed regime means, regime contrasts, incremental targets, and MSM coefficients. Continuous ordinary-TMLE fits support MSM coefficients too. Select a reported alias when the fit contains more than one eligible parameter.

fitted target	what the surface preserves
`RegimeMean` or `RegimeContrast`	the `Static`, `Rule`, or `Stochastic` policy at each baseline row
`MSMProjection`	the built-in link, grid design, coefficient names, and known projection measure
`IncrementalMean` or `IncrementalEffect`	odds multipliers, names, and the reference

The surface refits the treatment and outcome models in each non-anchor cell. It reports additive movement and preserves fixed observation weights and baseline strata. It checks callbacks against stored inputs before the first draw. Continuous MSMs reevaluate observed-dose functions after perturbation. They preserve the integration grid and apply quadrature once. Log and logit MSMs report coefficient differences without exponentiation.

Incremental targets and MSMs under every built-in link keep baseline-stratum support, because each replay is a complete refit that targets the stratum blocks as the fit did. Continuous MSM assessments mark the unimplemented dose-grid support diagnostic as unavailable and continue to the requested surface.

This example compares a known stochastic assignment with never treating.

import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression

from cleverly import CausalStudy, PointTreatment, RegimeContrast
from cleverly.datasets import make_linear_ate
from cleverly.interventions import Static, Stochastic
from cleverly.sensitivity import ConfounderStrengthGrid


def navigation_probability(w):
    probability = 0.25 + 0.5 * (np.asarray(w["W1"]) > 0)
    return np.column_stack([1.0 - probability, probability])


policy_frame, _ = make_linear_ate(n=160, seed=31)
policy_result = (
    CausalStudy(
        policy_frame,
        design=PointTreatment(outcome="Y", treatment="A", adjustment=("W1", "W2")),
    )
    .identify(
        RegimeContrast(
            regimens=(
                Static(0, name="never"),
                Stochastic(navigation_probability, name="policy", density_kind="known"),
            ),
            reference="never",
        )
    )
    .estimate(
        outcome_learner=LinearRegression(),
        treatment_learner=LogisticRegression(max_iter=1000),
        cross_fit=False,
        learner_folds=2,
        random_state=31,
    )
)
policy_surface = policy_result.sensitivity.simulated_confounding(
    estimand="ate_regime[policy vs never]",
    grid=ConfounderStrengthGrid(treatment=(0.0, 0.1), outcome=(0.0, 0.25)),
)
policy_surface.to_frame()

The probabilities follow the fitted treatment-level order. An incremental intervention instead depends on the fitted treatment mechanism. Each incremental cell rebuilds its intervention from that refitted mechanism. Movement includes the change in intervention probabilities. The surface refuses a multiplier-one mean, but accepts contrasts against that reference. The technical contract states the remaining policy and projection boundaries.

An ATT() or ATC() fit without baseline strata reports one eligible alias. The facade selects that sole alias when you pass the grid, exactly as it selects a sole counterfactual mean.

A binary RiskRatio or OddsRatio fit without baseline strata needs only the grid. Each cell reports the refitted ratio in estimate. Its displacement is the refitted log ratio minus the original log ratio. The surface and its frame record movement_scale="log_ratio". Exponentiate a displacement to get the refitted ratio divided by the original ratio.

A binary PopulationAttributableRisk or PopulationAttributableFraction fit without strata also needs only the grid. PAR compares the natural-course mean with the reference-arm counterfactual mean. PAF divides that difference by the natural-course risk. Each cell recomputes both means after perturbation and keeps the declared reference arm fixed.

Both surfaces report movement_scale="estimate_difference". A PAF displacement is therefore the difference of two fractions, and a negative estimate remains valid. A cell with zero natural-course risk retains a failure because PAF is undefined there. The surface refuses NaturalCourseMean: its observed mean has no counterfactual treatment term for this diagnostic.

Use ordinary TMLE for PAR and PAF. It supports marginal parameters and parameters within baseline strata. The identified effect’s method catalog refuses both targets under C-TMLE and DR-TMLE, including outcome-adaptive C-TMLE. The technical contract states their source and refusal boundaries.

This example evaluates PAR for a continuous outcome under binary treatment.

from sklearn.linear_model import LinearRegression, LogisticRegression

from cleverly import CausalStudy, PointTreatment, PopulationAttributableRisk
from cleverly.datasets import make_linear_ate
from cleverly.sensitivity import ConfounderStrengthGrid

attributable_frame, _ = make_linear_ate(n=160, seed=21)
attributable_result = CausalStudy(
    attributable_frame,
    design=PointTreatment(
        outcome="Y",
        treatment="A",
        adjustment=("W1", "W2", "W3", "W4"),
    ),
).estimate(
    PopulationAttributableRisk(reference=0),
    outcome_learner=LinearRegression(),
    treatment_learner=LogisticRegression(max_iter=1000),
    cross_fit=False,
    random_state=21,
    simultaneous=False,
)
attributable_surface = attributable_result.sensitivity.simulated_confounding(
    grid=ConfounderStrengthGrid(treatment=(0.0, 0.1), outcome=(0.0, 0.25)),
)
attributable_cells = attributable_surface.to_frame()

For a fit with baseline strata=, pass the complete conditional alias from result.estimates. The operation holds the baseline stratum fixed and refits all rows at every non-anchor cell. surface.stratum records the selected values. surface.association_population identifies the population used for the treatment correlation. Numeric calibration always describes the full original fitted population, as surface.calibration_population records.

Ordinary-TMLE ATT and ATC surfaces use the treated or control group after each perturbation. Their movement includes population change. Read surface.conditioning_arm and each cell’s target_population_fraction before interpreting that movement. A baseline stratum remains fixed even when treatment-group membership changes within it. CTMLE and DRTMLE refuse ATT and ATC when they estimate, so no such fit reaches this surface.

A cell on an ATT or ATC surface moves through three channels. The flip can open a confounding path, it can misclassify the treatment, and it can change who the parameter conditions on. An association near zero rules out the confounding channel only, so it does not identify misclassification here. summary() prints that guard, and its table carries the population fraction of each cell. The assessment battery reports warning when the conditioning group keeps under half its unperturbed share. The population contract states the exact rule.

This example selects one baseline stratum through its structured key.

import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression

from cleverly import ATT, CausalStudy, PointTreatment
from cleverly.datasets import make_linear_ate
from cleverly.sensitivity import ConfounderStrengthGrid

population_frame, _ = make_linear_ate(n=160, seed=21)
population_frame["V"] = np.where(population_frame["W1"] > 0, "high", "low")
population_result = CausalStudy(
    population_frame,
    design=PointTreatment(
        outcome="Y",
        treatment="A",
        adjustment=("W1", "W2", "W3", "W4", "V"),
        strata=("V",),
    ),
).estimate(
    ATT(),
    outcome_learner=LinearRegression(),
    treatment_learner=LogisticRegression(max_iter=1000),
    cross_fit=False,
    random_state=21,
    simultaneous=False,
)
population_alias = next(
    alias for alias, key in population_result.parameter_keys.items() if key.stratum == ("high",)
)
population_surface = population_result.sensitivity.simulated_confounding(
    estimand=population_alias,
    grid=ConfounderStrengthGrid(treatment=(0.0, 0.1), outcome=(0.0, 0.25)),
)
population_cells = population_surface.to_frame()

A continuous-dose fit uses signed treatment strengths. Pass one exact modified-policy mean or contrast alias because a continuous result has no bare ate parameter.

from cleverly import ModifiedTreatmentPolicyEffect
from cleverly.datasets import make_shift_dose
from cleverly.interventions import Shift

shift_frame, _ = make_shift_dose(n=200, seed=21)
shift_study = CausalStudy(
    shift_frame,
    design=PointTreatment(
        outcome="Y",
        treatment="A",
        adjustment=("W1", "W2", "W3"),
        treatment_kind="continuous",
    ),
)
shift_result = shift_study.estimate(
    ModifiedTreatmentPolicyEffect(
        policies=(
            Shift(0.0, cap=3.0, name="natural course"),
            Shift(0.5, cap=3.0, name="up half"),
        )
    ),
    outcome_learner=LinearRegression(),
    treatment_learner=LogisticRegression(max_iter=1000),
    cross_fit=False,
    learner_folds=2,
    random_state=21,
    simultaneous=False,
)
shift_alias = "ate_policy[up half vs natural course]"
shift_surface = shift_result.sensitivity.simulated_confounding(
    estimand=shift_alias,
    grid=ConfounderStrengthGrid(
        treatment=(-0.25, 0.0, 0.25),
        outcome=(0.0, 0.25, 0.50),
    ),
    benchmark_covariates=("W1", "W2"),
    random_state=21,
)

A ModifiedTreatmentPolicy fit reports policy means instead. Select one mean with an alias such as ey_policy[up half]. The surface reports that mean and its signed displacement in each cell.

Select a policy whose delta is nonzero. A zero-delta shift assigns every unit its own dose, so its mean is \(E[Y]\) and no common cause can move it through the treatment. The surface refuses that mean by name. It still accepts an ate_policy[...] contrast that uses the zero-delta policy as its reference, because the contrast keeps its treatment dependence.

The continuous treatment law is \(A'=A+k_AU\). It keeps the declared modified treatment policies fixed during each ordinary-TMLE refit. The outcome laws and common-randomness contract stay the same as the binary surface.

On a binary fit, the treatment flip is non-differential misclassification. The association it induces between the latent variable and the treatment depends on the treated fraction. That association is zero on a balanced design. The technical reference measures it.

Each cell reports its own realised association in induced_treatment_association. The frame carries the same value in a column of that name, and summary() prints it. On a binary fit, check the column before you read a treatment movement as confounding. A value near the anchor can reflect misclassification alone.

For a fixed-weight fit, the association and numeric calibration use the weighted empirical law. Calibration weights feature scaling, model fitting, prediction-change fractions, correlations, and standard deviations. A common scale change to the weights leaves these values unchanged.

On a continuous fit, the latent variable changes the dose by construction, so the association grows with the treatment strength. A confounding path also needs the latent variable to enter the outcome, and only a nonzero outcome strength puts it there. A cell in the zero outcome-strength column therefore carries no confounding path, whatever its association. Its movement reports the dose perturbation alone. The technical reference states the same reading rule.

The zero treatment-strength column carries no confounding path either. Its movement reports the outcome perturbation alone. The Gaussian outcome law subtracts a level from every row, and an ate_policy[...] contrast removes most of that level. An ey_policy[...] policy mean keeps it, so read the zero treatment-strength column of a policy-mean surface as an artifact of the outcome law.

The operation refits the complete estimator at each nonzero strength pair. The zero cell equals the original estimate exactly. Additive surfaces record movement_scale="estimate_difference".

Read the output as a qualitative stress surface. It is not a corrected estimate, bound, p-value, confidence interval, robustness value, or pass/fail result. Calibration reports model-dependent source conventions for numeric covariates. It does not change the declared grid.

Inspect result.sensitivity.capability("simulated_confounding") before a combined run. Its available, status, and reason fields read the same fit-wide check as direct execution. The check is one ordered list, and it stops at the first boundary your fit meets. Six stops wait on published theory: longitudinal, multi-arm, missing-outcome, intermediate, estimated-weight, and clustered fits. Twelve provenance and shape stops read the result type, the replay estimator, the repeat provenance, the outcome family, the observation weights, and the identification metadata. The result-type stop runs second, because every later stop reads a field that only a point-treatment result declares.

The capability row names the same stop that direct execution would raise, for all eighteen. Take a fit whose backdoor provenance is not a registered explicit adjustment set. Its row reports available=False and the reason "simulated_confounding needs registered explicit-adjustment backdoor provenance". The technical reference lists the twelve stops in order. Each refusal occurs before calibration, a latent draw, or a refit.

Estimated-weight replay needs the fitted weight model, target-population semantics, and a regeneration rule. The fitted result does not store that contract.

A fixed-weight collaborative fit reruns its selector or outcome-adaptive treatment fit at each cell. It uses the normalized row weights in every nuisance fit, loss, penalty, targeting step, and plug-in.

The R ctmle and archived ctmle3 sources provide no weighted comparator. The fixed-weight collaborative surface makes no numerical parity claim with those implementations.

Missing-outcome replay needs a joint law for the response indicator, treatment, and outcome. Holding the response indicator fixed can break missing at random after treatment changes. Intermediate and longitudinal fits need their own ordered latent laws. Clustered fits need a source-backed choice among row-level, cluster-level, and mixed latent causes.

A combined call excludes refits and retargets by default. A row skipped for cost alone names the flag that would run it. A row that also needs a choice names the choice instead. Longitudinal operations without a published derivation are reported unavailable rather than borrowing point-treatment formulas, and each refusal gives the reason its own capability row declares.

Persistence and replayability#

from cleverly import load

result.save("analysis.joblib")
restored = load("analysis.joblib")
assert restored.parameter_keys == result.parameter_keys
assert type(restored.method) is type(result.method)
assert type(restored.method.models.outcome_learner) is type(result.method.models.outcome_learner)

Read result.diagnostics.capabilities before you run an operation on a loaded artifact. Each row declares the replay it needs, and available is False when the artifact cannot supply it. The row says which slot is missing, so you learn the answer before the operation runs.

A result can hold a copied or modified estimator whose configuration the package refuses before a refit. A stratified fold policy, a cross-fitted continuous outcome with q_bounds=None, and a C-TMLE fit on clustered data are examples. The refit_nuisances slot runs every check that refit() runs before a learner, so a design refusal of an estimator subclass counts too. The table gives what such a result keeps.

operation	on such a result
point estimates and cached-nuisance retargets, such as `truncation_curve()`	run. `replayability.retarget_cached_nuisances` is `True`. A guarded DR-TMLE curve also runs. It retargets the cached primary nuisances and refits the reduced regressions inside that step. Its row reports the `refit` cost, but it does not require the `refit_nuisances` slot of the outer estimator. The curve refuses a reduction setting that a fit refuses, such as `targeting="one_step"` with `reduced_crossfit="nested"`. It also refuses a `guard`, `reduction` or `reduced_crossfit` other than the fitted one
refits, such as `refute()` and `simulated_confounding()`	`unavailable`. `replayability.refit_nuisances` is `False`, with the code `point_replay_refit_configuration`
`estimator.refit()`	raises `CapabilityError` before any learner

A new fit under a supported configuration restores the refits.

The saved artifact carries the assessment cache. A result you derive with dataclasses.replace does not. The cache key records the operation and its arguments, and it records nothing about the result that answered them, so a derived result starts with an empty cache of its own.

The artifact contains the complete result graph, including nuisance estimator templates, so successfully restored results retain refit-based assessment. save() writes the graph with cloudpickle. It saves an importable function or class by reference. It saves a lambda, a closure, and an object defined in a notebook or in __main__ by value. A result therefore keeps the rules, policy densities, and learners that you wrote inline. A by-value object loads only under the Python minor version that saved it.

The loader is joblib.load, which uses pickle. Loading can execute arbitrary code, so load only trusted artifacts. Use an environment with the same Python minor version and compatible sklearn and third-party estimator versions.

result.save() records the cleverly version that wrote the artifact. When another version saved it, cleverly.load() warns with VersionMismatchWarning and loads the artifact as saved. The load runs no migration. The statuses, reports, and cached assessments of such an artifact can differ from what this version computes, so fit the analysis again with this version. Development snapshots between two releases share one version string, so the check does not see a change between them.