Methods, learners, and cross-fitting#

Method presets#

"tmle" is the ordinary default. CollaborativeTMLEMethod and DRTMLEMethod select compatible estimator variants; they do not change the causal question.

from sklearn.linear_model import LinearRegression, LogisticRegression
from cleverly import CollaborativeTMLEMethod, DRTMLEMethod, ModelSpec

parametric = ModelSpec(
    outcome_learner=LinearRegression(),
    treatment_learner=LogisticRegression(max_iter=1000),
)

collaborative = effect.estimate(
    method=CollaborativeTMLEMethod(models=parametric, strategy="greedy"),
    random_state=3,
)
doubly_robust = effect.estimate(
    method=DRTMLEMethod(models=parametric),
    random_state=3,
)

Call effect.available_methods() before fitting when method availability matters. An unavailable method comes with a capability reason.

SDRMethod selects the sequentially doubly robust estimator for a longitudinal functional. A point-treatment functional lists it as unavailable. The estimator fits no fluctuation, so it refuses a non-default submodel_alpha, max_iter, tol, stopping, step_bound, step_search or iterate. Its pseudo_learner fits a transform that can leave \([0, 1]\), so give it a learner of a real-valued target, such as LinearRegression(). The default is a regression Super Learner with no clip. The sequentially doubly robust estimator states its contract.

IPSIMethod selects the incremental estimator of Kennedy (2019) for IncrementalMean or IncrementalEffect on a longitudinal design. A point-treatment functional lists it as unavailable. It fits no fluctuation either, so it refuses the same settings. At a tilt node the treatment mechanism defines the plan, and the estimator is not doubly robust. Give treatment_learner a learner that can fit the mechanism at every node. Incremental interventions over time states its contract.

The collaborative fit above reports no confidence interval, no p-value, and no standard error. The "greedy" strategy is the default. "ordered", "discrete", "lasso", "learners", and "oat" refuse the same three accessors. Each estimate keeps plugin_std_error and plugin_interval as a diagnostic. For an interval, use TMLEMethod or strategy="derivative". Collaborative TMLE gives the reason for each strategy.

C-TMLE0: a derivative covariate with an interval#

strategy="derivative" fits one TMLE with an extra covariate for each arm. That covariate is the derivative of the clever covariate along the lasso penalty. The strategy selects nothing, so it reports the TMLE interval. Declare two penalties in lambdas=: the mechanism first, then its partner. Omit lambdas= to choose the mechanism penalty by cross-validation.

derivative = effect.estimate(
    method=CollaborativeTMLEMethod(models=parametric, strategy="derivative", lambdas=(0.02, 0.01)),
    random_state=3,
)
print(derivative.extra["ctmle"].describe())

The fit prints the penalty pair and the number of training sets. The default cross-fit has 10. The C-TMLE0 section lists the other two sources of the pair and the refusals.

A lasso grid that starts at a cross-validated penalty#

strategy="lasso" takes its penalties from lambdas=. To start the grid at a cross-validated penalty, fit LogisticRegressionCV with an L1 penalty on standardized covariates. Then convert its C with lambda = 1 / (C n). Use the saga solver. The liblinear solver penalizes the intercept, and the lasso candidates do not.

l1_ratios=(1.0,) selects the L1 penalty in scikit-learn 1.8 and later. cleverly requires scikit-learn 1.8 or later. Earlier releases ignore l1_ratios without penalty="elasticnet" and fit an L2 penalty. use_legacy_attributes=False makes C_ one number.

import numpy as np
from sklearn.linear_model import LogisticRegressionCV
from sklearn.preprocessing import StandardScaler

scaled = StandardScaler().fit_transform(frame[["W1", "W2", "W3"]])
search = LogisticRegressionCV(
    Cs=10,
    l1_ratios=(1.0,),
    solver="saga",
    scoring="neg_log_loss",
    max_iter=5000,
    use_legacy_attributes=False,
)
search.fit(scaled, frame["A"])
start = 1.0 / (search.C_ * len(frame))
lambdas = start * np.geomspace(1.0, 0.01, 6)
lasso = effect.estimate(
    method=CollaborativeTMLEMethod(models=parametric, strategy="lasso", lambdas=lambdas),
    random_state=3,
)

StandardScaler divides by the population standard deviation, as the lasso candidates do. tests/unit/test_ctmle_candidate_sequences.py::TestConfiguration::test_the_public_lambda_cv_recipe runs this recipe. It also checks that the search sets some coefficients to exactly 0, which an L2 search never does.

Declared learner candidates#

strategy="learners" reads treatment_candidates= in the declared order, then appends treatment_learner on every covariate. Order the candidates from the least to the most adaptive. Each candidate must fit the treatment better than the candidate before it. The search tries the candidates in this order, so an unordered list can stop the search before a confounder enters.

Nested candidates are the safest declaration. Make each candidate read the covariates of the candidate before it, and add more. The table shows a declaration on the instrument law of tests/studies/ctmle_sequence_properties.py, where W1 is the confounder and W2 the instrument.

declaration	candidates, then the terminal	treatment loss
nested	`W2`, then `W1` and `W2`, then every covariate	decreases, except for ties
not nested	`W2`, then `W1`, then every covariate	rises at the `W1` candidate

The fit warns with a UserWarning when the treatment negative log likelihood of a candidate rises by more than 1% above the candidate before it. The warning names each such candidate by its index and label. The fit continues, because a nested candidate with an uninformative covariate can rise by sampling noise under the propensity bounds. The 1% threshold separates the two cases on the instrument law. There the nested terminal rises by about 0.2%, and the W1 candidate by about 19%. CTMLESelection.treatment_risk_increases holds the same indices. tests/unit/test_ctmle_candidate_sequences.py::TestLearners checks the warning.

Both variants are named here with a parametric learner on purpose. Neither fits a fixed number of nuisance models: collaborative TMLE fits one per candidate along its selection path, and DR-TMLE re-fits its reduced regressions on every round of the alternation that solves its extra score equations, because those regressions are regressions on the current targeted pair. Each of those fits is then multiplied by the cross-fitting folds and again by the learner library. Leaving both at the default library is the difference between seconds and hours on the same data, so decide the learner deliberately for these two rather than inheriting it.

Immutable configuration#

from cleverly import CrossFitting, Inference, ModelSpec, Runtime, TMLEMethod, Targeting

method = TMLEMethod(
    models=ModelSpec(
        outcome_learner=LinearRegression(),
        treatment_learner=LogisticRegression(max_iter=1000),
    ),
    cross_fitting=CrossFitting(n_folds=5, learner_folds=3, repeats=1),
    targeting=Targeting(g_bounds="auto", algorithm="iterative"),
    inference=Inference(alpha=0.05, simultaneous=False),
    runtime=Runtime(random_state=3, n_jobs=1),
)
result = effect.estimate(method=method)

Keyword shortcuts such as n_folds=, alpha=, and random_state= normalize into these same groups. A method object makes the normalized configuration serializable and reviewable.

This example fits a binary outcome, so its Targeting declares no q_bounds. Read Two fold layers below before you copy the shape onto a continuous outcome, which needs its support declared there.

Learner choices#

Every nuisance slot takes an sklearn-compatible estimator object. Strings such as "glm" and "default" are rejected so the configuration always identifies the actual model being fitted.

When learners are omitted, cleverly constructs a task-appropriate SuperLearner over three concrete sklearn candidates: histogram gradient boosting, a random forest, and lasso (LassoCV for a mean regression or L1 logistic cross-validation for a probability). No optional model package is imported. Install XGBoost or LightGBM yourself and pass its estimator object when that is the model you want.

Construct an ensemble explicitly by passing model objects, optionally paired with report names:

from sklearn.ensemble import HistGradientBoostingRegressor, RandomForestRegressor

from cleverly import SuperLearner

outcome_model = SuperLearner(
    library=[
        ("boost", HistGradientBoostingRegressor(random_state=3)),
        ("forest", RandomForestRegressor(n_jobs=1, random_state=3)),
        ("linear", LinearRegression()),
    ],
    n_folds=3,
    random_state=3,
)

The estimator clones each candidate before fitting. The objects supplied by the caller therefore remain unfitted and can be reused in another method configuration.

Library size multiplies with both fold layers below, so it is the first thing to reach for when a fit is slower than expected.

Outcome regressions need predictions on the outcome’s mean scale. Binary treatment and censoring mechanisms need probability predictions. Continuous treatment policies use a conditional density model. cleverly validates learner task compatibility and sample-weight support.

Use GLM for the sequential baselines#

cleverly.learners.GLM is an unpenalized maximum-likelihood fit with a canonical link. Its score and information are public, so SequentialGComputation and SequentialIPW stack them into an interval. See Report the sequential baselines.

behavior	what `GLM` does
`family="binomial"` on a target with three or more labels	raises `ValueError`. Pass `family="multinomial"` or `"auto"`
a design column that is constant, or a combination of other columns, on the fit rows	aliases the column. It gets no coefficient
`alias_`	holds each aliased column as a combination of the active columns
`outside_span(x)`	returns `True` for each row where an aliased column departs from that combination

A prediction through an aliased column assumes that the column has no effect. On a row that departs from the fit rows, the data cannot support that assumption. Sequential g-computation refuses such a fit with DataError, because no follower has that history. The technical page lists this refusal for g-computation only.

import numpy as np

from cleverly.learners import GLM

x = np.array([[1.0, 1.0], [1.0, 0.0], [1.0, 1.0], [1.0, 0.0]])
model = GLM().fit(x, np.array([0, 1, 1, 0]))
print(model.active_)
print(model.outside_span(np.array([[0.0, 1.0], [1.0, 1.0]])))

The first column is constant on the fit rows, so the fit aliases it with the intercept. The row [0.0, 1.0] departs from that relation.

Use GLM for the parametric g-formula#

ParametricGFormula fits a GLM for every covariate, outcome and hazard model by default. The fit chooses the family of each model from its response: binomial for a binary column, multinomial for a categorical one and Gaussian for a normal one. A GLM with another family is refused. The forward draw of a Gaussian column needs the residual variance of the fit, so a Gaussian covariate accepts a GLM only. A data-adaptive learner for any other model gives the status adaptive_baseline_plugin. See Simulate the parametric g-formula.

Choose ParametricGFormula over SequentialGComputation when the plan fixes a continuous dose, or when you need the forward-simulation form. Choose SequentialGComputation when you doubt a covariate law, because it models no covariate.

Two fold layers#

n_folds defines the outer cross-fitting split used to keep each observation out of the nuisance fit that predicts it. Default 10.
learner_folds defines internal model-selection folds for learners such as a Super Learner. Default 5.
repeats repeats the outer split and reports the median of the draws. Default 1. The value must be at least 1, and a value above 1 requires enabled=True. A repeated fit refuses a simultaneous band, because the band needs the joint influence curves of one draw. Pass simultaneous=False to a repeated fit.
enabled=False sets a one-fold analysis with no cross-fitting.
stratify_by="none" draws unstratified folds, so the split reads neither the treatment nor the outcome. This is the default, and it is the only policy a fit that draws a split accepts.
stratify_by="treatment" and stratify_by="treatment+outcome" are refused when the fit draws folds. A selector-based collaborative fit refuses them at every setting, because its search draws selection folds without cross-fitting. In-sample outcome-adaptive C-TMLE draws no selection folds and accepts an unused policy.

CrossFitting raises MethodConfigurationError when you construct a declaration its own checks refuse. Four inputs earn one: repeats below 1, repeats above 1 with enabled=False, enabled=True with fewer than two folds, and a stratify_by above. The engine raises ValueError with the same reason. Fold and outcome-scale rules gives every message and the audit behind it.

A cross-fitted fit of a continuous outcome needs a declared Targeting(q_bounds=(lower, upper)) equal to the outcome’s known support. With q_bounds=None the scale would come from every observed outcome, held-out rows included. The fit is refused before any learner. Declare the support, or fit in sample with CrossFitting(enabled=False). Never read the interval off the sample.

Because the split reads no treatment, it cannot keep a rare arm in every training complement. The fit checks the realized draw instead, before the first learner. Each arm must appear in at least two independent units, which are rows for iid data and clusters under id=. Each training complement must hold every arm, and both observed outcome classes of a binary outcome. A package classification SuperLearner needs two rows of each class of its target in each complement, which the default library makes the common case.

None of these refusals names a redraw. The remedy is CrossFitting(enabled=False), or more observations at the rare level.

missing-outcome fit	what it needs beyond the rules above	contract table
binary `NaturalCourseMean`, stacked	a binary outcome, pooled targeting, and one repeat	natural-course contracts
arm-indexed means and contrasts, cross-fitted	ordinary TMLE, and no `ATT` or `ATC` target	arm-indexed contract
arm-indexed means and contrasts, per-arm `CTMLE(strategy="oat")`	a respondent in each arm and stratum of each training complement, and a nonrespondent in each complement. The fit reports a point estimate only	per-arm design contract
shift, incremental, regime, MSM, and controlled-direct-effect targets, cross-fitted	no contract. The fit refuses them before the first learner	not written yet

To fit any target in the table in sample, set CrossFitting(enabled=False). One fold balances nothing, so the fold policy makes no difference there.

The two layers multiply: one nuisance fit at the defaults is 10 × 5 model fits per library candidate, before an estimator variant multiplies it again. The examples in this documentation set n_folds=5, learner_folds=3 to stay quick to run, which is not a recommendation for an analysis.

Keep all rows from one declared cluster in the same fold. A grouped draw does that by permuting the distinct cluster labels and cutting them into near-equal parts. Ordinary TMLE, DR-TMLE and longitudinal TMLE accept id= under cross-fitting. C-TMLE refuses id= at every setting, except strategy="derivative". Cross-fitting reduces empirical-process bias. It does not establish positivity, correct identification, or nuisance consistency.

Four fold declarations are refused by name. A blocked temporal split needs a row-level time ordering that no design role carries. A rolling-origin split nests its training sets, so no single fold holds out each row, and Folds stores exactly one fold per row. The rolling-origin refusal would survive a time index; it asks for a different storage contract. The two stratified policies are refused for a third reason, which the fold and outcome-scale rules give.

Reuse an outer split#

Every point-treatment result records its realized outer folds as an immutable split_plan. Most point-treatment methods accept that plan to compare methods on the same validation rows.

A fit accepts a plan only when the plan records how random_partition drew each repeat. Before the first learner runs, the fit draws each repeat again from that record and compares the labels. A plan whose labels no recorded draw produces could have been chosen by reading the outcome, and the fit refuses it. Two plans carry the record: the plan SplitPlan.from_folds copies off random_partition draws, and result.split_plan from a cross-fitted fit. A plan you build from labels alone carries no record, and neither does the one-fold plan of an in-sample fit.

from cleverly import SplitPlan
from cleverly.learners import random_partition

plan = SplitPlan.from_folds([random_partition(result.data.n, 5, seed=3)])
reused_method = TMLEMethod(
    models=method.models,
    cross_fitting=CrossFitting(
        n_folds=5,
        learner_folds=3,
        repeats=1,
        split_plan=plan,
    ),
    targeting=method.targeting,
    inference=method.inference,
    runtime=method.runtime,
)
reused_result = effect.estimate(method=reused_method)
assert reused_result.provenance.fold_fingerprint == plan.fingerprint

Set repeats to the repeat count the plan records. Set n_folds to the count the plan was declared under, which can exceed the plan’s own fold count when the data capped it. A declaration the plan cannot serve raises MethodConfigurationError instead of changing the plan.

A plan read off a result is bound to the rows that produced it, by position. Reuse it on those rows, in that order. Call plan.unbound() to reuse the labels on other rows deliberately. The method keeps the generator record and drops the binding.

The two stacked missing-outcome contracts do not accept split_plan=. They cover the binary NaturalCourseMean and the cross-fitted arm-indexed means and contrasts. Each audited contract covers package-generated near-balanced folds only. The result still records the realized plan for provenance. The natural-course table and the arm-indexed table list the other refusals.

Reusable outer split plans states the whole contract: the counts, the row binding, what validation checks, and every refusal.

Targeting and bounds#

g_bounds="auto" chooses target-aware treatment-mechanism truncation. The bound changes the finite-sample estimating procedure and should be reported with support diagnostics. The logistic submodel bound, submodel_alpha, is separate from the confidence interval’s alpha.

Four Targeting options choose how the targeting step runs. Each default runs the shipped step.

option	what it chooses
`stopping="scaled_var"` or `"sample_size"`	the `tmle3` rule that decides the score equation is solved. The result prints the score residual in standard errors
`step_bound=` and `step_search=`	the longest step of one iteration, and whether a line search sets its length
`iterate="best"`	the iterate of smallest score norm, in place of the last

ModelSpec(outcome_by_arm=True) fits one outcome regression for each treatment arm. CollaborativeTMLEMethod(patience=p) stops the scoring of the candidate path early. PointTreatment(treatment_adjustment=[...]) lets the treatment mechanism read a subset of the adjustment set. That declaration restricts the statistical model. Stopping rules, step bound and best iterate and treatment-mechanism covariates state the contracts.

Unknown settings and settings an engine cannot use raise MethodConfigurationError before fitting rather than being ignored. A longitudinal fit refuses a point-only control such as repeats= instead of accepting it and discarding it.