CV-TMLE: flexible learners and navigator teams#

The situation#

A regional network offers transition navigation to adults who go home from hospital. At the discharge-home order, staff record four baseline variables. A patient then receives the navigation offer or usual discharge support. Thirty days later, the patient reports a transition score. Navigator teams deliver the offer.

item	in this program
unit	one discharge home of an adult patient
time zero	the discharge-home order
baseline covariates \(W\), measured before time zero	`discharge_risk`, `prior_utilization`, `medication_burden`, `age`
treatment \(A\), set at time zero	`transition_navigation`: 1 for the offer, 0 for usual support
outcome \(Y\), measured at day 30	`transition_score`, a share of the maximum score
counterfactual outcomes	\(Y^1\), the score if the patient receives the offer, and \(Y^0\), the score under usual support. The data show at most one of them for each patient

Staff do not assign the offer at random. Patients with different baseline variables receive the offer at different rates.

The data are synthetic, so each law has a known true effect. The page compares estimates with that truth only to teach. A real program has no truth. Steps 2 to 6 use a law with no teams, which isolates one problem. Step 7 uses a second law with teams.

The causal graph#

          ┌──────────────────────┐
          │                      ▼
          W ────────► A ───────► Y ◄──── U  (navigator team: Step 7 law only)
                             ▲           │
                             └───────────┘  U changes the size of the A → Y effect

node or edge	what it asserts	what the analysis does with it
\(W \to A\) and \(W \to Y\)	each baseline variable changes the chance of an offer and the score	the four variables are confounders, so the analysis adjusts for all four
\(A \to Y\)	the offer can change the score	the estimand measures this effect
\(U \to Y\), and \(U\) modifies \(A \to Y\)	a team’s unrecorded practice shifts the score and the benefit of the offer	\(U\) is an effect modifier that the patients of one team share. Their outcomes are then dependent, so Step 7 makes the team the independent unit
no \(U \to A\)	team practice does not change who receives an offer	\(U\) is not a confounder, so adjustment for \(W\) still identifies the effect. With this arrow, \(U\) would confound, and no cluster declaration removes that bias
no arrow from one patient’s \(A\) to another patient’s \(Y\)	one patient’s offer does not change another patient’s score	this is the no-interference assumption
no hidden common cause of \(A\) and \(Y\)	the four recorded variables block every back-door path	this is the no-unmeasured-confounding assumption. Data cannot verify it. Step 10 measures how strong a violation must be to change the answer

Readmission and death occur after time zero, so they are not in \(W\). Adjustment for a post-treatment variable would remove part of the effect. The protocol uses the score regardless of readmission, and it scores death before day 30 as the worst score. Every patient in these data reports a score, so the graph has no censoring node.

The decisions on this page#

decision the problem forces	the step that makes it
which estimand answers the sponsor, and which assumptions identify it	the question and the estimand
whether flexible learners need cross-fitting, and what it leaves unsolved	why this method, Steps 5 and 6
which unit is independent when patients share navigator teams	Step 7
how to report an estimate that depends on a random fold split	Step 8
what the diagnostics and the sensitivity analysis can and cannot check	Steps 9 and 10

The question and the estimand#

The program sponsor asks how much the mean 30-day transition score would change if every eligible discharge received the offer, compared with usual support for every discharge. The answer decides whether the network extends the offer to all eligible patients.

part	definition
target population	adults with a discharge-home order at a participating hospital
intervention	the offer for every eligible discharge, against usual support for every eligible discharge
counterfactual outcome	\(Y^a\), the 30-day score under strategy \(a\)
estimand	the average treatment effect (ATE), \(\psi = E[Y^1] - E[Y^0]\)

An estimand is the number the question asks for. Write it before you choose any model. Estimands lists the other questions cleverly can ask. Simpler quantities answer different questions.

simpler quantity	why it does not answer the sponsor
the mean score of offered patients minus the mean score of the others	the two groups differ in \(W\), so the difference mixes the effect with the difference in baseline variables
the effect among patients who received the offer	it describes the patients whom staff selected, not a network-wide offer
the effect in one team	it describes one team. Step 7 shows that teams differ

The four assumptions in the next table identify the ATE. Then \(\psi = E_W[\bar Q(1, W) - \bar Q(0, W)]\), where \(\bar Q(a, W) = E[Y \mid A = a, W]\). Every estimator on this page targets this one quantity.

assumption	what it requires here	what can check it
consistency	the observed score of an offered patient is \(Y^1\), so the offer has one version	scientific judgment. The protocol records the version
no interference	one patient’s offer does not change another patient’s score	scientific judgment. The protocol cites reserved navigator capacity
no unmeasured confounding	\(Y^a\) is independent of \(A\) given \(W\)	scientific judgment. Step 10 measures sensitivity to a violation
positivity	every kind of patient has a nonzero chance of each arm	the data, in part. Step 9 describes the fitted propensity model, not the population

Why this method#

feature of the problem	method choice	what the choice still needs
one decision at time zero, one outcome at a fixed horizon, no censoring	point-treatment TMLE. Longitudinal and survival methods do not apply	the four assumptions above
a static contrast of everyone offered against no one offered	the ATE. A stochastic or shift intervention would answer a different policy question	positivity in both arms
\(\bar Q\) and the propensity \(g(W) = P(A = 1 \mid W)\) are nonlinear, with interactions	flexible learners for both nuisances	the product of the two nuisance errors must shrink faster than \(1/\sqrt{n}\)
flexible learners can fail the complexity condition of in-sample proofs	cross-fitting, which is CV-TMLE	one nuisance fit for each fold
a score between 0 and 1	the declared support `q_bounds=(0.0, 1.0)`	a cross-fitted continuous outcome needs it
patients share navigator teams, and team practice modifies the effect	the `cluster=` role: whole teams in each fold, and team totals in the variance	enough teams. Step 7 states the rule

No outcome is missing, so the page uses no missing-outcome method. The question has one binary contrast, so the page fits no marginal structural model.

An interval proof that reuses observations controls an empirical-process term with a complexity condition, such as a Donsker condition. Boosted trees need not satisfy it, even when they predict well. With cross-fitting, each patient’s nuisance prediction comes from a model that never saw that patient. That removes the complexity condition. It does not repair a slow learner, poor support, or confounding. The CV-TMLE reference lists the conditions that remain.

term	plain meaning
nuisance	a model the estimate needs but the question does not ask about. Here, \(\bar Q\) and \(g\). Point-treatment TMLE defines both
targeting	a small update to the outcome model, weighted by the treatment model, that removes first-order bias. Methods and learners defines it
outer fold	one part of the sample. The nuisance models for its rows train on the other parts
influence curve	how much each row moves the estimate. Its variance gives the standard error. Inference defines it
cluster	a group of rows that are not independent of each other. Here, one navigator team

Step 1: set up#

Every fit below passes its learners, fold count, and seed explicitly, so a rerun reproduces the stored outputs.

import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, HistGradientBoostingRegressor
from sklearn.linear_model import LogisticRegression

import cleverly

pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 20)
print("cleverly", cleverly.__version__)

cleverly 0.1.3

What this output tells you. The stored outputs in this notebook came from the version named above. A different version can print different numbers.

Step 2: the data#

navigation_data draws the rows from the generator make_nonlinear_bounded under the program’s column names. The code prints the column names, the first rows, and the true ATE.

from cleverly.datasets import navigation_data

frame, truth = navigation_data(n=3_000, seed=34)
print("rows and columns:", frame.shape)
print(list(frame.columns))
print(frame.head().round(3))
print()
print(f"population ATE: {truth['ate']:.3f}")

rows and columns: (3000, 6)
['transition_score', 'transition_navigation', 'discharge_risk', 'prior_utilization', 'medication_burden', 'age']
   transition_score  transition_navigation  discharge_risk  prior_utilization  medication_burden    age
0             0.454                    0.0          -0.040             -1.258              2.574  0.482
1             0.791                    1.0           0.644             -0.208              0.058  0.337
2             0.735                    1.0           0.195             -0.609              0.615  0.490
3             0.295                    0.0          -0.118             -0.719             -0.879 -1.008
4             0.642                    1.0           0.277             -0.351              0.871 -0.888

population ATE: 0.163

What this output tells you. Each row is one discharge. The four covariates are independent standard-normal draws.

feature of the law	what it means for this page
\(\bar Q\) and \(g\) are nonlinear, and the effect varies with `discharge_risk` and `prior_utilization`	a main-terms model cannot represent either nuisance, so the page uses flexible learners
the true propensity lies between 5% and 95%	every patient has at least a 5% chance of each arm
the score lies between 0 and 1	the support belongs to the instrument. Step 5 declares it
the law draws each patient independently, with no team	Steps 5 and 6 isolate the data-reuse problem. Step 7 uses a second law with teams

The true ATE is 0.163.

Step 3: write the protocol#

This page asks the question of the shared study design. It changes only the estimator, so the code uses navigation_protocol() unchanged.

from cleverly.datasets import navigation_protocol

protocol = navigation_protocol()
print("\n".join(protocol.summary_lines()))

causal study protocol: schema 1; 623fc2c240615d26
target population: Adults with a discharge-home order at a participating hospital during the enrollment period
eligibility: ['Age 18 years or older', 'Discharge home ordered at a participating hospital']
time zero: Discharge-home order, after baseline measurement and before the navigation offer
treatment strategies: ['Offer standard transition navigation', 'Provide usual discharge support']
treatment versions: ['Bedside transition plan and two scheduled navigator contacts within 30 days', 'No access to the transition-navigation offer']
outcome: Patient-reported transition score, as a share of the maximum score
horizon: 30 days after discharge
intercurrent-event handling: ['Use the transition score regardless of readmission', 'Analyze the offer regardless of completed contacts', 'The protocol scores death before day 30 as the worst transition score (composite strategy)']
interference unit: Individual patient
assumption rationale: ['The recorded baseline variables cover the measured common causes', 'The standardized offer and version records support consistency', 'Reserved navigator capacity and access controls support no interference']

What this output tells you. The fingerprint is 623fc2c240615d26. The point-treatment tutorial prints the same fingerprint, because the two pages record the same design.

The interference unit is the individual patient. A shared team is a statistical dependence, not interference. StudyProtocol has no field for that dependence, so Step 7 declares it through the design’s cluster= role. A cluster declaration cannot repair interference. If patients compete for navigator slots, one patient’s assignment changes another patient’s outcome, and the estimand itself changes.

Step 4: design and identification#

The design names the column roles, and the estimand names the contrast.

from cleverly import ATE, CausalStudy, PointTreatment

study = CausalStudy(
    frame,
    design=PointTreatment(
        outcome="transition_score",
        treatment="transition_navigation",
        adjustment=("discharge_risk", "prior_utilization", "medication_burden", "age"),
    ),
    protocol=protocol,
)
effect = study.identify(ATE(reference=0))
print(effect.summary(protocol="fingerprint"))

average treatment effect, E[Y^a] - E[Y^reference]
identified by explicit-adjustment: E_W[E(transition_score | transition_navigation=a, W)] - E_W[E(transition_score | transition_navigation=0, W)] for a in [1]
adjustment/history: ['discharge_risk', 'prior_utilization', 'medication_burden', 'age']
required nuisances: ['outcome_regression', 'treatment_mechanism']
assumptions:
  - consistency: Y = Y^a when A = a
  - no interference: one unit's potential outcome does not depend on other units' treatment assignments
  - no unmeasured confounding: Y^a is independent of A given W
  - positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]
causal study protocol: schema 1; 623fc2c240615d26

What this output tells you. The summary prints the observed-data formula of the question section, the two required nuisances, and the four assumptions. Nothing in it names folds or learners. Those belong to the method, so a change of estimator cannot change the question.

Step 5: estimate with cross-fitting#

setting	value	why
`outcome_learner`	gradient boosting	\(\bar Q\) is nonlinear, with interactions
`treatment_learner`	a shallow, regularized booster	\(g\) is nonlinear. Depth 2, a small learning rate, and an L2 penalty keep the fitted propensities from being too extreme
`CrossFitting(n_folds=5)`	five outer folds	each row’s predictions come from models fit on the other four folds. The package default is ten. Five keeps this page fast
`Targeting(q_bounds=(0.0, 1.0))`	the declared score support	fixes the outcome scale before any fold is drawn
`Runtime(random_state=34, n_jobs=1)`	a fixed seed, one process	makes the fold draw and the fit reproducible

Declare q_bounds for a cross-fitted continuous outcome. Without it, the outcome scale would come from all observed scores. Each fold’s model would then use a scale that the held-out rows helped set. cleverly refuses that fit. The name declared_support carries one support to every fit below.

Do not use the default HistGradientBoostingClassifier as a treatment learner below 10,000 rows. Below that size, scikit-learn does not stop it early. The rules for code cells give the consequence.

from cleverly import CrossFitting, ModelSpec, Runtime, Targeting, TMLEMethod


def shallow_booster():
    """A shallow, regularized boosted propensity learner."""
    return HistGradientBoostingClassifier(
        max_depth=2,
        learning_rate=0.05,
        max_iter=200,
        l2_regularization=1.0,
        random_state=34,
    )


boosted = ModelSpec(
    outcome_learner=HistGradientBoostingRegressor(random_state=34),
    treatment_learner=shallow_booster(),
)
declared_support = Targeting(q_bounds=(0.0, 1.0))
cross_fitted = effect.estimate(
    method=TMLEMethod(
        models=boosted,
        cross_fitting=CrossFitting(n_folds=5),
        targeting=declared_support,
        runtime=Runtime(random_state=34, n_jobs=1),
    )
)
print(cross_fitted.summary(protocol="fingerprint"))
print()
print(f"population ATE: {truth['ate']:.3f}")

Targeted maximum likelihood estimation
======================================
n = 3000; covariates = 4; P(A=1) = 0.4597
causal estimand: average treatment effect, E[Y^a] - E[Y^reference]
identification: explicit-adjustment; E_W[E(transition_score | transition_navigation=a, W)] - E_W[E(transition_score | transition_navigation=0, W)] for a in [1]
required nuisances: outcome_regression, treatment_mechanism
identification assumptions: consistency: Y = Y^a when A = a; no interference: one unit's potential outcome does not depend on other units' treatment assignments; no unmeasured confounding: Y^a is independent of A given W; positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]
causal study protocol: schema 1; 623fc2c240615d26
outcome family: gaussian; fluctuation: logistic (iterative)
stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled
propensity truncated to [0.0114, 0.9886]; ATT/ATC to [0.025, 0.975]
data e25f7a021cacfaa6 | folds c7f4eafaaf319249 | cleverly 0.1.3 | 2026-10-16T01:16:49+00:00

estimand  psi      std_err   95% CI              p_value
--------  -------  --------  ------------------  -------
ate       0.15731  0.005444  [0.14664, 0.16798]  <1e-4  

population ATE: 0.163

What this output tells you. The construction line reads stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled. The fit cross-fits both nuisances and then fits one targeting update on all rows.

The line propensity truncated to [0.0114, 0.9886] gives the default bound, \(5 / (\sqrt{n} \ln n)\) at \(n = 3000\). Step 9 shows that no unit reaches it.

The estimate is 0.15731, with a standard error of 0.005444. The 95% interval [0.14664, 0.16798] contains the true ATE of 0.163 on this draw.

The Step 2 law draws patients independently, so this interval treats patients as independent. Step 7 shows the declaration that a program with teams needs.

A review probe, reviews/notebook-review/probes/cross-fitting-final/sweep.py, repeats this fit on 200 draws of the same law. The interval contained the true ATE on 188 of them. The mean standard error was 0.93 times the standard deviation of the estimates. The probe is not a registered study.

Step 6: compare with in-sample nuisance fits#

This comparison shows why the page cross-fits. The second fit uses the same learners and the same declared support, with CrossFitting(enabled=False). That selects ordinary TMLE.

item	in-sample TMLE	CV-TMLE
estimand	the ATE of Step 4	the same ATE
what the interval needs beyond the four assumptions	the product-rate condition, and a complexity condition on the learners	the product-rate condition

The expected mechanism is overfitting. The boosted \(\bar Q\) fits the rows that it later predicts. Its residuals on those rows are then too small, so the influence-curve variance is too small. Data reuse can also add a bias term that the targeting step does not remove.

The code prints both fits, whether each interval contains the true ATE, and the distance from the truth in each fit’s own standard errors.

in_sample = effect.estimate(
    method=TMLEMethod(
        models=boosted,
        cross_fitting=CrossFitting(enabled=False),
        targeting=declared_support,
        runtime=Runtime(random_state=34, n_jobs=1),
    )
)


def summary_line(result, marker):
    """The first line of the result summary that contains ``marker``."""
    return next(line.strip() for line in result.summary().splitlines() if marker in line)


def side_by_side(fits, target=None, columns=("psi", "std_err", "ci_lower", "ci_upper")):
    """One row per labelled fit, and whether each interval contains ``target`` when given."""
    table = pd.concat({label: fitted.to_frame() for label, fitted in fits.items()})
    table = table.droplevel(1)[list(columns)]
    if target is not None:
        table["covers truth"] = table["ci_lower"].le(target) & table["ci_upper"].ge(target)
    return table


print(summary_line(in_sample, ": nuisances "))
comparison = side_by_side(
    {"no cross-fitting": in_sample, "cross-fitted": cross_fitted}, truth["ate"]
)
print(comparison.round(4))
se_ratio = in_sample["ate"].std_error / cross_fitted["ate"].std_error
print(f"in-sample SE / cross-fitted SE: {se_ratio:.2f}")
for label, fitted in (("in-sample", in_sample), ("cross-fitted", cross_fitted)):
    miss = abs(truth["ate"] - fitted["ate"].psi) / fitted["ate"].std_error
    print(f"{label} distance from the truth, in its standard errors: {miss:.1f}")
print(f"population ATE: {truth['ate']:.3f}")

TMLE (in-sample nuisances): nuisances fitted in-sample (cross_fit=False)
                     psi  std_err  ci_lower  ci_upper  covers truth
no cross-fitting  0.1553   0.0040    0.1476    0.1631          True
cross-fitted      0.1573   0.0054    0.1466    0.1680          True
in-sample SE / cross-fitted SE: 0.73
in-sample distance from the truth, in its standard errors: 1.9
cross-fitted distance from the truth, in its standard errors: 1.0
population ATE: 0.163

What this output tells you. The first line confirms that the new fit used in-sample nuisances. On this draw, the in-sample standard error is 0.73 times the cross-fitted one. Both intervals contain the true ATE. The truth lies 1.9 in-sample standard errors from the in-sample estimate, and 1.0 cross-fitted standard errors from the cross-fitted estimate.

One draw cannot show a coverage rate. The review probe repeats both fits on 200 draws of the same law.

over 200 draws	in-sample TMLE	CV-TMLE
intervals that contained the true ATE	153	188
mean standard error divided by the standard deviation of the estimates	0.70	0.93
mean estimate minus the true ATE	-0.0024	0.0003

The in-sample interval missed on about one draw in four. A calibrated 95% interval misses on about one draw in 20. The probe shows both parts of the mechanism. The in-sample standard error is too small, and the in-sample estimate sits slightly low. On 90% of the draws, the in-sample standard error was between 0.68 and 0.73 times the cross-fitted one.

The registered stacked CV-TMLE study fits this same law 400 times at \(n = 500\), with an unpruned regression tree for \(\bar Q\). Its in-sample control covers 0.5225 of the time, and its cross-fitted fit covers 0.945. Neither the study nor the probe shows coverage for other learners or other laws.

Step 7: declare navigator teams#

The program has 200 navigator teams with 15 patients each. Team practice, such as supervision and local follow-up, changes how much a patient gains from the offer. The graph shows this as \(U\).

make_clustered draws a second synthetic law with this structure. It has its own two covariates, team_law_w1 and team_law_w2, and a binary outcome: 1 when a patient reports the top box of the transition score. The estimand of this step is therefore the ATE on the top-box outcome, a risk difference in this teaching law. It is not the Step 2 ATE on the score. In this law, the team factor does not change who receives an offer. About 46% of teams have a negative mean effect of the offer (review probe reviews/notebook-review/probes/cross-fitting-final/truth.py).

item	patients as units	teams as units
estimand	the top-box risk difference in the population of teams and their patients	the same risk difference
folds	patients split at random	whole teams in each fold
variance	the variance of patient influence values	the variance of team totals of the influence values, across teams

The expected mechanism is shared effect modification. Patients of one team share \(U\), so their influence values move together. A variance that treats patients as independent then understates the uncertainty.

Both fits cross-fit, with logistic learners for both nuisances. The true propensity of this law is a main-terms logistic model in the two covariates. The fitted \(\bar Q\) is misspecified, because the unrecorded \(U\) enters the true outcome model through an interaction with \(A\). The estimate therefore relies on the correct propensity model. The outcome is binary, so its support is known and needs no declaration. The protocol changes only the outcome field, and the code prints the new fingerprint.

from dataclasses import replace

from cleverly.datasets import make_clustered

team_frame, team_truth = make_clustered(n=3_000, seed=34, cluster_size=15, family="binomial")
team_frame = team_frame.rename(
    columns={
        "Y": "transition_top_box",
        "A": "transition_navigation",
        "W1": "team_law_w1",
        "W2": "team_law_w2",
        "cluster": "navigator_team",
    }
)
team_protocol = replace(protocol, outcome="Top-box patient-reported transition score")
simple = TMLEMethod(
    models=ModelSpec(
        outcome_learner=LogisticRegression(max_iter=1000, random_state=34),
        treatment_learner=LogisticRegression(max_iter=1000, random_state=34),
    ),
    cross_fitting=CrossFitting(n_folds=5),
    runtime=Runtime(random_state=34, n_jobs=1),
)
patients_design = PointTreatment(
    outcome="transition_top_box",
    treatment="transition_navigation",
    adjustment=("team_law_w1", "team_law_w2"),
)
teams_design = replace(patients_design, cluster="navigator_team")

ignoring = (
    CausalStudy(team_frame, design=patients_design, protocol=team_protocol)
    .identify(ATE(reference=0))
    .estimate(method=simple)
)
clustered = (
    CausalStudy(team_frame, design=teams_design, protocol=team_protocol)
    .identify(ATE(reference=0))
    .estimate(method=simple)
)


def folds_spanned(result):
    """The largest number of outer folds that one navigator team spans in ``result``."""
    labels = pd.Series(result.split_plan.assignments[0], index=team_frame.index)
    return labels.groupby(team_frame["navigator_team"]).nunique().max()


team_comparison = side_by_side(
    {"patients as units": ignoring, "teams as units": clustered}, team_truth["ate"]
)
team_comparison["folds one team spans"] = [folds_spanned(ignoring), folds_spanned(clustered)]
print(team_comparison.round(4))
team_se_ratio = clustered["ate"].std_error / ignoring["ate"].std_error
print(f"teams SE / patients SE: {team_se_ratio:.2f}")
print("navigator teams:", clustered.data.n_clusters)
print(f"population ATE: {team_truth['ate']:.3f}")
print("team protocol fingerprint:", team_protocol.fingerprint)
print("team protocol outcome:", team_protocol.outcome)

                      psi  std_err  ci_lower  ci_upper  covers truth  folds one team spans
patients as units  0.1237   0.0195    0.0856    0.1619          True                     5
teams as units     0.1256   0.0301    0.0667    0.1845          True                     1
teams SE / patients SE: 1.54
navigator teams: 200
population ATE: 0.104
team protocol fingerprint: 7df34d840b4e8d1a
team protocol outcome: Top-box patient-reported transition score

What this output tells you. The two point estimates are 0.1237 and 0.1256. They differ only because the team declaration regroups the folds. Declaring teams makes the standard error 1.54 times larger on this draw. Both intervals contain the true ATE of 0.104 on this draw. The team protocol’s fingerprint is 7df34d840b4e8d1a, and its outcome field names the top-box score.

The fold count shows the other change. Without the declaration, one team spans 5 outer folds. A patient’s nuisance model can then train on outcomes from the patient’s own team. Those training rows are not independent of the predicted row, which is the independence that cross-fitting uses. With it, each team spans 1 fold.

Both intervals target the same population risk difference. The patient-level variance ignores the variation of the effect between teams, so it understates the uncertainty. The registered clustered CV-TMLE study fits this law with teams of 10. Its independent-row variance covers the ATE 0.84125 of the time, and its cluster-robust variance covers 0.9475. The design effect grows with team size, so the independent-row coverage for teams of 15 is not the study’s number.

The cluster variance includes the factor \(m/(m-1)\) for \(m\) teams. Clusters gives the rules for the number of teams.

number of teams \(m\)	what the fit reports
40 or more	a normal-reference interval
10 to 39	a \(t\)-reference interval. Clusters gives its degrees of freedom
fewer than 10	a point estimate only
fewer than the requested folds	fewer folds, with a warning

This fit has 200 teams, so it reports a normal-reference interval.

Step 8: the fold draw#

The fold split is random, and a different split gives a slightly different estimate. Report that dependence rather than pick a convenient split. repeats= runs the complete estimator on several fold draws. It reports the median estimate, and its variance includes the spread between draws.

repeated = effect.estimate(
    method=TMLEMethod(
        models=boosted,
        cross_fitting=CrossFitting(n_folds=5, repeats=3),
        targeting=declared_support,
        runtime=Runtime(random_state=34, n_jobs=1),
    )
)
print(summary_line(repeated, "median over"))
print(side_by_side({"median of 3 draws": repeated}).round(4))
repeated_nuisance = repeated.diagnostics.run_all().report("nuisance_models")
print(repeated_nuisance.repeat_spread_frame().round(4).to_string(index=False))

(median over 3 independent draws of the split; variance includes within-draw uncertainty and split dispersion)
                      psi  std_err  ci_lower  ci_upper
median of 3 draws  0.1567   0.0055    0.1459    0.1674
estimand  n_repeats  standard_deviation  reported_standard_error  ratio_to_standard_error
     ate          3              0.0004                   0.0055                   0.0679

What this output tells you. The median over 3 draws is 0.1567, with a standard error of 0.0055. The single Step 5 draw had a standard error of 0.005444.

The spread row gives a standard deviation of 0.0004 across the three draws, which is 0.0679 of the reported standard error. On this sample, the fold draw moves the estimate much less than the sampling uncertainty does. Three draws estimate that spread imprecisely, and the report defines no threshold for the ratio. The repeated CV-TMLE evidence gives the repeated-sampling results for the median report.

Step 9: diagnostics#

The assessment of the Step 5 fit lists the checks that ran and the rows that need attention. The code then prints the support, nuisance-model, and score-equation reports. The last line gives the support detail of the in-sample fit of Step 6.

assessment = cross_fitted.assess()
print(assessment.summary())
print("needs attention:", tuple(item.name for item in assessment.attention))
support = assessment.report("support")
nuisance = assessment.report("nuisance_models")
scores = assessment.report("score_equations")
print()
print(support.summary())
print()
print(nuisance.summary())
print()
print(scores.summary())
print()
smallest = support.propensity_quantiles["overall"][0.0]
print(f"smallest estimated g(W), before truncation: {smallest:.4f}")
truncated = support.truncated
print(f"truncated: {truncated['count']:.0f} units ({truncated['fraction']:.2%})")
for arm, ess in support.effective_sample_size.items():
    ratio = f"{ess['effective']:.1f} ({ess['ratio']:.1%})"
    print(f"{arm}: {ess['n']:.0f} rows, effective sample size {ratio}")
print()
print("in-sample support:", in_sample.diagnostics.run_all()["support"].detail)

Returned results
----------------
surface      operation            result                                                                                                                                                                                    
-----------  -------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
validation   support              maximum truncated fraction 0.0%; minimum effective-sample-size ratio 82.5%; group load: mean:h1 1137.7/3000 Kish-equivalent mask rows (37.9%; 37.9% all; draw 01 of 01); not estimator ESS
validation   nuisance_models      2 nuisance model report(s) are available                                                                                                                                                  
sensitivity  omitted_confounding  at the default strengths; cf_y=0.03, cf_d=0.03, rho=1; bias-adjusted interval [0.1486, 0.166]                                                                                             
sensitivity  robustness_value     point robustness value 0.4187; confidence-limit value 0.398                                                                                                                               
sensitivity  elements             sigma2=0.01809, nu2=4.536, max_bias=0.2865                                                                                                                                                
sensitivity  contour              grid 20 x 20; cf_d range [0, 0.15]; cf_y range [0, 0.15]; value range [0.1107, 0.1573]; inspect the retained frame                                                                        
sensitivity  evalue               point=3.092, limit=2.936, source scale=mean difference; approximate conversion                                                                                                            

Checks
------
status  count  operations                
------  -----  --------------------------
passed  1      validation.score_equations

Not run
-------
status          count  operations                       
--------------  -----  ---------------------------------
deferred        4      diagnostics.truncation_curve     
                       diagnostics.refute               
                       sensitivity.benchmark            
                       sensitivity.simulated_confounding
not_applicable  3      diagnostics.corrections          
                       sensitivity.missingness          
                       sensitivity.tipping_gamma        

Full ledger: call to_frame(). Next steps: call next_steps().
Retained payloads: call report(...).
needs attention: ()

Positivity / overlap diagnostics
--------------------------------
n = 3000; propensity truncated to [0.0114, 0.9886]

g(W) quantile  0%      1%      5%      25%     50%     75%     95%     99%     100%  
-------------  ------  ------  ------  ------  ------  ------  ------  ------  ------
overall        0.0239  0.1185  0.1930  0.3521  0.4663  0.5683  0.7164  0.7987  0.8649
treated        0.0579  0.1711  0.2548  0.4145  0.5193  0.6091  0.7649  0.8107  0.8649
control        0.0239  0.0954  0.1662  0.3095  0.4227  0.5242  0.6543  0.7703  0.8510

arm      n     effective n  ESS / n  top 1% weight  top 5% weight
-------  ----  -----------  -------  -------------  -------------
treated  1379  1137.7       0.825    0.037          0.122        
control  1621  1454.9       0.898    0.030          0.102        

threshold  P(g < t)  P(g > 1-t)
---------  --------  ----------
0.01       0.0000    0.0000    
0.025      0.0003    0.0000    
0.05       0.0010    0.0000    
0.1        0.0063    0.0000    

truncated: 0 unit(s) (0.00%); most extreme untruncated g(W) = 0.02391

group  equation  target rows  Kish-equivalent rows  Kish / target  Kish / total  top 1% load  top 5% load  max |w h|  zero load  g bound           clipped  
-----  --------  -----------  --------------------  -------------  ------------  -----------  -----------  ---------  ---------  ----------------  ---------
mean   h1        3000/3000    1137.7                0.379          0.379         0.066        0.214        17.28      1621       [0.0114, 0.9886]  0 (0.00%)
(absolute residual-multiplier load from the fitted equation; Kish values describe residual-multiplier concentration, not residual contributions or estimator information, influence, or precision; target rows are score-mask rows and include structural zeros)
max |clever covariate| (mean): 17.28

VERDICT: no truncation-driven fragility detected (0.0% of units clipped at the marginal treatment-mechanism bound [0.0114, 0.9886]). The narrowest arm has a Kish-equivalent weight count of 83% of its nominal rows. This is concentration, not estimator information, and no threshold is applied because none is derived. Absolute-load concentration is greatest for group 'mean', equation 'h1': 1137.7 Kish-equivalent rows out of 3000 score-mask rows (38%; 38% of 3000 total rows). This is a descriptive residual-multiplier concentration index. It excludes the residual and is not estimator effective sample size, information, actual influence, precision, or a positivity test.

Nuisance model diagnostics
----------------------------------------
model       auc     brier   log_loss  r2      mse     cal_slope  cal_se  reg_slope
----------  ------  ------  --------  ------  ------  ---------  ------  ---------
propensity  0.6737  0.2259  0.6427    -       -       0.9611     0.0624  -        
outcome     -       -       -         0.6653  0.0181  -          -       0.9650   
Calibration slope: logistic recalibration of the out-of-fold predictions with one intercept per fold.
A finding needs the 95.0% interval to lie above 0 and exclude 1, a Bonferroni level over 1 tested model(s).

VERDICT: nuisance fits look reasonable.

Score-equation check
--------------------
target  kind             |score|    before     threshold  ratio     ok 
------  ---------------  ---------  ---------  ---------  --------  ---
mean    fluctuation      0.000e+00  4.084e-03  9.939e-08  0.00e+00  yes
ate     influence curve  0.000e+00  -          9.939e-08  0.00e+00  yes

PASS: the targeting step solved the estimated efficient score equation.

smallest estimated g(W), before truncation: 0.0239
truncated: 0 units (0.00%)
treated: 1379 rows, effective sample size 1137.7 (82.5%)
control: 1621 rows, effective sample size 1454.9 (89.8%)

in-sample support: maximum truncated fraction 0.0%; minimum effective-sample-size ratio 89.6%; group load: mean:h1 1235.0/3000 Kish-equivalent mask rows (41.2%; 41.2% all; draw 01 of 01); not estimator ESS

What this output tells you. The Returned results rows are calculations that ran, not passes.

output part	what it shows on this draw
`Checks` and `needs attention`	the score-equation check passed, and no row needs attention
nuisance model diagnostics	the out-of-fold calibration slope of the propensity is 0.9611
support fields	the smallest fitted \(g(W)\) is 0.0239. No unit sits at the truncation bound. The treated arm keeps an effective-sample-size ratio of 82.5%
in-sample support	the in-sample fit reports a minimum effective-sample-size ratio of 89.6%

The calibration slope comes from a logistic recalibration of the out-of-fold propensities. A slope of 1 means that their spread matches the observed offer rates. On 90% of the probe draws, it lay between 0.90 and 1.01. The effective-sample-size ratio is the share of rows that the weights are worth. cleverly derives no threshold for it, so the report gives 82.5% no grade.

These reports describe the fitted propensity model, not the treatment law. The fitted model puts a share of 0.0063 of the rows below 0.1, in the P(g < t) column. In the law, 1.9% of patients have a true propensity below 0.1 (review probe truth.py). The fitted share fell below the law’s share on 196 of the 200 probe draws. A calibration slope near 1 does not show that the model sees the tail.

The in-sample fit reports more even weights, from the same assignment law. Its minimum ratio was higher on all 200 probe draws. The overfitted propensities sit closer to the observed arms. A better in-sample diagnostic is therefore no reason to prefer the in-sample fit.

Step 10: sensitivity to unmeasured confounding#

No fold construction can check the no-unmeasured-confounding assumption. The omitted-variable analysis asks how strong a hidden confounder would have to be to move the estimate to zero.

term	plain meaning
`cf_y`	the share of the remaining outcome variation that a hidden confounder explains
`cf_d`	the share of the Riesz representer’s second moment that only the hidden confounder adds
`rho`	how closely the confounder’s two effects align. `rho=1` is the worst case
robustness value	the equal `cf_y` and `cf_d` that moves the point estimate to zero at `rho=1`
\(\nu^2\)	the second moment of the Riesz representer. For the ATE it is \(E[1 / \{g(1 - g)\}]\)

The bound scales with \(\sqrt{\sigma^2 \nu^2}\), where \(\sigma^2\) is the residual outcome variance. \(\nu^2\) is finite because the true propensity lies between 5% and 95%. In this law it is about 4.83 (review probe truth.py). The code uses the default doubly robust estimate of \(\nu^2\) and the default strengths, cf_y = cf_d = 0.03. Those strengths are not calibrated to this program. The point-treatment tutorial calibrates them against a recorded covariate.

sensitivity = cross_fitted.sensitivity
elements = sensitivity.elements(estimand="ate")
robustness = sensitivity.robustness_value()
bounds = sensitivity.omitted_confounding()
print(f"sigma^2 (residual outcome variance): {elements.sigma2:.6f}")
print(f"nu^2 (doubly robust):                {elements.nu2:.4f}")
print(f"robustness value:                    {robustness['rv']:.3f}")
print(f"robustness value at the 95% limit:   {robustness['rva']:.3f}")
print()
print(bounds.summary())

sigma^2 (residual outcome variance): 0.018093
nu^2 (doubly robust):                4.5363
robustness value:                    0.419
robustness value at the 95% limit:   0.398

Omitted-variable sensitivity for 'ate'
--------------------------------------------
estimate 0.15731; bias scale sqrt(sigma^2 nu^2) = 0.28648
assumed confounding: cf_y = 0.03, cf_d = 0.03, rho = 1 -> bias <= 0.0087264
bias-adjusted bounds:  [0.14859, 0.16604]
with 95% one-sided CIs: [0.13962, 0.17499] (the sign of the effect survives)

robustness value RV   = 0.4187: a confounder with cf_y = cf_d = 41.9% (the share of the residual outcome variation it explains, and the share of the Riesz representer's second moment only it adds) would move the estimate to 0.
robustness value RVa  = 0.3980: the same, for the 95% confidence bound.

What this output tells you. The robustness value is 0.419. A confounder with cf_y and cf_d both equal to 0.419 would move the point estimate to zero. The value at the 95% limit is 0.398. At the default strengths, the one-sided 95% limits of the bias-adjusted bounds are [0.13962, 0.17499], and they exclude zero.

The fitted \(\nu^2\) is 4.5363, below the law’s 4.83. The expected value of the doubly robust estimate is the law’s \(\nu^2\) minus the mean squared error of the fitted Riesz representer. It therefore reads low when the fitted propensity has an error, and Step 9 shows that it misses part of the tail. On the 200 probe draws, its mean was 4.50. A \(\nu^2\) that is 7% low makes the bias scale about 3% low, so the robustness value reads slightly high. At the law’s \(\nu^2\), this draw gives a robustness value of 0.409 rather than 0.419 (review probe summarize.py).

These numbers hold under the omitted-variable model, its worst-case alignment, and the fitted propensity. They do not show that no stronger confounder exists. The omitted-variable bounds section defines each quantity.

Method evidence#

The fits on this page use one draw of a synthetic law. The registered studies in the table measure each method over repeated draws. Each study page gives its law, its nuisance models, and the result of each cell.

fit on this page	registered study	law of the study
Steps 5 and 6, cross-fitted and in-sample fits	Stacked point-treatment CV-TMLE	this page’s Step 2 law
Step 7, navigator teams	Clustered point-treatment CV-TMLE	the Step 7 team law

No registered study uses the boosted learners or the five folds of this page.

A review probe is a script under reviews/notebook-review/probes/. The step readings quote its results. A review probe is not a registered study.

The technical entry lists the evidence and the limits of the method.

Interpret the result#

question	answer
what the estimate means	in the Step 2 law, the estimated effect of an offer to every eligible discharge is 0.15731 of the maximum score, against usual support for every discharge
population and intervention	adults with a discharge-home order at a participating hospital. The intervention is the offer, analyzed regardless of completed contacts. The estimate is not the effect of completed navigation
uncertainty	the 95% Wald interval [0.14664, 0.16798], from the cross-fitted influence curve. It treats patients as independent, which holds only because the Step 2 law has no teams
with teams	the program’s patients share teams, so its primary analysis must declare them. In the Step 7 law, the team-level interval of the top-box risk difference is [0.0667, 0.1845]. Without the declaration, the registered study’s interval covered 0.84125 of the time
evidence for the interpretation	the registered studies of Method evidence, the review probe of Steps 5 and 6, the diagnostics of Step 9, and the sensitivity analysis of Step 10
assumptions that stay unverified	consistency, no interference, no unmeasured confounding, positivity in the population, and the product-rate condition on the learners
conclusions the analysis does not support	that every patient or team benefits. In the Step 7 law, about 46% of teams have a negative mean effect. Also, that this interval covers at 95% for other learners or laws, or that no stronger hidden confounder exists

Where to go next#

Cross-fitting addresses only data reuse. If you doubt one nuisance model, read the product-rate condition. The interval needs it. A declared known assignment mechanism removes it. DR-TMLE addresses it only in the listed cases. If you doubt which baseline variables belong in the assignment model, read collaborative TMLE.

The examples index lists every tutorial in the program.