Point-treatment TMLE: did transition navigation improve the experience score?#

This tutorial answers one causal question from observational data. It writes the question as an estimand and fits a targeted maximum likelihood estimator (TMLE). It then checks the fit and says what the result supports. Point-treatment TMLE gives the parameter, the influence curve, and the algorithm.

The situation#

A regional health plan offers a standard transition-navigation protocol to adults when a discharge home is ordered. The offer is a bedside plan and two scheduled navigator contacts within 30 days. Discharge teams decided who received the offer. They used a recorded risk process, and nobody randomized the offer.

Each row of the data is one discharge. The table gives the variables in time order, and separates the observed variables from the counterfactual ones.

variable	role	when it is fixed	observed?
`discharge_risk`, `prior_utilization`, `medication_burden`, `age` (together, W)	baseline covariates	before the offer	yes
`transition_navigation` (A)	treatment: 1 for the offer, 0 for usual support	at the discharge-home order	yes
`transition_score` (Y)	outcome: the score as a share of the maximum, so it lies between 0 and 1	30 days after discharge	yes
\(Y^1\) and \(Y^0\)	the score the patient would have with the offer, and without it	30 days after discharge	no. Each patient shows at most one of them, \(Y = Y^A\)

The graph states the causal assumptions of the analysis.

  W ───────────────┐
  │                ▼
  └──────▶ A ────▶ Y

  W  = discharge_risk, prior_utilization, medication_burden, age
  A  = transition_navigation (the offer)
  Y  = transition_score at day 30
  not drawn: an unrecorded cause U with U ──▶ A and U ──▶ Y

arrow or absent arrow	what it asserts	in the synthetic law of this page
W ──▶ A	the baseline profile changes the chance of an offer	all four covariates enter g, through a linear term, a square, an interaction, and an indicator
W ──▶ Y	the baseline profile changes the score in either arm	all four covariates enter Q, through a sine, a square, an interaction, and an absolute value
A ──▶ Y	the offer can change the score. The size of this effect is the target	the effect varies with the covariates
no U	no unrecorded variable causes both the offer and the score	true by construction. In a real program it is an assumption
no node after A	nothing measured after the offer enters the analysis	completed contacts and length of stay occur after the offer, so they are not adjusted for

All four covariates are confounders, because each opens a path A ◀── W ──▶ Y. The analysis must block these paths.

There is no mediator, censoring variable, missing outcome, or death in this law. The outcome mean is the expit of a sum. On the score scale, every covariate therefore modifies the effect. discharge_risk and prior_utilization also change the effect on the logit scale. The graph cannot show whether a real U exists. Step 10 asks how strong it would need to be.

The question#

The program sponsor must decide whether to offer navigation to every eligible discharge. The sponsor asks one question. How much would the mean 30-day transition score change if every eligible discharge received the offer, rather than usual support?

element	this question
target population	adults with a discharge-home order at a participating hospital
intervention	the offer, against usual support, analyzed regardless of completed contacts
counterfactual outcome	\(Y^a\), the 30-day score under offer status \(a\)
estimand	the average treatment effect (ATE), \(\psi = E[Y^1] - E[Y^0]\)

The sponsor’s decision applies to every eligible discharge, so the estimand is the ATE. The effect among the patients who received the offer (the ATT) answers a question about the past rollout. The effect among the others (the ATC) answers a question about an extension to them. The effect varies with discharge_risk, which also drives the offer, so the three differ.

A regression coefficient on the offer is not the causal estimand of this question. It equals the ATE when the regression model is correct and has no effect modification. In general it does not.

identification assumption	what it means for this program	how it is assessed
consistency	an offer always means the declared bedside plan and two scheduled contacts	scientific judgment, from the program’s records of the offer
no interference	one patient’s offer does not change another patient’s score	scientific judgment, from the capacity and access controls of the design
no unmeasured confounding	the recorded baseline variables block every common cause of the offer and the score	scientific judgment. Step 10 measures how strong a violation would need to be
positivity	each baseline profile has some chance of an offer and of usual support	partly checkable. Step 8 describes the fitted overlap

Why this method#

Each choice below follows from a feature of the problem.

feature of the problem	method choice
one treatment decision at time zero, and one outcome at day 30	point-treatment TMLE. No treatment or confounder varies over time, so longitudinal TMLE is not needed
a static contrast, with a true propensity between 0.05 and 0.95	the ATE of “offer everyone” against “offer no one”. Positivity holds in this law, so a stochastic or modified policy is not needed
the score is observed for every patient	no missing-outcome model and no censoring model
both nuisance functions are nonlinear	flexible learners for Q and g. Step 7 shows what misspecified linear models do
a flexible learner can overfit the rows it predicts	cross-fitting. Method evidence links a registered study of this choice
the score lies between 0 and 1 by its definition	`q_bounds=(0.0, 1.0)` and a logistic fluctuation, so the targeted predictions stay on the score’s scale
the sponsor needs an interval	the Wald interval from the targeted influence curve

Terms#

term	plain meaning	reference
estimand	the number the question asks for, written before any model is chosen	estimands
nuisance	a model the estimate needs but the question does not ask about. Here, the outcome regression Q and the treatment mechanism g	point-treatment TMLE
targeting	a small update to Q, weighted by g, that removes the first-order bias of the plug-in estimate	targeting and bounds
influence curve	each row’s contribution to the estimate’s error. Its variance gives the standard error	inference
cross-fitting	each row’s nuisance prediction comes from models fit without that row	CV-TMLE
positivity	every kind of patient has some chance of each arm, so both arms have data to compare	diagnostics

Step 1: set up#

Every fit below passes its learners, fold count, and random seed explicitly. A rerun therefore reproduces the stored outputs.

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, HistGradientBoostingRegressor
from sklearn.linear_model import LinearRegression, LogisticRegression

import cleverly

pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 20)
print("cleverly", cleverly.__version__)

cleverly 0.1.3

What this output tells you. The stored outputs came from the version named above. Another version can print different numbers.

Step 2: the data#

The data come from a synthetic law with a known answer, so later steps can compare each estimate with the truth. A real analysis has no truth. navigation_data draws the rows of the generator make_nonlinear_bounded and gives the columns the program’s names.

from cleverly.datasets import navigation_data

frame, truth = navigation_data(n=3_000, seed=21)
print("rows and columns:", frame.shape)
print("column names:", list(frame.columns))
print(frame.head().round(3))
print()
print("known values of the synthetic law:")
for key in ("ey1", "ey0", "ate", "att", "atc"):
    print(f"  {key}: {truth[key]:.3f}")

rows and columns: (3000, 6)
column names: ['transition_score', 'transition_navigation', 'discharge_risk', 'prior_utilization', 'medication_burden', 'age']
   transition_score  transition_navigation  discharge_risk  prior_utilization  medication_burden    age
0             0.748                    0.0           0.359              1.511             -1.786  1.687
1             0.554                    1.0          -0.047             -0.800             -0.803 -1.083
2             0.463                    0.0          -0.224              0.834              0.584  0.638
3             0.084                    0.0          -1.695             -1.571              1.554  0.969
4             0.948                    1.0           2.183              1.210             -1.024  1.285

known values of the synthetic law:
  ey1: 0.567
  ey0: 0.404
  ate: 0.163
  att: 0.177
  atc: 0.151

What this output tells you. The columns are the ones the situation table names. Every score lies between 0 and 1. The covariates are independent standard-normal draws, so a negative age is below the average age. Real covariates would be correlated.

The truth line ate is \(E[Y^1] - E[Y^0] = 0.163\). The ATT is 0.177 and the ATC is 0.151, so the choice of estimand changes the answer.

Step 3: association first#

The graph predicts that the unadjusted contrast is confounded. The recorded risk process offers navigation more often to patients with a higher discharge_risk, and that covariate also changes the score. The unadjusted difference \(E[Y \mid A = 1] - E[Y \mid A = 0]\) equals the ATE only when no such path exists. The code compares the arms before any adjustment.

covariates = ["discharge_risk", "prior_utilization", "medication_burden", "age"]
by_arm = frame.groupby("transition_navigation")[["transition_score", *covariates]].mean()
print(by_arm.round(3))
print()
print("share offered navigation:", round(float(frame["transition_navigation"].mean()), 3))
unadjusted = by_arm.loc[1.0, "transition_score"] - by_arm.loc[0.0, "transition_score"]
print(f"unadjusted difference in mean score: {unadjusted:.3f}")
print(f"population ATE:                      {truth['ate']:.3f}")

                       transition_score  discharge_risk  prior_utilization  medication_burden    age
transition_navigation                                                                               
0.0                               0.387          -0.219              0.004             -0.024 -0.011
1.0                               0.585           0.226              0.012             -0.047  0.053

share offered navigation: 0.465
unadjusted difference in mean score: 0.198
population ATE:                      0.163

What this output tells you. The offered patients score 0.198 higher on average, and the true ATE is 0.163. The arms differ before the offer. The mean discharge_risk is 0.226 among offered patients and -0.219 among the others.

Two mechanisms of this law raise the unadjusted difference above the ATE. First, the offered patients would score higher than the others under usual support alone. Second, the effect grows with discharge_risk, so the offered patients gain more than the average patient. An estimator must compare patients with the same baseline profile. The next two steps state that comparison and the assumptions that make it possible.

Step 4: write the protocol#

A StudyProtocol records the scientific design before any model runs. It uses target-trial terms: the population, the time zero, the strategies, the outcome, and the handling of later events. Every result fitted from it carries its fingerprint. navigation_protocol() in cleverly.datasets holds the program’s record. A real analysis calls StudyProtocol(...) with the same ten fields.

from cleverly.datasets import navigation_protocol

protocol = navigation_protocol()
print("\n".join(protocol.summary_lines()))

causal study protocol: schema 1; 623fc2c240615d26
target population: Adults with a discharge-home order at a participating hospital during the enrollment period
eligibility: ['Age 18 years or older', 'Discharge home ordered at a participating hospital']
time zero: Discharge-home order, after baseline measurement and before the navigation offer
treatment strategies: ['Offer standard transition navigation', 'Provide usual discharge support']
treatment versions: ['Bedside transition plan and two scheduled navigator contacts within 30 days', 'No access to the transition-navigation offer']
outcome: Patient-reported transition score, as a share of the maximum score
horizon: 30 days after discharge
intercurrent-event handling: ['Use the transition score regardless of readmission', 'Analyze the offer regardless of completed contacts', 'The protocol scores death before day 30 as the worst transition score (composite strategy)']
interference unit: Individual patient
assumption rationale: ['The recorded baseline variables cover the measured common causes', 'The standardized offer and version records support consistency', 'Reserved navigator capacity and access controls support no interference']

What this output tells you. The first line gives the schema version and the fingerprint 623fc2c240615d26. Three fields set the meaning of the answer.

protocol field	what it fixes for this question
target population and time zero	the effect is about adults with a discharge-home order at a participating hospital. Follow-up starts at the order, before the offer, so eligibility cannot depend on the offer
treatment strategies and versions	the treatment is the offer. The protocol analyzes the offer regardless of completed contacts, so the estimand is the effect of offering navigation, not of receiving it
outcome and intercurrent events	the score is a share of the maximum. A death before day 30 counts as the worst score. The synthetic law draws no deaths

The outcome field is the evidence for the score support that Step 6 declares. A fit cannot recover that support from the data. The protocol does not store the contrast. The typed estimand in the next step owns it.

Step 5: design and identification#

The design says which column plays which role. The estimand says which contrast the question needs. Identification writes the estimand as a quantity of the observed data, and lists the assumptions that make the two equal.

from cleverly import ATE, CausalStudy, PointTreatment

study = CausalStudy(
    frame,
    design=PointTreatment(
        outcome="transition_score",
        treatment="transition_navigation",
        adjustment=tuple(covariates),
    ),
    protocol=protocol,
)
effect = study.identify(ATE(reference=0))

print(effect.summary(protocol="fingerprint"))

average treatment effect, E[Y^a] - E[Y^reference]
identified by explicit-adjustment: E_W[E(transition_score | transition_navigation=a, W)] - E_W[E(transition_score | transition_navigation=0, W)] for a in [1]
adjustment/history: ['discharge_risk', 'prior_utilization', 'medication_burden', 'age']
required nuisances: ['outcome_regression', 'treatment_mechanism']
assumptions:
  - consistency: Y = Y^a when A = a
  - no interference: one unit's potential outcome does not depend on other units' treatment assignments
  - no unmeasured confounding: Y^a is independent of A given W
  - positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]
causal study protocol: schema 1; 623fc2c240615d26

What this output tells you. Under the four printed assumptions, the ATE equals the printed formula \(E_W[Q(1, W) - Q(0, W)]\). Here \(Q(a, W) = E(Y \mid A = a, W)\). The formula needs two nuisances: Q, and the treatment mechanism \(g(W) = P(A = 1 \mid W)\). The question section states which assumptions need scientific judgment.

The shared study design states how the program supports consistency and no interference. An unrecorded discharge-team judgment that affects both the offer and recovery would violate no unmeasured confounding, and no estimator repairs that. A real protocol should also evaluate site, navigator team, calendar time, and language access as confounders.

Step 6: estimate the ATE#

The method follows the choices of “Why this method”. The configuration is written out in full.

setting	value	what it does
`outcome_learner`	gradient boosting	fits Q
`treatment_learner`	a shallow, regularized booster	fits g
`CrossFitting(n_folds=5)`	five folds	predicts each row from models that did not see that row
`Targeting(q_bounds=(0.0, 1.0))`	the declared score support	fixes the outcome scale from the protocol, not from the observed scores
`Inference(alpha=0.05)`	95% interval	sets the interval level
`Runtime(random_state=21, n_jobs=1)`	fixed seed, one process	makes the fit reproducible

Do not use the default HistGradientBoostingClassifier as a treatment learner below 10,000 rows. Below that size, scikit-learn does not stop it early, and its fitted propensities are too extreme. The rules for code cells state this requirement, and Step 8 checks the calibration of the fitted propensities.

from cleverly import CrossFitting, Inference, ModelSpec, Runtime, Targeting, TMLEMethod


def shallow_booster():
    """A shallow, regularized boosted propensity learner."""
    return HistGradientBoostingClassifier(
        max_depth=2,
        learning_rate=0.05,
        max_iter=200,
        l2_regularization=1.0,
        random_state=21,
    )


flexible = TMLEMethod(
    models=ModelSpec(
        outcome_learner=HistGradientBoostingRegressor(random_state=21),
        treatment_learner=shallow_booster(),
    ),
    cross_fitting=CrossFitting(n_folds=5),
    targeting=Targeting(q_bounds=(0.0, 1.0)),
    inference=Inference(alpha=0.05),
    runtime=Runtime(random_state=21, n_jobs=1),
)
result = effect.estimate(method=flexible)

estimate = result["ate"]
print(result.summary(protocol="fingerprint"))
print()
print(f"estimate:        {estimate.psi:.3f}")
print(f"standard error:  {estimate.std_error:.4f}")
print(f"95% CI:          ({estimate.ci[0]:.3f}, {estimate.ci[1]:.3f})")
print(f"population ATE:  {truth['ate']:.3f}")

Targeted maximum likelihood estimation
======================================
n = 3000; covariates = 4; P(A=1) = 0.4653
causal estimand: average treatment effect, E[Y^a] - E[Y^reference]
identification: explicit-adjustment; E_W[E(transition_score | transition_navigation=a, W)] - E_W[E(transition_score | transition_navigation=0, W)] for a in [1]
required nuisances: outcome_regression, treatment_mechanism
identification assumptions: consistency: Y = Y^a when A = a; no interference: one unit's potential outcome does not depend on other units' treatment assignments; no unmeasured confounding: Y^a is independent of A given W; positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]
causal study protocol: schema 1; 623fc2c240615d26
outcome family: gaussian; fluctuation: logistic (iterative)
stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled
propensity truncated to [0.0114, 0.9886]; ATT/ATC to [0.025, 0.975]
data 67649085bf54f0fd | folds 01ad882979feb330 | cleverly 0.1.3 | 2026-10-16T01:16:37+00:00

estimand  psi      std_err   95% CI              p_value
--------  -------  --------  ------------------  -------
ate       0.16658  0.005482  [0.15584, 0.17733]  <1e-4  

estimate:        0.167
standard error:  0.0055
95% CI:          (0.156, 0.177)
population ATE:  0.163

What this output tells you. The summary names the construction as stacked CV-TMLE. The nuisances are cross-fitted over 5 folds, and one targeting step runs on the pooled out-of-fold predictions. The propensity truncation bound is [0.0114, 0.9886].

The estimate is 0.167 with a standard error of 0.0055. The 95% interval is (0.156, 0.177), and it contains the true ATE of 0.163 on this draw. One draw is not a coverage result. A review probe, reviews/notebook-review/probes/point-treatment-tmle-final/sweep.py, repeats this fit on 200 draws of the same law. The interval contained the true ATE on 189 of them, and the mean standard error was 0.98 times the standard deviation of the estimates. The probe is not a registered study.

The interval is asymptotically valid under positivity, the declared independence of the rows, and a rate condition on the nuisances. The rate condition asks that the errors of Q and g, multiplied together, shrink faster than \(1/\sqrt{n}\).

Step 7: what targeting can repair, and what it cannot#

Double robustness means that the TMLE point estimate stays consistent when either Q or g is consistent. The reason is the remainder of the estimator, which is a product of the error of Q and the error of g. The product shrinks when one factor shrinks.

This step compares four estimators of the same ATE, with the same identification formula from Step 5. Name the mechanism before reading the numbers. The law’s Q and g contain a sine, squares, interactions, and an absolute value. A main-terms linear or logistic regression cannot represent them, so each linear model is inconsistent here. The boosted learners can approximate them as n grows.

estimator	Q	g	expected behavior
g-computation plug-in	linear	none	inherits the bias of the linear Q
TMLE	linear	booster	consistent through g. The g error shrinks, so the product remainder shrinks
TMLE	booster	logistic	consistent through Q
TMLE	linear	logistic	neither nuisance is consistent, so the remainder need not shrink

The plug-in averages \(\hat Q(1, W) - \hat Q(0, W)\) over the rows. With a main-terms linear Q, that average equals the regression coefficient on the offer. The plug-in has no influence-curve interval, so the code prints none. Each TMLE fit copies the Step 6 method and changes only the learners.

from dataclasses import replace

from cleverly.learners import thread_limit

dr_points = {}

features = frame[["transition_navigation", *covariates]]
with thread_limit():
    linear_q = LinearRegression().fit(features, frame["transition_score"])
offered = linear_q.predict(features.assign(transition_navigation=1.0))
not_offered = linear_q.predict(features.assign(transition_navigation=0.0))
plug_in = float(np.mean(offered - not_offered))
print(
    f"{'plug-in, linear Q':22s} psi={plug_in:6.3f}  {'':20s}miss={abs(plug_in - truth['ate']):.3f}"
)


def fit(outcome_learner, treatment_learner, label):
    method = replace(
        flexible,
        models=ModelSpec(outcome_learner=outcome_learner, treatment_learner=treatment_learner),
    )
    point = effect.estimate(method=method)["ate"]
    dr_points[label] = point
    low, high = point.ci
    miss = abs(point.psi - truth["ate"])
    print(f"{label:22s} psi={point.psi:6.3f}  CI=({low:.3f}, {high:.3f})  miss={miss:.3f}")


fit(LinearRegression(), shallow_booster(), "linear Q, flexible g")
fit(
    HistGradientBoostingRegressor(random_state=21),
    LogisticRegression(max_iter=1000, random_state=21),
    "flexible Q, linear g",
)
fit(LinearRegression(), LogisticRegression(max_iter=1000, random_state=21), "both linear")
print(f"population ATE: {truth['ate']:.3f}")

plug-in, linear Q      psi= 0.148                      miss=0.015

linear Q, flexible g   psi= 0.164  CI=(0.150, 0.179)  miss=0.002

flexible Q, linear g   psi= 0.167  CI=(0.157, 0.177)  miss=0.005
both linear            psi= 0.144  CI=(0.130, 0.158)  miss=0.019
population ATE: 0.163

What this output tells you. The plug-in from the linear Q misses the true ATE by 0.015. TMLE with the same linear Q and the boosted g misses by 0.002, and its interval contains 0.163. TMLE with the boosted Q and the logistic g also contains it. With both models linear, TMLE misses by 0.019, and its interval (0.130, 0.158) excludes the true value. Neither of its nuisances is consistent, so targeting cannot remove the product of their errors.

One draw shows a pattern, not a property. Two review probes in reviews/notebook-review/probes/point-treatment-tmle-final/ repeat these fits on 200 draws. summary.log labels the TMLE fits as “Step 8”.

estimator	mean error over 200 draws	interval contains the truth	probe
plug-in, linear Q	-0.018	no interval	`plugin.py`
TMLE, linear Q and boosted g	-0.0022	195 of 200	`sweep.py`
TMLE, boosted Q and logistic g	-0.0006	190 of 200	`sweep.py`
TMLE, both linear	-0.0218	30 of 200	`sweep.py`

The both-linear miss is therefore a bias, not sampling variation. Its mean error is larger than the plug-in’s, so targeting with a wrong g can move a wrong Q further from the truth.

caution	why
a real analysis does not know which nuisance is consistent	the linear models here omit terms that only the synthetic law reveals
one nuisance inconsistent	the point estimate can stay consistent, but the influence-curve interval need not be valid. DR-TMLE addresses this only in the listed cases
one law is not a general result	another law, another size, or other learners can change the pattern

This is why Step 6 uses flexible learners for both nuisances. Flexible learners make a consistent Q and g more likely. They do not guarantee it.

Step 8: diagnostics, what the fit can check#

The assessment of the Step 6 fit holds its diagnostic reports. The code prints the rows that need attention and three reports. The support report describes overlap, the sample view of positivity. The nuisance report gives the calibration of the fitted propensities. The score check confirms that targeting solved its equation.

assessment = result.assess()
print("needs attention:", tuple(item.name for item in assessment.attention))

support = assessment.report("support")
nuisance = assessment.report("nuisance_models")
scores = assessment.report("score_equations")
print()
print(support.summary())
print()
print(nuisance.summary())
print()
print(scores.summary().splitlines()[-1])

needs attention: ()

Positivity / overlap diagnostics
--------------------------------
n = 3000; propensity truncated to [0.0114, 0.9886]

g(W) quantile  0%      1%      5%      25%     50%     75%     95%     99%     100%  
-------------  ------  ------  ------  ------  ------  ------  ------  ------  ------
overall        0.0704  0.1314  0.2187  0.3703  0.4690  0.5589  0.7132  0.7840  0.8815
treated        0.1087  0.1842  0.2800  0.4265  0.5040  0.5969  0.7305  0.7938  0.8815
control        0.0704  0.1230  0.1949  0.3208  0.4311  0.5163  0.6723  0.7520  0.8799

arm      n     effective n  ESS / n  top 1% weight  top 5% weight
-------  ----  -----------  -------  -------------  -------------
treated  1396  1212.5       0.869    0.034          0.112        
control  1604  1447.7       0.903    0.028          0.101        

threshold  P(g < t)  P(g > 1-t)
---------  --------  ----------
0.01       0.0000    0.0000    
0.025      0.0000    0.0000    
0.05       0.0000    0.0000    
0.1        0.0027    0.0000    

truncated: 0 unit(s) (0.00%); most extreme untruncated g(W) = 0.070385

group  equation  target rows  Kish-equivalent rows  Kish / target  Kish / total  top 1% load  top 5% load  max |w h|  zero load  g bound           clipped  
-----  --------  -----------  --------------------  -------------  ------------  -----------  -----------  ---------  ---------  ----------------  ---------
mean   h1        3000/3000    1212.5                0.404          0.404         0.060        0.198        9.199      1604       [0.0114, 0.9886]  0 (0.00%)
(absolute residual-multiplier load from the fitted equation; Kish values describe residual-multiplier concentration, not residual contributions or estimator information, influence, or precision; target rows are score-mask rows and include structural zeros)
max |clever covariate| (mean): 9.199

VERDICT: no truncation-driven fragility detected (0.0% of units clipped at the marginal treatment-mechanism bound [0.0114, 0.9886]). The narrowest arm has a Kish-equivalent weight count of 87% of its nominal rows. This is concentration, not estimator information, and no threshold is applied because none is derived. Absolute-load concentration is greatest for group 'mean', equation 'h1': 1212.5 Kish-equivalent rows out of 3000 score-mask rows (40%; 40% of 3000 total rows). This is a descriptive residual-multiplier concentration index. It excludes the residual and is not estimator effective sample size, information, actual influence, precision, or a positivity test.

Nuisance model diagnostics
----------------------------------------
model       auc     brier   log_loss  r2      mse     cal_slope  cal_se  reg_slope
----------  ------  ------  --------  ------  ------  ---------  ------  ---------
propensity  0.6611  0.2291  0.6498    -       -       0.9564     0.0662  -        
outcome     -       -       -         0.6746  0.0176  -          -       0.9546   
Calibration slope: logistic recalibration of the out-of-fold predictions with one intercept per fold.
A finding needs the 95.0% interval to lie above 0 and exclude 1, a Bonferroni level over 1 tested model(s).

VERDICT: nuisance fits look reasonable.

PASS: the targeting step solved the estimated efficient score equation.

What this output tells you. No row needs attention.

output part	what it shows on this draw	what it cannot show
positivity and overlap	no unit sits at the truncation bound. The smallest fitted propensity is 0.070385. The effective-sample-size ratio is 0.869 in the treated arm and 0.903 in the control arm	the true propensity. The report describes the fitted model
nuisance model diagnostics	the out-of-fold calibration slope of the propensity is 0.9564, with a standard error of 0.0662. The verdict reads `nuisance fits look reasonable`	that the propensity model is correct
score-equation check	targeting solved the estimated efficient score equation	that the estimate is unbiased

A calibration slope well below 1 means that the fitted propensities are more extreme than the observed offer rates. The default booster that Step 6 avoids produces that pattern below 10,000 rows. A slope of 0.9564 with that standard error does not differ from 1.

The fitted model can still miss part of the law. It puts a share of 0.0027 of the rows below 0.1, in the P(g < t) column. In the law, about 1.9% of patients have a true propensity below 0.1 (review probe reviews/notebook-review/probes/point-treatment-tmle-final/truth.py). The report gives no positivity verdict, because no universal cutoff applies.

Step 9: truncation, does the bound drive the estimate?#

Truncation clips each fitted propensity into a bound, so no row gets an extreme weight. A clip adds bias that the standard error does not include. The code retargets the estimate at a grid of bounds and at the fitted bound, without refitting the nuisances. The grid ends with two tight bounds, 0.1 and 0.2, which clip rows on this fit.

curve = result.diagnostics.truncation_curve()
columns = ["bound", "psi", "std_err", "ci_lower", "ci_upper", "truncated_fraction"]
print(curve[columns].round(4).to_string(index=False))

 bound    psi  std_err  ci_lower  ci_upper  truncated_fraction
0010 0.1666   0.0055    0.1558    0.1773              0.0000
0050 0.1666   0.0055    0.1558    0.1773              0.0000
0100 0.1666   0.0055    0.1558    0.1773              0.0000
0114 0.1666   0.0055    0.1558    0.1773              0.0000
0250 0.1666   0.0055    0.1558    0.1773              0.0000
0500 0.1666   0.0055    0.1558    0.1773              0.0000
1000 0.1666   0.0055    0.1558    0.1773              0.0027
2000 0.1674   0.0053    0.1570    0.1778              0.0417

What this output tells you. The fitted bound is 0.0114, from the default rule \(5 / (\sqrt{n} \log n)\) at \(n = 3000\). From 0.0010 to 0.0500 the truncated fraction is 0.0000. The default bound therefore changes no row, and the estimate stays at 0.1666. At 0.2000 the truncated fraction is 0.0417, and the estimate moves to 0.1674. Over this grid, the choice of bound moves the reported estimate by less than one standard error. A flat curve does not verify positivity.

Step 10: sensitivity, what the fit cannot check#

Diagnostics cannot see an unmeasured confounder. Sensitivity analysis asks how strong a confounder would need to be to explain the result away.

term	plain meaning
`cf_y`	the share of the remaining outcome variation that a hidden confounder explains
`cf_d`	the share of the Riesz representer’s second moment that only the hidden confounder adds. For the ATE, the representer is the inverse-propensity weight
`rho`	how closely the confounder’s two effects align. `rho=1` is the worst case
robustness value	the equal `cf_y` and `cf_d` that moves the point estimate to zero at `rho=1`
benchmark	a refit without one recorded covariate. It measures what that covariate adds to the outcome regression and to \(\nu^2\)

A benchmark makes the strengths concrete: “a confounder as strong as discharge_risk”. The code benchmarks the strongest recorded confounder, discharge_risk, and a weaker one, medication_burden. It then bounds the bias at each benchmark’s strengths, with the default rho=1.

benchmark() reports gain statistics. Chernozhukov et al. (arXiv:2112.13398v6, Appendix E.5, Remark 8) use these gains as proxies for the strengths that the bound reads. The call omitted_confounding(benchmark=record) therefore passes them to the bound unchanged.

sensitivity = result.sensitivity
robustness = sensitivity.robustness_value()
strong = sensitivity.benchmark(covariates=("discharge_risk",))
benchmark = sensitivity.benchmark(covariates=("medication_burden",))
strong_bounds = sensitivity.omitted_confounding(benchmark=strong)
bounds = sensitivity.omitted_confounding(benchmark=benchmark)
print(f"robustness value:                  {robustness['rv']:.3f}")
print(f"robustness value at the 95% limit: {robustness['rva']:.3f}")
print(f"nu^2, doubly robust estimate:      {sensitivity.elements(estimand='ate').nu2:.3f}")
for name, record, bound in (
    ("discharge_risk", strong, strong_bounds),
    ("medication_burden", benchmark, bounds),
):
    print()
    print(f"as strong as {name}:")
    print(f"  estimate without it {record.psi_short:.4f}")
    print(f"  gain in outcome variation {record.gain_y:.4f}; cf_y {record.cf_y:.4f}")
    print(f"  gain in nu^2 {record.gain_d:.4f}; cf_d {record.cf_d:.4f}")
    print(f"  bias <= {bound.bias:.4f} at rho = {bound.rho:g}")
    print(f"  bias-adjusted bounds  [{bound.lower:.4f}, {bound.upper:.4f}]")
    print(f"  one-sided 95% limits  [{bound.ci_lower:.4f}, {bound.ci_upper:.4f}]")

robustness value:                  0.445
robustness value at the 95% limit: 0.425
nu^2, doubly robust estimate:      4.418

as strong as discharge_risk:
  estimate without it 0.2198
  gain in outcome variation 1.3360; cf_y 1.0000
  gain in nu^2 0.0282; cf_d 0.0282
  bias <= 0.0474 at rho = 1
  bias-adjusted bounds  [0.1191, 0.2140]
  one-sided 95% limits  [0.1099, 0.2230]

as strong as medication_burden:
  estimate without it 0.1699
  gain in outcome variation 0.1122; cf_y 0.1122
  gain in nu^2 0.0587; cf_d 0.0587
  bias <= 0.0233 at rho = 1
  bias-adjusted bounds  [0.1433, 0.1899]
  one-sided 95% limits  [0.1342, 0.1989]

What this output tells you. The robustness value is 0.445. A confounder with cf_y and cf_d both equal to 0.445 would move the point estimate to zero at rho=1. The value at the 95% limit is 0.425.

benchmark	estimate without it	`cf_y`	`cf_d`	bias at most	bias-adjusted bounds	one-sided 95% limits
`discharge_risk`	0.2198	1.0000	0.0282	0.0474	[0.1191, 0.2140]	[0.1099, 0.2230]
`medication_burden`	0.1699	0.1122	0.0587	0.0233	[0.1433, 0.1899]	[0.1342, 0.1989]

Without discharge_risk, the residual outcome variance rises by a gain of 1.3360. A share cannot exceed 1, so the benchmark clips cf_y to 1.0000. This strength means that the hidden confounder explains all of the remaining outcome variation. The bound is finite there, because the treatment strength cf_d stays at 0.0282.

At the strengths of either benchmark and the worst alignment, both one-sided limits exclude zero on this draw.

limitation	effect on the reading
the bound reads the fitted \(\nu^2 = 4.418\), and the law’s value is about 4.83	the doubly robust estimate of \(\nu^2\) has expectation equal to the law’s value minus the squared error of the fitted representer. It reads low whenever the fitted propensity has an error, so the bounds are slightly narrow. With the law’s value, the bias bound would be about 5% larger, and the robustness value would be 0.431 (review probe `summarize.py`)
a benchmark is one draw	over the 200 probe draws, the one-sided limits at the `medication_burden` strengths excluded zero on every draw (`summary.log`)
a benchmark is an analogy	it assumes that a hidden confounder adds to the recorded covariates what the dropped covariate adds to the others

The analysis supports one statement. On this draw, a hidden confounder as strong as discharge_risk, at the worst alignment, does not move either one-sided limit to zero. The bound does not cover a confounder that changes the offer more than discharge_risk does. The program must judge whether an unrecorded team judgment could be that strong. The omitted-variable bounds section defines each quantity.

Method evidence#

The fits on this page use one draw of a synthetic law. The registered study in the table measures the method over repeated draws. The study page gives its law, its nuisance models, and the result of each cell.

fit on this page	registered study	law of the study
Step 6, CV-TMLE	Stacked point-treatment CV-TMLE	this page’s law

No registered study uses the boosted learners or the five folds of this page.

A review probe is a script under reviews/notebook-review/probes/. The step readings quote its results. A review probe is not a registered study.

The technical entry lists the evidence and the limits of the method.

Step 11: answer the question#

The sponsor asked how much the mean 30-day transition score would change if every eligible discharge received the offer. The table gives the answer and its limits.

item	the answer on this page
estimate	the offer to every eligible discharge would raise the mean score by 0.167 of the maximum score, against usual support
uncertainty	the 95% interval is (0.156, 0.177). It reflects sampling variation only
population and intervention	adults with a discharge-home order at a participating hospital. The intervention is the offer, analyzed regardless of completed contacts
supporting evidence	the diagnostics of Step 8, the flat truncation curve of Step 9, and the bounds of Step 10. The registered study supports the cross-fitting choice, and the review probe supports this configuration on this law
unverified assumptions	consistency, no interference, and no unmeasured confounding. Positivity is only partly checked

The analysis does not support these conclusions:

the effect of completing the navigator contacts, because the protocol defines the offer as the treatment;
the effect among the patients who received the offer, which is the ATT;
the same effect at other hospitals or for other eligibility rules;
robustness to a hidden confounder that changes the offer more than discharge_risk does;
a general performance claim for the estimator, from one synthetic draw.

Where to go next#

This page treated discharges as independent rows. Read CV-TMLE and cross-fitting for the same question at network scale, where patients share navigator teams. If your concern is which baseline variables belong in the assignment model, read collaborative TMLE.

To report the ATT or the ATC instead, read estimands. To keep a fit and replay its assessment later, use result.save() and cleverly.load().

The examples index lists every tutorial in the program.