Survey non-response: the patients who never answered#
A hospital program offers transition navigation at discharge. On day 30 it sends a survey that scores how well the transition went, and the survey closes on day 45. About a quarter of patients never return it. The program sponsor asks for the effect of navigation on the score of every eligible patient, including the patients who did not answer.
This notebook builds that analysis from the question to the interpretation. It reuses the navigation program of the point-treatment tutorial and adds one problem: the outcome is missing for some patients.
The situation#
element |
in this program |
|---|---|
population |
adults with a discharge-home order at a participating hospital |
time zero |
the discharge-home order, after the baseline record and before the offer |
baseline variables \(W\) |
|
treatment \(A\) |
|
response indicator \(\Delta\) |
|
outcome \(Y\) |
|
counterfactual outcome \(Y(a)\) |
the score that the patient would report under arm \(a\). No row records it under both arms, and no row records it when \(\Delta = 0\) |
The data are synthetic, so the true effect is known. A real program has no such truth. Each comparison with the truth below is a teaching device, not a check that a real analysis can run.
The causal graph#
The graph lists every arrow of the synthetic law, missing_outcome_bounded_dgp() in
cleverly.datasets. W1 is discharge_risk, W2 is age, and W3 is prior_utilization.
W1 -> A W2 -> A
W1 -> Y W2 -> Y W3 -> Y A -> Y
W1 -> Δ W3 -> Δ A -> Δ
arrow |
what it asserts |
what the analysis does with it |
|---|---|---|
|
discharge risk and age change the chance of an offer |
W1 and W2 confound the effect of A on Y. The treatment model \(g(W) = P(A = 1 \mid W)\) adjusts for them |
|
each baseline variable changes the score |
the outcome regression \(Q(A, W)\) uses all three |
|
navigation changes the score |
this is the effect to estimate. Its size falls as W1 rises, so W1 also modifies the effect |
|
response falls as discharge risk rises, and it rises with prior utilization |
W3 does not confound A and Y, but the adjustment set must contain it. Without W3, the response model omits a common cause of \(\Delta\) and Y |
|
navigation changes who answers |
the response model \(\pi(A, W) = P(\Delta = 1 \mid A, W)\) conditions on the arm |
An arrow that the graph leaves out is also an assumption.
absent arrow |
the assumption that it encodes |
can the data check it? |
|---|---|---|
|
missingness at random (MAR): given A and W, response carries no information about the score |
no |
a hidden common cause of A and Y |
no unmeasured confounding: given W, the offer is independent of \(Y(a)\) |
no |
an arrow from one patient to another |
no interference: one patient’s offer does not change another patient’s score |
no |
|
prior utilization does not change the offer, and age does not change response. Both hold in this law |
identification does not use them, because every model adjusts for all of W |
The graph has no death node, because the synthetic law draws no deaths. Step 4 states what deaths would change.
The causal question#
The sponsor decides whether to offer navigation to every eligible patient. The question is how much the mean transition score changes if every eligible patient gets the offer, compared with none.
element |
definition |
|---|---|
target population |
every eligible patient, whether or not the patient returns the survey |
intervention |
the navigation offer for all (\(a = 1\)), against usual support for all (\(a = 0\)) |
counterfactual outcome |
\(Y(a)\), the score under arm \(a\) |
estimand |
the average treatment effect (ATE), \(\psi = E[Y(1)] - E[Y(0)]\), in units of the score scale |
The decision applies to every eligible patient, so the estimand averages over all of them. An
average over respondents only is a different quantity: the conditional effect standardized to the
respondents’ covariates (Step 7). It is also not an effect in one fixed subgroup. Response occurs
after the offer, and navigation changes it (A -> Δ).
Under the assumptions of the next table, the observed data identify the ATE:
The inner regression uses respondents only. The outer mean runs over every patient.
assumption |
statement |
support |
|---|---|---|
consistency |
\(Y = Y(a)\) when \(A = a\) |
the protocol defines one version of each arm. Scientific judgment |
no interference |
one patient’s offer does not change another patient’s score |
reserved navigator capacity. Scientific judgment |
no unmeasured confounding |
\(Y(a)\) is independent of A given W |
the recorded baseline variables. Scientific judgment |
missingness at random |
\(Y(a)\) is independent of \(\Delta\) given \(A = a\) and W |
scientific judgment. Step 9 measures how far it must fail to change the conclusion |
treatment positivity |
\(P(A = a \mid W) > 0\) for each arm |
partly checkable. Step 8 reads the fitted probabilities |
response positivity |
\(P(\Delta = 1 \mid A = a, W) > 0\) for each arm |
partly checkable. Step 8 reads the fitted probabilities |
Why this method#
feature of the problem |
the method feature that it calls for |
|---|---|
one decision at time zero, with baseline covariates only |
point-treatment TMLE. No time-varying treatment or confounding needs a longitudinal fit |
missing scores, with response that depends on A and W |
the design declares |
two models that can each be wrong |
double robustness: the estimate is consistent when Q is correct, or when the product \(g \cdot \pi\) is correct for each arm |
an outcome surface with curvature and an effect that changes with W1 |
a flexible learner for Q, a boosted tree ensemble |
a flexible learner |
cross-fitting: each row’s nuisance predictions come from models fitted without that row |
a score with a known range of 0 to 1 |
|
Targeting updates the initial outcome regression along the clever covariate
The factor \(1/\pi\) reweights the respondents toward the whole population. After the update, the estimate solves the estimated efficient influence-curve equation. The interval needs the product of the errors of Q and of \(g \cdot \pi\) to shrink faster than \(n^{-1/2}\). Missing outcomes gives the influence curve, the remainder, and the stacked CV-TMLE that this page fits.
term |
plain meaning |
|---|---|
a model that the estimate needs but the question does not ask about. Here Q, g, and \(\pi\) |
|
the update of Q along H that removes the first-order bias of the initial fit |
|
how much each row moves the estimate. Its variance gives the standard error |
|
each row’s nuisance prediction comes from models fitted without that row |
Step 1: set up#
Every fit below passes its learners, its fold policy, and its random seed, so a rerun reproduces the stored outputs.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import LogisticRegression
import cleverly
pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 20)
print("cleverly", cleverly.__version__)
cleverly 0.1.3
What this output tells you. The stored outputs came from the version named above. A different version can print different numbers.
Step 2: the data#
The code draws 4,000 patients from make_missing_outcome_bounded() and renames the columns to the
program’s names. The default strength of 2 gives the outcome curvature, an effect that falls with
discharge risk, and a strong link from discharge risk to response.
from cleverly.datasets import make_missing_outcome_bounded
program_names = {
"Y": "transition_score",
"A": "transition_navigation",
"W1": "discharge_risk",
"W2": "age",
"W3": "prior_utilization",
"Delta": "responded",
}
frame, truth = make_missing_outcome_bounded(n=4_000, seed=71)
frame = frame.rename(columns=program_names)
print("rows and columns:", frame.shape)
print(list(frame.columns))
print(frame.head().round(3))
print()
print(f"response rate: {frame['responded'].mean():.3f}")
print(f"missing transition score: {int(frame['transition_score'].isna().sum())} rows")
print(f"population ATE: {truth['ate']:.4f}")
rows and columns: (4000, 6)
['transition_score', 'transition_navigation', 'discharge_risk', 'age', 'prior_utilization', 'responded']
transition_score transition_navigation discharge_risk age prior_utilization responded
0 0.707 1.0 0.049 0.518 -0.110 1.0
1 0.567 1.0 -0.089 0.503 0.511 1.0
2 0.560 0.0 0.345 -1.421 -0.570 1.0
3 0.685 1.0 -0.307 0.740 0.045 1.0
4 0.174 0.0 -0.394 0.378 0.326 1.0
response rate: 0.763
missing transition score: 949 rows
population ATE: 0.1466
What this output tells you. Each row is one discharge. The response rate is 0.763, so 949 of
the 4,000 scores are missing. A missing score is NaN. The true ATE for every eligible patient is
0.1466 on the 0 to 1 scale.
Step 3: who answers the survey#
Before any model, compare the respondents with the patients who did not answer.
covariates = ["discharge_risk", "age", "prior_utilization"]
by_response = frame.groupby("responded")[covariates].mean()
print("mean baseline covariate by response status:")
print(by_response.round(3))
mean baseline covariate by response status:
discharge_risk age prior_utilization
responded
0.0 0.781 0.012 -0.215
1.0 -0.229 -0.012 0.056
What this output tells you. Patients who did not respond have a mean discharge_risk of
0.781. Respondents have a mean of -0.229. Respondents also have more prior utilization, 0.056
against -0.215. Both gaps follow the arrows W1 -> Δ and W3 -> Δ.
The effect falls as discharge risk rises, and respondents have lower discharge risk. The average effect among respondents is therefore larger than the ATE. Step 7 measures that difference.
Step 4: write the protocol#
A StudyProtocol records the scientific design before any model runs. The page starts from
navigation_protocol(), the protocol of the
shared study design, and changes three fields.
field |
what changes here |
|---|---|
|
it states that non-respondents belong to the population |
|
it names the survey window and the 0 to 1 range of the score |
|
it adds missingness at random for the survey response |
from dataclasses import replace
from cleverly.datasets import navigation_protocol
program = navigation_protocol()
protocol = replace(
program,
target_population=f"{program.target_population}, whether or not they return the survey",
outcome=(
f"{program.outcome}, from the survey sent on day 30 and closed on day 45; "
"the score lies between 0 and 1"
),
assumption_rationale=(
program.assumption_rationale[0],
"Given the arm and the baseline variables, survey response carries no further "
"information about the unobserved score (missingness at random)",
*program.assumption_rationale[1:],
),
)
print("\n".join(protocol.summary_lines()))
causal study protocol: schema 1; 1f8351d839cc8afe
target population: Adults with a discharge-home order at a participating hospital during the enrollment period, whether or not they return the survey
eligibility: ['Age 18 years or older', 'Discharge home ordered at a participating hospital']
time zero: Discharge-home order, after baseline measurement and before the navigation offer
treatment strategies: ['Offer standard transition navigation', 'Provide usual discharge support']
treatment versions: ['Bedside transition plan and two scheduled navigator contacts within 30 days', 'No access to the transition-navigation offer']
outcome: Patient-reported transition score, as a share of the maximum score, from the survey sent on day 30 and closed on day 45; the score lies between 0 and 1
horizon: 30 days after discharge
intercurrent-event handling: ['Use the transition score regardless of readmission', 'Analyze the offer regardless of completed contacts', 'The protocol scores death before day 30 as the worst transition score (composite strategy)']
interference unit: Individual patient
assumption rationale: ['The recorded baseline variables cover the measured common causes', 'Given the arm and the baseline variables, survey response carries no further information about the unobserved score (missingness at random)', 'The standardized offer and version records support consistency', 'Reserved navigator capacity and access controls support no interference']
What this output tells you. The first line gives the fingerprint 1f8351d839cc8afe. The fit
of Step 6 carries the same fingerprint.
The program protocol scores a death before day 30 as the worst score (a composite strategy). This synthetic law draws no deaths, so the strategy does not change the estimate here. In a real program, a death row has an observed score of 0. Response then depends on survival, and MAR given A and W alone fails. That analysis needs a regression on survival before the response model. The point-treatment fit of this page has one response model on A and W, and it does not compute that regression.
Non-response is not an intercurrent event. It hides a score that exists. StudyProtocol has no
field for a missing-data rule. The page records the MAR argument in assumption_rationale, and
Step 5 declares the response column as a design role. The program adjustment set also names
medication burden and site factors. A real protocol keeps them.
Step 5: design and identification#
The design declares responded as the response indicator with missingness=. The rows with a
missing score stay in the data, because they belong to the target population.
from cleverly import ATE, CausalStudy, PointTreatment
study = CausalStudy(
frame,
design=PointTreatment(
outcome="transition_score",
treatment="transition_navigation",
adjustment=tuple(covariates),
missingness="responded",
),
protocol=protocol,
)
effect = study.identify(ATE(reference=0))
print(effect.summary(protocol="fingerprint"))
average treatment effect, E[Y^a] - E[Y^reference]
identified by explicit-adjustment: E_W[E(transition_score | transition_navigation=a, responded=1, W)] - E_W[E(transition_score | transition_navigation=0, responded=1, W)] for a in [1]
adjustment/history: ['discharge_risk', 'age', 'prior_utilization']
required nuisances: ['outcome_regression', 'treatment_mechanism', 'missingness_mechanism']
assumptions:
- consistency: Y = Y^a when A = a
- no interference: one unit's potential outcome does not depend on other units' treatment assignments
- no unmeasured confounding: Y^a is independent of A given W
- positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]
- missingness at random for responded: transition_score is independent of responded given (transition_navigation, W)
- response positivity for responded: P(responded = 1 | transition_navigation = a, W) > 0 almost surely wherever the target functional evaluates arm a
causal study protocol: schema 1; 1f8351d839cc8afe
What this output tells you. The formula is the identification formula of
the causal question. It fits the outcome regression among respondents
(responded=1) and averages it over all patients. The required nuisances are Q, g, and \(\pi\)
(missingness_mechanism). The assumptions match the table of the causal question. No output of
this page can check missingness at random or no unmeasured confounding.
Step 6: estimate the ATE#
setting |
value |
why |
|---|---|---|
|
boosted trees of depth 3, with early stopping |
the outcome surface has curvature and an |
|
main-effects logistic regression |
the offer depends on discharge risk and age through a smooth score |
|
main-effects logistic regression on A and W |
response depends on the arm, discharge risk, and prior utilization through a smooth score |
|
five outer folds |
the boosted trees are flexible, so in-sample predictions can overfit the rows they score |
|
the known range of the score |
the logistic fluctuation works on this range. The bounds come from the instrument, not from the observed scores |
|
fixed seed, one process |
the fit is reproducible |
In this synthetic law the two logistic models have the true form. A real analysis cannot know that. A flexible learner for g and \(\pi\) is the safer default when the sample allows it.
from cleverly import CrossFitting, ModelSpec, Runtime, Targeting, TMLEMethod
method = TMLEMethod(
models=ModelSpec(
outcome_learner=HistGradientBoostingRegressor(
max_iter=200, learning_rate=0.05, max_depth=3, early_stopping=True, random_state=71
),
treatment_learner=LogisticRegression(max_iter=1000, random_state=71),
missingness_learner=LogisticRegression(max_iter=1000, random_state=71),
),
cross_fitting=CrossFitting(n_folds=5),
targeting=Targeting(q_bounds=(0.0, 1.0)),
runtime=Runtime(random_state=71, n_jobs=1),
)
full = effect.estimate(method=method)
estimate = full["ate"]
print(full.summary(protocol="fingerprint"))
print()
print(f"estimate: {estimate.psi:.4f}")
print(f"standard error: {estimate.std_error:.4f}")
print(f"95% CI: ({estimate.ci[0]:.4f}, {estimate.ci[1]:.4f})")
print(f"population ATE: {truth['ate']:.4f}")
Targeted maximum likelihood estimation
======================================
n = 4000 (3051 with an observed outcome); covariates = 3; P(A=1) = 0.4993
causal estimand: average treatment effect, E[Y^a] - E[Y^reference]
identification: explicit-adjustment; E_W[E(transition_score | transition_navigation=a, responded=1, W)] - E_W[E(transition_score | transition_navigation=0, responded=1, W)] for a in [1]
required nuisances: outcome_regression, treatment_mechanism, missingness_mechanism
identification assumptions: consistency: Y = Y^a when A = a; no interference: one unit's potential outcome does not depend on other units' treatment assignments; no unmeasured confounding: Y^a is independent of A given W; positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]; missingness at random for responded: transition_score is independent of responded given (transition_navigation, W); response positivity for responded: P(responded = 1 | transition_navigation = a, W) > 0 almost surely wherever the target functional evaluates arm a
causal study protocol: schema 1; 1f8351d839cc8afe
outcome family: gaussian; fluctuation: logistic (iterative)
stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled
propensity truncated to [0.009532, 0.9905]; ATT/ATC to [0.025, 0.975]
P(Delta=1|A,W) truncated to [0.01, 1]
data 20a917c8c326d554 | folds b3d93054673ed5ae | cleverly 0.1.3 | 2026-10-16T01:17:18+00:00
estimand psi std_err 95% CI p_value
-------- ------- -------- ------------------ -------
ate 0.14782 0.005219 [0.13759, 0.15805] <1e-4
estimate: 0.1478
standard error: 0.0052
95% CI: (0.1376, 0.1581)
population ATE: 0.1466
What this output tells you. The header reports 4000 rows, of which 3051 have an observed
outcome. The line stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled
confirms the fold layer: every nuisance prediction is out of fold, and one fluctuation targets
all rows. The estimator clips each fitted probability into its bound before it divides by it:
[0.009532, 0.9905] for the propensity and [0.01, 1] for the response probability.
The estimate is 0.1478 with a standard error of 0.0052. The 95% interval (0.1376, 0.1581) contains
the true ATE of 0.1466. One covered interval is not evidence of coverage.
The review probe reviews/notebook-review/probes/sn-bounded/probe.py repeats this fit on 500
draws. The interval covered the ATE on 478 of 500 draws.
Step 7: what a complete-case fit estimates#
A common shortcut drops the patients who did not answer and runs the ordinary analysis. The two fits target different quantities. With \(\tau(W) = E[Y(1) - Y(0) \mid W]\) for the conditional effect:
fit |
rows |
target |
|---|---|---|
Step 6 |
all 4,000, with |
\(\psi = E[\tau(W)]\), the ATE of every eligible patient |
complete case |
respondents only, with no response model |
\(E[\tau(W) \mid \Delta = 1]\), the conditional effect averaged over the respondents’ covariates |
The two targets differ only when a predictor of response also modifies the effect. In this law, discharge risk does both. The complete-case fit drops the response model, but it still needs MAR. It conditions on \(\Delta = 1\), and response depends on the arm. Under the same assumptions, it answers a different question.
The code fits the complete-case analysis with the Step 6 method. It then computes the complete-case target for the respondents of this draw from the law’s outcome means. The study carries no protocol, because its rows no longer match the protocol’s target population.
from cleverly.datasets import missing_outcome_bounded_dgp
def complete_case_fit(data):
"""Drop the non-respondents, forget the indicator, and fit the Step 6 method."""
kept = data[data["responded"] == 1].drop(columns=["responded"])
design = PointTreatment(
outcome="transition_score",
treatment="transition_navigation",
adjustment=tuple(covariates),
)
fit = CausalStudy(kept, design=design).identify(ATE(reference=0)).estimate(method=method)
return fit, kept
def show(label, result, target):
point = result["ate"]
low, high = point.ci
covered = low <= target <= high
print(
f"{label:22s} psi={point.psi:.4f} se={point.std_error:.4f} "
f"CI=({low:.4f}, {high:.4f}) covers the ATE={covered}"
)
complete_case, respondents = complete_case_fit(frame)
show("complete cases only", complete_case, truth["ate"])
show("missingness declared", full, truth["ate"])
print(f"population ATE: {truth['ate']:.4f}")
print("rows used by the complete-case fit:", len(respondents), "of", len(frame))
# The law's outcome means give the true conditional effect at each respondent's covariates.
law = missing_outcome_bounded_dgp()
w = respondents[covariates].to_numpy()
respondent_effect = float(np.mean(law.outcome_mean(w, 1.0, None) - law.outcome_mean(w, 0.0, None)))
distance = (complete_case["ate"].psi - respondent_effect) / complete_case["ate"].std_error
print(f"complete-case target for these respondents: {respondent_effect:.4f}")
print(f"complete-case estimate minus that target, in standard errors: {distance:.1f}")
complete cases only psi=0.1722 se=0.0052 CI=(0.1621, 0.1823) covers the ATE=False
missingness declared psi=0.1478 se=0.0052 CI=(0.1376, 0.1581) covers the ATE=True
population ATE: 0.1466
rows used by the complete-case fit: 3051 of 4000
complete-case target for these respondents: 0.1711
complete-case estimate minus that target, in standard errors: 0.2
What this output tells you. The complete-case fit uses 3051 of 4000 rows. Its estimate is 0.1722, and its interval (0.1621, 0.1823) excludes the ATE of 0.1466.
The complete-case target is 0.1711 for the respondents of this draw, and 0.173 in the population. The complete-case estimate lies 0.2 standard errors from the target of this draw. So the complete-case fit estimates its own target well. That target is the wrong one for the sponsor’s question.
A review probe repeated both fits on 500 draws of this law. The complete-case interval contained the ATE on 0 of 500 draws. It contained the complete-case target of 0.173 on 480 draws (0.960). The mean complete-case estimate lay 0.00043 above that target, with a Monte Carlo SE of 0.00023.
The comparison does not show that a complete-case fit is always biased. When the effect does not vary with the predictors of response, the two targets are equal.
Step 8: diagnostics, what the data can check#
assess() collects the validation, diagnostic, and sensitivity operations for the Step 6 fit.
include_retargets=True adds the operations that reuse the cached nuisance predictions without a
refit. The arguments mapping configures the two missingness operations that Step 9 reads.
The code prints each row that ran or that runs on request. It then prints the support report and the nuisance report.
assessment = full.assess(
include_retargets=True,
arguments={
"missingness": {
"gamma": (0.0, 1.0, 2.0, 4.0, 8.0),
"arm_gamma": {0: 0.0, 1: -1.0},
},
"tipping_gamma": {
"arm_gamma": {0: 0.0, 1: -1.0},
},
},
)
ledger = assessment.to_frame()
ran = ledger[~ledger["status"].isin(["unavailable", "not_applicable"])]
print(ran[["surface", "check", "status"]].to_string(index=False))
print("needs attention:", tuple(item.name for item in assessment.attention))
support = assessment.report("support")
nuisance = assessment.report("nuisance_models")
print()
print(support.summary())
print()
print(nuisance.summary())
surface check status
validation score_equations passed
validation support completed
validation nuisance_models completed
diagnostics truncation_curve completed
diagnostics refute deferred
sensitivity missingness completed
sensitivity tipping_gamma completed
needs attention: ()
Positivity / overlap diagnostics
--------------------------------
n = 4000; propensity truncated to [0.009532, 0.9905]
g(W) quantile 0% 1% 5% 25% 50% 75% 95% 99% 100%
------------- ------ ------ ------ ------ ------ ------ ------ ------ ------
overall 0.1807 0.2516 0.3126 0.4210 0.5003 0.5807 0.6837 0.7529 0.8336
treated 0.1807 0.2680 0.3332 0.4490 0.5271 0.6019 0.7018 0.7691 0.8336
control 0.2012 0.2429 0.2985 0.3980 0.4745 0.5491 0.6546 0.7213 0.8145
arm n effective n ESS / n top 1% weight top 5% weight
------- ---- ----------- ------- ------------- -------------
treated 1997 1875.0 0.939 0.021 0.087
control 2003 1894.1 0.946 0.021 0.084
threshold P(g < t) P(g > 1-t)
--------- -------- ----------
0.01 0.0000 0.0000
0.025 0.0000 0.0000
0.05 0.0000 0.0000
0.1 0.0000 0.0000
truncated: 0 unit(s) (0.00%); most extreme untruncated g(W) = 0.16643
group equation target rows Kish-equivalent rows Kish / target Kish / total top 1% load top 5% load max |w h| zero load g bound clipped
----- -------- ----------- -------------------- ------------- ------------ ----------- ----------- --------- --------- ------------------ ---------
mean h0 3051/4000 930.0 0.305 0.232 0.098 0.249 41.19 1531 [0.009532, 0.9905] 0 (0.00%)
(absolute residual-multiplier load from the fitted equation; Kish values describe residual-multiplier concentration, not residual contributions or estimator information, influence, or precision; target rows are score-mask rows and include structural zeros)
max |clever covariate| (mean): 81.76
mechanism min 1% 5% median ESS / n top 1% weight top 5% weight clipped
---------------- ------ ------ ------ ------ ------- ------------- ------------- ---------
P(Delta=1|A,W) 0.0382 0.1302 0.2905 0.8285 0.852 0.038 0.115 0 (0.00%)
P(A=a,Delta=1|W) 0.0081 0.0457 0.1223 0.3883 0.739 0.049 0.131 0 (0.00%)
(P(Delta=1|A,W) truncated to [0.01, 1]; P(A=a,Delta=1|W) truncated to [0.009532, 0.9905] x [0.01, 1], factor by factor; each row counts both arms)
VERDICT: no truncation-driven fragility detected (0.0% of units clipped at the marginal treatment-mechanism bound [0.009532, 0.9905]). The narrowest arm has a Kish-equivalent weight count of 94% of its nominal rows. This is concentration, not estimator information, and no threshold is applied because none is derived. Absolute-load concentration is greatest for group 'mean', equation 'h0': 930.0 Kish-equivalent rows out of 3051 score-mask rows (30%; 23% of 4000 total rows). This is a descriptive residual-multiplier concentration index. It excludes the residual and is not estimator effective sample size, information, actual influence, precision, or a positivity test.
Nuisance model diagnostics
----------------------------------------
model auc brier log_loss r2 mse cal_slope cal_se reg_slope
----------- ------ ------ -------- ------ ------ --------- ------ ---------
propensity 0.6265 0.2379 0.6684 - - 0.9861 0.0724 -
missingness 0.8055 0.1373 0.4294 - - 0.9972 0.0398 -
outcome - - - 0.7421 0.0160 - - 1.0126
Calibration slope: logistic recalibration of the out-of-fold predictions with one intercept per fold.
A finding needs the 97.5% interval to lie above 0 and exclude 1, a Bonferroni level over 2 tested model(s).
VERDICT: nuisance fits look reasonable.
What this output tells you. Read the parts in order.
output part |
what it shows on this draw |
|---|---|
the status table |
the score-equation check |
the first support tables |
the fitted propensity ranges from 0.1807 to 0.8336, and no unit is truncated |
the |
the fitted response probability has a minimum of 0.0382. The product of the arm and response probabilities, \(P(A = a, \Delta = 1 \mid W)\), has a minimum of 0.0081 |
the nuisance report |
out-of-fold measures: an outcome |
Positivity here is a property of the product \(g \cdot \pi\), not of either factor. A patient with a middle chance of the offer and a low chance of response still has a small product. The clever covariate divides by that product, and its largest absolute value here is 81.76. The table minimum of 0.0081 also counts each patient’s other arm, so it can lie below 1/81.76.
The nuisance measures are out of fold, so they describe prediction on rows each model did not see. They do not show that a model is correctly specified. The support report describes the fitted g and \(\pi\). It cannot show that they are the true mechanisms. These outputs are implementation checks. They are not evidence for the identification assumptions.
Step 9: sensitivity, what the data cannot check#
The data cannot test missingness at random. This step asks instead how far one stated departure must go to change the conclusion. The missingness curve is a pattern-mixture analysis. Within each arm and covariate profile, it shifts the mean of the unobserved scores away from the respondents’ mean. Missingness tilt and tipping gamma defines both operations.
element |
meaning |
|---|---|
|
the shift added to the logit of the fitted respondents’ mean. The declared |
|
the usual-support arm stays at MAR. A positive |
|
each row reuses the targeted outcome model of |
tipping gamma |
the |
|
they reuse the standard error of the fit, so they ignore uncertainty about |
The tilt mixes the two means with the fitted response probability. The estimate therefore moves by
the mean, over all patients, of \(1 - \hat\pi(1, W)\) times the shift of the unobserved navigation-arm
mean. The code divides each move by the mean of \(1 - \hat\pi(1, W)\). The result is a mean in score
units, weighted by the chance of non-response under navigation. A gamma of 30 puts every
unobserved navigation-arm mean at nearly 0, the floor of the scale.
from cleverly.sensitivity import missingness_tilt
missingness_curve = assessment.report("missingness")
tipping_gamma = assessment.report("tipping_gamma")
ci_tipping_gamma = full.sensitivity.tipping_gamma(arm_gamma={0: 0.0, 1: -1.0}, use_ci=True)
print(missingness_curve.round(4).to_string(index=False))
print()
print("tipping gamma, the estimate reaches 0:", tipping_gamma)
print(f"tipping gamma, the interval reaches 0: {ci_tipping_gamma:.3f}")
# The fitted response probability under navigation, clipped as the fit clips it.
response_under_navigation = full.nuisance.bounded_missingness(0.01)[:, full.nuisance.arms.index(1)]
nonresponse = float(np.mean(1.0 - response_under_navigation))
# Gamma = 30 moves every unobserved navigation-arm mean to nearly 0, the floor of the scale.
at = missingness_tilt(
full, [ci_tipping_gamma, 30.0], estimands=["ate"], arm_gamma={0: 0.0, 1: -1.0}
)
ci_move, floor_move = full["ate"].psi - at["psi"].to_numpy()
ci_mean_shift = ci_move / nonresponse
mar_mean = floor_move / nonresponse
print(f"mean non-response probability under navigation: {nonresponse:.3f}")
print(f"MAR mean of the unobserved navigation-arm scores: {mar_mean:.3f}")
print(f"mean shift at the interval tipping gamma: {ci_mean_shift:.3f}")
print(f"estimate when those scores sit at the floor: {at['psi'].iloc[1]:.4f}")
gamma estimand psi std_err ci_lower ci_upper is_mar gamma[0] gamma[1]
0.0 ate 0.1478 0.0052 0.1376 0.1581 True 0.0 -0.0
1.0 ate 0.1057 0.0052 0.0955 0.1159 False 0.0 -1.0
2.0 ate 0.0635 0.0052 0.0533 0.0737 False 0.0 -2.0
4.0 ate 0.0174 0.0052 0.0072 0.0277 False 0.0 -4.0
8.0 ate 0.0037 0.0052 -0.0065 0.0139 False 0.0 -8.0
tipping gamma, the estimate reaches 0: None
tipping gamma, the interval reaches 0: 4.921
mean non-response probability under navigation: 0.200
MAR mean of the unobserved navigation-arm scores: 0.727
mean shift at the interval tipping gamma: 0.688
estimate when those scores sit at the floor: 0.0025
What this output tells you. The gamma = 0 row reproduces the Step 6 estimate, 0.1478. The
estimate falls as gamma rises, to 0.0174 at gamma = 4 and 0.0037 at gamma = 8. It does not
reach 0 inside the search range, so the point tipping gamma is None. The interval reaches 0 at a
gamma of 4.921.
quantity |
value |
|---|---|
the MAR mean of the unobserved navigation-arm scores, weighted by non-response |
0.727 |
the mean shift at which the interval reaches 0 |
0.688 |
the estimate when every unobserved navigation-arm score sits at the floor of 0 |
0.0025 |
The score is bounded, so this departure has a worst case. On this draw, even if every navigation-arm patient who did not answer scored 0, the estimate would stay just above 0. The interval reaches 0 only when those patients score, on average, 0.688 below their MAR prediction of 0.727. Each value comes from the fitted outcome model, so it carries that model’s error.
This tilt keeps usual support at MAR. A departure that also raised the unobserved usual-support scores would reach 0 sooner. The curve does not show why patients did not answer. The analyst must decide whether a departure of that size is plausible for this survey.
A review probe repeated this fit on 500 draws. The interval reached 0 on 485 draws. Between the 5th and 95th percentiles, its tipping gamma ran from 3.17 to 5.48. The estimate itself reached 0 inside the search range on 358 draws. So the margin of 0.0025 at the floor is a property of this draw. On this law, the worst case lies near 0.
Method evidence#
The fits on this page use one draw of a synthetic law. The registered study in the table measures the method over repeated draws. The study page gives its law, its nuisance models, and the result of each cell.
fit on this page |
registered study |
law of the study |
|---|---|---|
Step 6, CV-TMLE with missing outcomes |
another law |
No registered study fits a misspecified learner on continuous covariates, as Step 6 does.
A review probe is a script under reviews/notebook-review/probes/. The step readings quote its
results. A review probe is not a registered study.
The technical entry lists the evidence and the limits of the method.
What the result means#
question |
answer for the Step 6 fit |
|---|---|
what the estimate means |
if every eligible patient got the offer, the mean transition score would be 0.1478 higher on the 0 to 1 scale than under usual support for all |
population and intervention |
every eligible patient of the protocol, respondent or not. The offer at the discharge-home order, against usual support |
uncertainty |
the 95% Wald interval (0.1376, 0.1581), from the estimated influence curve of the cross-fitted fit |
evidence |
the registered study of Method evidence, and the review probes of Steps 6, 7, and 9. The Step 8 outputs are implementation checks |
unverified assumptions |
missingness at random, no unmeasured confounding, consistency, and no interference |
not supported |
an effect among respondents, a claim that MAR holds, an estimate for a program with deaths before day 30, and a general claim about the estimator from this draw |
The strongest assumption stays untestable. Non-response that the experience itself causes breaks MAR. An example is a patient who ignores the survey because navigation failed. Step 9 shows how large that departure must be: the interval reaches 0 only when such patients score, on average, 0.688 below comparable respondents. Report the estimate as conditional on an argument about the survey process.
The response model reads only baseline variables. The fit cannot condition response on a variable recorded after the offer, such as contact attempts, or on survival. Each needs a sequential regression over that variable, which this point-treatment fit does not compute.
Where to go next#
When the outcome is an event over time, censoring takes the place of non-response. Time-to-event outcomes asks a readmission question with death as a competing event and disenrollment as censoring.
The examples index lists every tutorial in the program.