{ "cells": [ { "cell_type": "markdown", "id": "title", "metadata": {}, "source": [ "# CV-TMLE: flexible learners and navigator teams\n", "\n", "## The situation\n", "\n", "A regional network offers transition navigation to adults who go home from hospital. At the\n", "discharge-home order, staff record four baseline variables. A patient then receives the\n", "navigation offer or usual discharge support. Thirty days later, the patient reports a transition\n", "score. Navigator teams deliver the offer.\n", "\n", "| item | in this program |\n", "| --- | --- |\n", "| unit | one discharge home of an adult patient |\n", "| time zero | the discharge-home order |\n", "| baseline covariates $W$, measured before time zero | `discharge_risk`, `prior_utilization`, `medication_burden`, `age` |\n", "| treatment $A$, set at time zero | `transition_navigation`: 1 for the offer, 0 for usual support |\n", "| outcome $Y$, measured at day 30 | `transition_score`, a share of the maximum score |\n", "| counterfactual outcomes | $Y^1$, the score if the patient receives the offer, and $Y^0$, the score under usual support. The data show at most one of them for each patient |\n", "\n", "Staff do not assign the offer at random. Patients with different baseline variables receive the\n", "offer at different rates.\n", "\n", "The data are synthetic, so each law has a known true effect. The page compares estimates with\n", "that truth only to teach. A real program has no truth. Steps 2 to 6 use a law with no teams, which\n", "isolates one problem. Step 7 uses a second law with teams.\n", "\n", "## The causal graph\n", "\n", "```text\n", " ┌──────────────────────┐\n", " │ ▼\n", " W ────────► A ───────► Y ◄──── U (navigator team: Step 7 law only)\n", " ▲ │\n", " └───────────┘ U changes the size of the A → Y effect\n", "```\n", "\n", "| node or edge | what it asserts | what the analysis does with it |\n", "| --- | --- | --- |\n", "| $W \\to A$ and $W \\to Y$ | each baseline variable changes the chance of an offer and the score | the four variables are confounders, so the analysis adjusts for all four |\n", "| $A \\to Y$ | the offer can change the score | the estimand measures this effect |\n", "| $U \\to Y$, and $U$ modifies $A \\to Y$ | a team's unrecorded practice shifts the score and the benefit of the offer | $U$ is an effect modifier that the patients of one team share. Their outcomes are then dependent, so Step 7 makes the team the independent unit |\n", "| no $U \\to A$ | team practice does not change who receives an offer | $U$ is not a confounder, so adjustment for $W$ still identifies the effect. With this arrow, $U$ would confound, and no cluster declaration removes that bias |\n", "| no arrow from one patient's $A$ to another patient's $Y$ | one patient's offer does not change another patient's score | this is the no-interference assumption |\n", "| no hidden common cause of $A$ and $Y$ | the four recorded variables block every back-door path | this is the no-unmeasured-confounding assumption. Data cannot verify it. Step 10 measures how strong a violation must be to change the answer |\n", "\n", "Readmission and death occur after time zero, so they are not in $W$. Adjustment for a\n", "post-treatment variable would remove part of the effect. The protocol uses the score regardless of\n", "readmission, and it scores death before day 30 as the worst score. Every patient in these data\n", "reports a score, so the graph has no censoring node.\n", "\n", "## The decisions on this page\n", "\n", "| decision the problem forces | the step that makes it |\n", "| --- | --- |\n", "| which estimand answers the sponsor, and which assumptions identify it | the question and the estimand |\n", "| whether flexible learners need cross-fitting, and what it leaves unsolved | why this method, Steps 5 and 6 |\n", "| which unit is independent when patients share navigator teams | Step 7 |\n", "| how to report an estimate that depends on a random fold split | Step 8 |\n", "| what the diagnostics and the sensitivity analysis can and cannot check | Steps 9 and 10 |" ] }, { "cell_type": "markdown", "id": "plan", "metadata": {}, "source": [ "## The question and the estimand\n", "\n", "The program sponsor asks how much the mean 30-day transition score would change if every eligible\n", "discharge received the offer, compared with usual support for every discharge. The answer decides\n", "whether the network extends the offer to all eligible patients.\n", "\n", "| part | definition |\n", "| --- | --- |\n", "| target population | adults with a discharge-home order at a participating hospital |\n", "| intervention | the offer for every eligible discharge, against usual support for every eligible discharge |\n", "| counterfactual outcome | $Y^a$, the 30-day score under strategy $a$ |\n", "| estimand | the average treatment effect (ATE), $\\psi = E[Y^1] - E[Y^0]$ |\n", "\n", "An estimand is the number the question asks for. Write it before you choose any model.\n", "[Estimands](../user-guide/estimands.md) lists the other questions `cleverly` can ask. Simpler\n", "quantities answer different questions.\n", "\n", "| simpler quantity | why it does not answer the sponsor |\n", "| --- | --- |\n", "| the mean score of offered patients minus the mean score of the others | the two groups differ in $W$, so the difference mixes the effect with the difference in baseline variables |\n", "| the effect among patients who received the offer | it describes the patients whom staff selected, not a network-wide offer |\n", "| the effect in one team | it describes one team. Step 7 shows that teams differ |\n", "\n", "The four assumptions in the next table identify the ATE. Then\n", "$\\psi = E_W[\\bar Q(1, W) - \\bar Q(0, W)]$, where $\\bar Q(a, W) = E[Y \\mid A = a, W]$. Every\n", "estimator on this page targets this one quantity.\n", "\n", "| assumption | what it requires here | what can check it |\n", "| --- | --- | --- |\n", "| consistency | the observed score of an offered patient is $Y^1$, so the offer has one version | scientific judgment. The protocol records the version |\n", "| no interference | one patient's offer does not change another patient's score | scientific judgment. The protocol cites reserved navigator capacity |\n", "| no unmeasured confounding | $Y^a$ is independent of $A$ given $W$ | scientific judgment. Step 10 measures sensitivity to a violation |\n", "| positivity | every kind of patient has a nonzero chance of each arm | the data, in part. Step 9 describes the fitted propensity model, not the population |\n", "\n", "## Why this method\n", "\n", "| feature of the problem | method choice | what the choice still needs |\n", "| --- | --- | --- |\n", "| one decision at time zero, one outcome at a fixed horizon, no censoring | point-treatment TMLE. Longitudinal and survival methods do not apply | the four assumptions above |\n", "| a static contrast of everyone offered against no one offered | the ATE. A stochastic or shift intervention would answer a different policy question | positivity in both arms |\n", "| $\\bar Q$ and the propensity $g(W) = P(A = 1 \\mid W)$ are nonlinear, with interactions | flexible learners for both nuisances | the product of the two nuisance errors must shrink faster than $1/\\sqrt{n}$ |\n", "| flexible learners can fail the complexity condition of in-sample proofs | cross-fitting, which is CV-TMLE | one nuisance fit for each fold |\n", "| a score between 0 and 1 | the declared support `q_bounds=(0.0, 1.0)` | a cross-fitted continuous outcome needs it |\n", "| patients share navigator teams, and team practice modifies the effect | the `cluster=` role: whole teams in each fold, and team totals in the variance | enough teams. Step 7 states the rule |\n", "\n", "No outcome is missing, so the page uses no missing-outcome method. The question has one binary\n", "contrast, so the page fits no marginal structural model.\n", "\n", "An interval proof that reuses observations controls an empirical-process term with a complexity\n", "condition, such as a Donsker condition. Boosted trees need not satisfy it, even when they predict\n", "well. With cross-fitting, each patient's nuisance prediction comes from a model that never saw\n", "that patient. That removes the complexity condition. It does not repair a slow learner, poor\n", "support, or confounding. The\n", "[CV-TMLE reference](../technical-reference/cv-tmle.md#what-this-solves) lists the conditions that\n", "remain.\n", "\n", "| term | plain meaning |\n", "| --- | --- |\n", "| nuisance | a model the estimate needs but the question does not ask about. Here, $\\bar Q$ and $g$. [Point-treatment TMLE](../technical-reference/point-treatment-tmle.md) defines both |\n", "| targeting | a small update to the outcome model, weighted by the treatment model, that removes first-order bias. [Methods and learners](../user-guide/methods-learners.md#targeting-and-bounds) defines it |\n", "| outer fold | one part of the sample. The nuisance models for its rows train on the other parts |\n", "| influence curve | how much each row moves the estimate. Its variance gives the standard error. [Inference](../technical-reference/inference.md) defines it |\n", "| cluster | a group of rows that are not independent of each other. Here, one navigator team |" ] }, { "cell_type": "markdown", "id": "setup-heading", "metadata": {}, "source": [ "## Step 1: set up\n", "\n", "Every fit below passes its learners, fold count, and seed explicitly, so a rerun reproduces the\n", "stored outputs." ] }, { "cell_type": "code", "execution_count": 1, "id": "setup", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:45.900971Z", "iopub.status.busy": "2026-10-16T01:16:45.900655Z", "iopub.status.idle": "2026-10-16T01:16:46.978946Z", "shell.execute_reply": "2026-10-16T01:16:46.977599Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cleverly 0.1.3\n" ] } ], "source": [ "import pandas as pd\n", "from sklearn.ensemble import HistGradientBoostingClassifier, HistGradientBoostingRegressor\n", "from sklearn.linear_model import LogisticRegression\n", "\n", "import cleverly\n", "\n", "pd.set_option(\"display.width\", 130)\n", "pd.set_option(\"display.max_columns\", 20)\n", "print(\"cleverly\", cleverly.__version__)" ] }, { "cell_type": "markdown", "id": "setup-reading", "metadata": {}, "source": [ "**What this output tells you.** The stored outputs in this notebook came from the version named\n", "above. A different version can print different numbers." ] }, { "cell_type": "markdown", "id": "data-heading", "metadata": {}, "source": [ "## Step 2: the data\n", "\n", "`navigation_data` draws the rows from the generator `make_nonlinear_bounded` under the program's\n", "column names. The code prints the column names, the first rows, and the true ATE." ] }, { "cell_type": "code", "execution_count": 2, "id": "data", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:46.980756Z", "iopub.status.busy": "2026-10-16T01:16:46.980480Z", "iopub.status.idle": "2026-10-16T01:16:47.562627Z", "shell.execute_reply": "2026-10-16T01:16:47.562057Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rows and columns: (3000, 6)\n", "['transition_score', 'transition_navigation', 'discharge_risk', 'prior_utilization', 'medication_burden', 'age']\n", " transition_score transition_navigation discharge_risk prior_utilization medication_burden age\n", "0 0.454 0.0 -0.040 -1.258 2.574 0.482\n", "1 0.791 1.0 0.644 -0.208 0.058 0.337\n", "2 0.735 1.0 0.195 -0.609 0.615 0.490\n", "3 0.295 0.0 -0.118 -0.719 -0.879 -1.008\n", "4 0.642 1.0 0.277 -0.351 0.871 -0.888\n", "\n", "population ATE: 0.163\n" ] } ], "source": [ "from cleverly.datasets import navigation_data\n", "\n", "frame, truth = navigation_data(n=3_000, seed=34)\n", "print(\"rows and columns:\", frame.shape)\n", "print(list(frame.columns))\n", "print(frame.head().round(3))\n", "print()\n", "print(f\"population ATE: {truth['ate']:.3f}\")" ] }, { "cell_type": "markdown", "id": "data-reading", "metadata": {}, "source": [ "**What this output tells you.** Each row is one discharge. The four covariates are independent\n", "standard-normal draws.\n", "\n", "| feature of the law | what it means for this page |\n", "| --- | --- |\n", "| $\\bar Q$ and $g$ are nonlinear, and the effect varies with `discharge_risk` and `prior_utilization` | a main-terms model cannot represent either nuisance, so the page uses flexible learners |\n", "| the true propensity lies between 5% and 95% | every patient has at least a 5% chance of each arm |\n", "| the score lies between 0 and 1 | the support belongs to the instrument. Step 5 declares it |\n", "| the law draws each patient independently, with no team | Steps 5 and 6 isolate the data-reuse problem. Step 7 uses a second law with teams |\n", "\n", "The true ATE is 0.163." ] }, { "cell_type": "markdown", "id": "protocol-heading", "metadata": {}, "source": [ "## Step 3: write the protocol\n", "\n", "This page asks the question of the [shared study design](index.md#the-shared-study-design). It\n", "changes only the estimator, so the code uses `navigation_protocol()` unchanged." ] }, { "cell_type": "code", "execution_count": 3, "id": "protocol", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:47.564060Z", "iopub.status.busy": "2026-10-16T01:16:47.563795Z", "iopub.status.idle": "2026-10-16T01:16:47.566453Z", "shell.execute_reply": "2026-10-16T01:16:47.566048Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "causal study protocol: schema 1; 623fc2c240615d26\n", "target population: Adults with a discharge-home order at a participating hospital during the enrollment period\n", "eligibility: ['Age 18 years or older', 'Discharge home ordered at a participating hospital']\n", "time zero: Discharge-home order, after baseline measurement and before the navigation offer\n", "treatment strategies: ['Offer standard transition navigation', 'Provide usual discharge support']\n", "treatment versions: ['Bedside transition plan and two scheduled navigator contacts within 30 days', 'No access to the transition-navigation offer']\n", "outcome: Patient-reported transition score, as a share of the maximum score\n", "horizon: 30 days after discharge\n", "intercurrent-event handling: ['Use the transition score regardless of readmission', 'Analyze the offer regardless of completed contacts', 'The protocol scores death before day 30 as the worst transition score (composite strategy)']\n", "interference unit: Individual patient\n", "assumption rationale: ['The recorded baseline variables cover the measured common causes', 'The standardized offer and version records support consistency', 'Reserved navigator capacity and access controls support no interference']\n" ] } ], "source": [ "from cleverly.datasets import navigation_protocol\n", "\n", "protocol = navigation_protocol()\n", "print(\"\\n\".join(protocol.summary_lines()))" ] }, { "cell_type": "markdown", "id": "protocol-reading", "metadata": {}, "source": [ "**What this output tells you.** The fingerprint is `623fc2c240615d26`. The\n", "[point-treatment tutorial](point-treatment-tmle.ipynb) prints the same fingerprint, because the two\n", "pages record the same design.\n", "\n", "The interference unit is the individual patient. A shared team is a statistical dependence, not\n", "interference. `StudyProtocol` has no field for that dependence, so Step 7 declares it through the\n", "design's `cluster=` role. A cluster declaration cannot repair interference. If patients compete\n", "for navigator slots, one patient's assignment changes another patient's outcome, and the estimand\n", "itself changes." ] }, { "cell_type": "markdown", "id": "identify-heading", "metadata": {}, "source": [ "## Step 4: design and identification\n", "\n", "The design names the column roles, and the estimand names the contrast." ] }, { "cell_type": "code", "execution_count": 4, "id": "identify", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:47.567472Z", "iopub.status.busy": "2026-10-16T01:16:47.567356Z", "iopub.status.idle": "2026-10-16T01:16:47.572649Z", "shell.execute_reply": "2026-10-16T01:16:47.572188Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "average treatment effect, E[Y^a] - E[Y^reference]\n", "identified by explicit-adjustment: E_W[E(transition_score | transition_navigation=a, W)] - E_W[E(transition_score | transition_navigation=0, W)] for a in [1]\n", "adjustment/history: ['discharge_risk', 'prior_utilization', 'medication_burden', 'age']\n", "required nuisances: ['outcome_regression', 'treatment_mechanism']\n", "assumptions:\n", " - consistency: Y = Y^a when A = a\n", " - no interference: one unit's potential outcome does not depend on other units' treatment assignments\n", " - no unmeasured confounding: Y^a is independent of A given W\n", " - positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]\n", "causal study protocol: schema 1; 623fc2c240615d26\n" ] } ], "source": [ "from cleverly import ATE, CausalStudy, PointTreatment\n", "\n", "study = CausalStudy(\n", " frame,\n", " design=PointTreatment(\n", " outcome=\"transition_score\",\n", " treatment=\"transition_navigation\",\n", " adjustment=(\"discharge_risk\", \"prior_utilization\", \"medication_burden\", \"age\"),\n", " ),\n", " protocol=protocol,\n", ")\n", "effect = study.identify(ATE(reference=0))\n", "print(effect.summary(protocol=\"fingerprint\"))" ] }, { "cell_type": "markdown", "id": "identify-reading", "metadata": {}, "source": [ "**What this output tells you.** The summary prints the observed-data formula of the question\n", "section, the two required nuisances, and the four assumptions. Nothing in it names folds or\n", "learners. Those belong to the method, so a change of estimator cannot change the question." ] }, { "cell_type": "markdown", "id": "estimate-heading", "metadata": {}, "source": [ "## Step 5: estimate with cross-fitting\n", "\n", "| setting | value | why |\n", "| --- | --- | --- |\n", "| `outcome_learner` | gradient boosting | $\\bar Q$ is nonlinear, with interactions |\n", "| `treatment_learner` | a shallow, regularized booster | $g$ is nonlinear. Depth 2, a small learning rate, and an L2 penalty keep the fitted propensities from being too extreme |\n", "| `CrossFitting(n_folds=5)` | five outer folds | each row's predictions come from models fit on the other four folds. The package default is ten. Five keeps this page fast |\n", "| `Targeting(q_bounds=(0.0, 1.0))` | the declared score support | fixes the outcome scale before any fold is drawn |\n", "| `Runtime(random_state=34, n_jobs=1)` | a fixed seed, one process | makes the fold draw and the fit reproducible |\n", "\n", "Declare `q_bounds` for a cross-fitted continuous outcome. Without it, the outcome scale would come\n", "from all observed scores. Each fold's model would then use a scale that the held-out rows helped\n", "set. `cleverly` refuses that fit. The name `declared_support` carries one support to every fit\n", "below.\n", "\n", "Do not use the default `HistGradientBoostingClassifier` as a treatment learner below 10,000 rows.\n", "Below that size, scikit-learn does not stop it early. The\n", "[rules for code cells](../development/example-notebooks.md#rules-for-code-cells) give the\n", "consequence." ] }, { "cell_type": "code", "execution_count": 5, "id": "estimate", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:47.573840Z", "iopub.status.busy": "2026-10-16T01:16:47.573613Z", "iopub.status.idle": "2026-10-16T01:16:49.955038Z", "shell.execute_reply": "2026-10-16T01:16:49.953822Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Targeted maximum likelihood estimation\n", "======================================\n", "n = 3000; covariates = 4; P(A=1) = 0.4597\n", "causal estimand: average treatment effect, E[Y^a] - E[Y^reference]\n", "identification: explicit-adjustment; E_W[E(transition_score | transition_navigation=a, W)] - E_W[E(transition_score | transition_navigation=0, W)] for a in [1]\n", "required nuisances: outcome_regression, treatment_mechanism\n", "identification assumptions: consistency: Y = Y^a when A = a; no interference: one unit's potential outcome does not depend on other units' treatment assignments; no unmeasured confounding: Y^a is independent of A given W; positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]\n", "causal study protocol: schema 1; 623fc2c240615d26\n", "outcome family: gaussian; fluctuation: logistic (iterative)\n", "stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled\n", "propensity truncated to [0.0114, 0.9886]; ATT/ATC to [0.025, 0.975]\n", "data e25f7a021cacfaa6 | folds c7f4eafaaf319249 | cleverly 0.1.3 | 2026-10-16T01:16:49+00:00\n", "\n", "estimand psi std_err 95% CI p_value\n", "-------- ------- -------- ------------------ -------\n", "ate 0.15731 0.005444 [0.14664, 0.16798] <1e-4 \n", "\n", "population ATE: 0.163\n" ] } ], "source": [ "from cleverly import CrossFitting, ModelSpec, Runtime, Targeting, TMLEMethod\n", "\n", "\n", "def shallow_booster():\n", " \"\"\"A shallow, regularized boosted propensity learner.\"\"\"\n", " return HistGradientBoostingClassifier(\n", " max_depth=2,\n", " learning_rate=0.05,\n", " max_iter=200,\n", " l2_regularization=1.0,\n", " random_state=34,\n", " )\n", "\n", "\n", "boosted = ModelSpec(\n", " outcome_learner=HistGradientBoostingRegressor(random_state=34),\n", " treatment_learner=shallow_booster(),\n", ")\n", "declared_support = Targeting(q_bounds=(0.0, 1.0))\n", "cross_fitted = effect.estimate(\n", " method=TMLEMethod(\n", " models=boosted,\n", " cross_fitting=CrossFitting(n_folds=5),\n", " targeting=declared_support,\n", " runtime=Runtime(random_state=34, n_jobs=1),\n", " )\n", ")\n", "print(cross_fitted.summary(protocol=\"fingerprint\"))\n", "print()\n", "print(f\"population ATE: {truth['ate']:.3f}\")" ] }, { "cell_type": "markdown", "id": "estimate-reading", "metadata": {}, "source": [ "**What this output tells you.** The construction line reads\n", "`stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled`. The fit\n", "cross-fits both nuisances and then fits one targeting update on all rows.\n", "\n", "The line `propensity truncated to [0.0114, 0.9886]` gives the default bound,\n", "$5 / (\\sqrt{n} \\ln n)$ at $n = 3000$. Step 9 shows that no unit reaches it.\n", "\n", "The estimate is 0.15731, with a standard error of 0.005444. The 95% interval\n", "[0.14664, 0.16798] contains the true ATE of 0.163 on this draw.\n", "\n", "The Step 2 law draws patients independently, so this interval treats patients as independent.\n", "Step 7 shows the declaration that a program with teams needs.\n", "\n", "A review probe, `reviews/notebook-review/probes/cross-fitting-final/sweep.py`, repeats this fit on\n", "200 draws of the same law. The interval contained the true ATE on 188 of them. The mean standard\n", "error was 0.93 times the standard deviation of the estimates. The probe is not a registered study." ] }, { "cell_type": "markdown", "id": "failure-mode-heading", "metadata": {}, "source": [ "## Step 6: compare with in-sample nuisance fits\n", "\n", "This comparison shows why the page cross-fits. The second fit uses the same learners and the same\n", "declared support, with `CrossFitting(enabled=False)`. That selects ordinary TMLE.\n", "\n", "| item | in-sample TMLE | CV-TMLE |\n", "| --- | --- | --- |\n", "| estimand | the ATE of Step 4 | the same ATE |\n", "| what the interval needs beyond the four assumptions | the product-rate condition, and a complexity condition on the learners | the product-rate condition |\n", "\n", "The expected mechanism is overfitting. The boosted $\\bar Q$ fits the rows that it later predicts.\n", "Its residuals on those rows are then too small, so the influence-curve variance is too small. Data\n", "reuse can also add a bias term that the targeting step does not remove.\n", "\n", "The code prints both fits, whether each interval contains the true ATE, and the distance from the\n", "truth in each fit's own standard errors." ] }, { "cell_type": "code", "execution_count": 6, "id": "failure-mode", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:49.957575Z", "iopub.status.busy": "2026-10-16T01:16:49.957160Z", "iopub.status.idle": "2026-10-16T01:16:50.476994Z", "shell.execute_reply": "2026-10-16T01:16:50.476179Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TMLE (in-sample nuisances): nuisances fitted in-sample (cross_fit=False)\n", " psi std_err ci_lower ci_upper covers truth\n", "no cross-fitting 0.1553 0.0040 0.1476 0.1631 True\n", "cross-fitted 0.1573 0.0054 0.1466 0.1680 True\n", "in-sample SE / cross-fitted SE: 0.73\n", "in-sample distance from the truth, in its standard errors: 1.9\n", "cross-fitted distance from the truth, in its standard errors: 1.0\n", "population ATE: 0.163\n" ] } ], "source": [ "in_sample = effect.estimate(\n", " method=TMLEMethod(\n", " models=boosted,\n", " cross_fitting=CrossFitting(enabled=False),\n", " targeting=declared_support,\n", " runtime=Runtime(random_state=34, n_jobs=1),\n", " )\n", ")\n", "\n", "\n", "def summary_line(result, marker):\n", " \"\"\"The first line of the result summary that contains ``marker``.\"\"\"\n", " return next(line.strip() for line in result.summary().splitlines() if marker in line)\n", "\n", "\n", "def side_by_side(fits, target=None, columns=(\"psi\", \"std_err\", \"ci_lower\", \"ci_upper\")):\n", " \"\"\"One row per labelled fit, and whether each interval contains ``target`` when given.\"\"\"\n", " table = pd.concat({label: fitted.to_frame() for label, fitted in fits.items()})\n", " table = table.droplevel(1)[list(columns)]\n", " if target is not None:\n", " table[\"covers truth\"] = table[\"ci_lower\"].le(target) & table[\"ci_upper\"].ge(target)\n", " return table\n", "\n", "\n", "print(summary_line(in_sample, \": nuisances \"))\n", "comparison = side_by_side(\n", " {\"no cross-fitting\": in_sample, \"cross-fitted\": cross_fitted}, truth[\"ate\"]\n", ")\n", "print(comparison.round(4))\n", "se_ratio = in_sample[\"ate\"].std_error / cross_fitted[\"ate\"].std_error\n", "print(f\"in-sample SE / cross-fitted SE: {se_ratio:.2f}\")\n", "for label, fitted in ((\"in-sample\", in_sample), (\"cross-fitted\", cross_fitted)):\n", " miss = abs(truth[\"ate\"] - fitted[\"ate\"].psi) / fitted[\"ate\"].std_error\n", " print(f\"{label} distance from the truth, in its standard errors: {miss:.1f}\")\n", "print(f\"population ATE: {truth['ate']:.3f}\")" ] }, { "cell_type": "markdown", "id": "failure-mode-reading", "metadata": {}, "source": [ "**What this output tells you.** The first line confirms that the new fit used in-sample\n", "nuisances. On this draw, the in-sample standard error is 0.73 times the cross-fitted one. Both\n", "intervals contain the true ATE. The truth lies 1.9 in-sample standard errors from the in-sample\n", "estimate, and 1.0 cross-fitted standard errors from the cross-fitted estimate.\n", "\n", "One draw cannot show a coverage rate. The review probe repeats both fits on 200 draws of the same\n", "law.\n", "\n", "| over 200 draws | in-sample TMLE | CV-TMLE |\n", "| --- | --- | --- |\n", "| intervals that contained the true ATE | 153 | 188 |\n", "| mean standard error divided by the standard deviation of the estimates | 0.70 | 0.93 |\n", "| mean estimate minus the true ATE | -0.0024 | 0.0003 |\n", "\n", "The in-sample interval missed on about one draw in four. A calibrated 95% interval misses on about\n", "one draw in 20. The probe shows both parts of the mechanism. The in-sample standard error is too\n", "small, and the in-sample estimate sits slightly low. On 90% of the draws, the in-sample standard\n", "error was between 0.68 and 0.73 times the cross-fitted one.\n", "\n", "The registered\n", "[stacked CV-TMLE study](../technical-reference/method-evidence/stacked-point-treatment-cv-tmle.md#theory-properties)\n", "fits this same law 400 times at $n = 500$, with an unpruned regression tree for $\\bar Q$. Its\n", "in-sample control covers 0.5225 of the time, and its cross-fitted fit covers 0.945. Neither the\n", "study nor the probe shows coverage for other learners or other laws." ] }, { "cell_type": "markdown", "id": "team-clusters-heading", "metadata": {}, "source": [ "## Step 7: declare navigator teams\n", "\n", "The program has 200 navigator teams with 15 patients each. Team practice, such as supervision and\n", "local follow-up, changes how much a patient gains from the offer. The graph shows this as $U$.\n", "\n", "`make_clustered` draws a second synthetic law with this structure. It has its own two covariates,\n", "`team_law_w1` and `team_law_w2`, and a binary outcome: 1 when a patient reports the top box of the\n", "transition score. The estimand of this step is therefore the ATE on the top-box outcome, a risk\n", "difference in this teaching law. It is not the Step 2 ATE on the score. In this law, the\n", "team factor does not change who receives an offer. About 46% of teams have a negative mean effect\n", "of the offer (review probe `reviews/notebook-review/probes/cross-fitting-final/truth.py`).\n", "\n", "| item | patients as units | teams as units |\n", "| --- | --- | --- |\n", "| estimand | the top-box risk difference in the population of teams and their patients | the same risk difference |\n", "| folds | patients split at random | whole teams in each fold |\n", "| variance | the variance of patient influence values | the variance of team totals of the influence values, across teams |\n", "\n", "The expected mechanism is shared effect modification. Patients of one team share $U$, so their\n", "influence values move together. A variance that treats patients as independent then understates\n", "the uncertainty.\n", "\n", "Both fits cross-fit, with logistic learners for both nuisances. The true propensity of this law is\n", "a main-terms logistic model in the two covariates. The fitted $\\bar Q$ is misspecified, because\n", "the unrecorded $U$ enters the true outcome model through an interaction with $A$. The estimate\n", "therefore relies on the correct propensity model. The outcome is binary, so its support is known\n", "and needs no declaration. The protocol changes only the outcome field, and the code prints the new\n", "fingerprint." ] }, { "cell_type": "code", "execution_count": 7, "id": "team-clusters", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:50.482960Z", "iopub.status.busy": "2026-10-16T01:16:50.482742Z", "iopub.status.idle": "2026-10-16T01:16:50.669260Z", "shell.execute_reply": "2026-10-16T01:16:50.668322Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " psi std_err ci_lower ci_upper covers truth folds one team spans\n", "patients as units 0.1237 0.0195 0.0856 0.1619 True 5\n", "teams as units 0.1256 0.0301 0.0667 0.1845 True 1\n", "teams SE / patients SE: 1.54\n", "navigator teams: 200\n", "population ATE: 0.104\n", "team protocol fingerprint: 7df34d840b4e8d1a\n", "team protocol outcome: Top-box patient-reported transition score\n" ] } ], "source": [ "from dataclasses import replace\n", "\n", "from cleverly.datasets import make_clustered\n", "\n", "team_frame, team_truth = make_clustered(n=3_000, seed=34, cluster_size=15, family=\"binomial\")\n", "team_frame = team_frame.rename(\n", " columns={\n", " \"Y\": \"transition_top_box\",\n", " \"A\": \"transition_navigation\",\n", " \"W1\": \"team_law_w1\",\n", " \"W2\": \"team_law_w2\",\n", " \"cluster\": \"navigator_team\",\n", " }\n", ")\n", "team_protocol = replace(protocol, outcome=\"Top-box patient-reported transition score\")\n", "simple = TMLEMethod(\n", " models=ModelSpec(\n", " outcome_learner=LogisticRegression(max_iter=1000, random_state=34),\n", " treatment_learner=LogisticRegression(max_iter=1000, random_state=34),\n", " ),\n", " cross_fitting=CrossFitting(n_folds=5),\n", " runtime=Runtime(random_state=34, n_jobs=1),\n", ")\n", "patients_design = PointTreatment(\n", " outcome=\"transition_top_box\",\n", " treatment=\"transition_navigation\",\n", " adjustment=(\"team_law_w1\", \"team_law_w2\"),\n", ")\n", "teams_design = replace(patients_design, cluster=\"navigator_team\")\n", "\n", "ignoring = (\n", " CausalStudy(team_frame, design=patients_design, protocol=team_protocol)\n", " .identify(ATE(reference=0))\n", " .estimate(method=simple)\n", ")\n", "clustered = (\n", " CausalStudy(team_frame, design=teams_design, protocol=team_protocol)\n", " .identify(ATE(reference=0))\n", " .estimate(method=simple)\n", ")\n", "\n", "\n", "def folds_spanned(result):\n", " \"\"\"The largest number of outer folds that one navigator team spans in ``result``.\"\"\"\n", " labels = pd.Series(result.split_plan.assignments[0], index=team_frame.index)\n", " return labels.groupby(team_frame[\"navigator_team\"]).nunique().max()\n", "\n", "\n", "team_comparison = side_by_side(\n", " {\"patients as units\": ignoring, \"teams as units\": clustered}, team_truth[\"ate\"]\n", ")\n", "team_comparison[\"folds one team spans\"] = [folds_spanned(ignoring), folds_spanned(clustered)]\n", "print(team_comparison.round(4))\n", "team_se_ratio = clustered[\"ate\"].std_error / ignoring[\"ate\"].std_error\n", "print(f\"teams SE / patients SE: {team_se_ratio:.2f}\")\n", "print(\"navigator teams:\", clustered.data.n_clusters)\n", "print(f\"population ATE: {team_truth['ate']:.3f}\")\n", "print(\"team protocol fingerprint:\", team_protocol.fingerprint)\n", "print(\"team protocol outcome:\", team_protocol.outcome)" ] }, { "cell_type": "markdown", "id": "team-clusters-reading", "metadata": {}, "source": [ "**What this output tells you.** The two point estimates are 0.1237 and 0.1256. They differ only\n", "because the team declaration regroups the folds. Declaring teams makes the standard error 1.54\n", "times larger on this draw. Both intervals contain the true ATE of 0.104 on this draw. The team\n", "protocol's fingerprint is `7df34d840b4e8d1a`, and its outcome field names the top-box score.\n", "\n", "The fold count shows the other change. Without the declaration, one team spans 5 outer folds. A\n", "patient's nuisance model can then train on outcomes from the patient's own team. Those training\n", "rows are not independent of the predicted row, which is the independence that cross-fitting\n", "uses. With it, each team\n", "spans 1 fold.\n", "\n", "Both intervals target the same population risk difference. The patient-level variance ignores\n", "the variation of the effect between teams, so it understates the uncertainty. The registered\n", "[clustered CV-TMLE study](../technical-reference/method-evidence/clustered-point-treatment-cv-tmle.md#theory-properties)\n", "fits this law with teams of 10. Its independent-row variance covers the ATE 0.84125 of the time,\n", "and its cluster-robust variance covers 0.9475. The design effect grows with team size, so the\n", "independent-row coverage for teams of 15 is not the study's number.\n", "\n", "The cluster variance includes the factor $m/(m-1)$ for $m$ teams.\n", "[Clusters](../technical-reference/inference.md#clusters) gives the rules for the number of teams.\n", "\n", "| number of teams $m$ | what the fit reports |\n", "| --- | --- |\n", "| 40 or more | a normal-reference interval |\n", "| 10 to 39 | a $t$-reference interval. [Clusters](../technical-reference/inference.md#clusters) gives its degrees of freedom |\n", "| fewer than 10 | a point estimate only |\n", "| fewer than the requested folds | fewer folds, with a warning |\n", "\n", "This fit has 200 teams, so it reports a normal-reference interval." ] }, { "cell_type": "markdown", "id": "repeated-folds-heading", "metadata": {}, "source": [ "## Step 8: the fold draw\n", "\n", "The fold split is random, and a different split gives a slightly different estimate. Report that\n", "dependence rather than pick a convenient split. `repeats=` runs the complete estimator on several\n", "fold draws. It reports the median estimate, and its variance includes the spread between draws." ] }, { "cell_type": "code", "execution_count": 8, "id": "repeated-folds", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:50.671244Z", "iopub.status.busy": "2026-10-16T01:16:50.670649Z", "iopub.status.idle": "2026-10-16T01:16:57.614087Z", "shell.execute_reply": "2026-10-16T01:16:57.613574Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(median over 3 independent draws of the split; variance includes within-draw uncertainty and split dispersion)\n", " psi std_err ci_lower ci_upper\n", "median of 3 draws 0.1567 0.0055 0.1459 0.1674\n", "estimand n_repeats standard_deviation reported_standard_error ratio_to_standard_error\n", " ate 3 0.0004 0.0055 0.0679\n" ] } ], "source": [ "repeated = effect.estimate(\n", " method=TMLEMethod(\n", " models=boosted,\n", " cross_fitting=CrossFitting(n_folds=5, repeats=3),\n", " targeting=declared_support,\n", " runtime=Runtime(random_state=34, n_jobs=1),\n", " )\n", ")\n", "print(summary_line(repeated, \"median over\"))\n", "print(side_by_side({\"median of 3 draws\": repeated}).round(4))\n", "repeated_nuisance = repeated.diagnostics.run_all().report(\"nuisance_models\")\n", "print(repeated_nuisance.repeat_spread_frame().round(4).to_string(index=False))" ] }, { "cell_type": "markdown", "id": "repeated-folds-reading", "metadata": {}, "source": [ "**What this output tells you.** The median over 3 draws is 0.1567, with a standard error of\n", "0.0055. The single Step 5 draw had a standard error of 0.005444.\n", "\n", "The spread row gives a standard deviation of 0.0004 across the three draws, which is 0.0679 of the\n", "reported standard error. On this sample, the fold draw moves the estimate much less than the\n", "sampling uncertainty does. Three draws estimate that spread imprecisely, and the report defines no\n", "threshold for the ratio. The\n", "[repeated CV-TMLE evidence](../technical-reference/method-evidence/repeated-cross-fitting.md)\n", "gives the repeated-sampling results for the median report." ] }, { "cell_type": "markdown", "id": "assessment-heading", "metadata": {}, "source": [ "## Step 9: diagnostics\n", "\n", "The assessment of the Step 5 fit lists the checks that ran and the rows that need attention. The\n", "code then prints the support, nuisance-model, and score-equation reports. The last line gives the\n", "support detail of the in-sample fit of Step 6." ] }, { "cell_type": "code", "execution_count": 9, "id": "assessment", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:57.615557Z", "iopub.status.busy": "2026-10-16T01:16:57.615310Z", "iopub.status.idle": "2026-10-16T01:16:57.644481Z", "shell.execute_reply": "2026-10-16T01:16:57.644046Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Returned results\n", "----------------\n", "surface operation result \n", "----------- ------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n", "validation support maximum truncated fraction 0.0%; minimum effective-sample-size ratio 82.5%; group load: mean:h1 1137.7/3000 Kish-equivalent mask rows (37.9%; 37.9% all; draw 01 of 01); not estimator ESS\n", "validation nuisance_models 2 nuisance model report(s) are available \n", "sensitivity omitted_confounding at the default strengths; cf_y=0.03, cf_d=0.03, rho=1; bias-adjusted interval [0.1486, 0.166] \n", "sensitivity robustness_value point robustness value 0.4187; confidence-limit value 0.398 \n", "sensitivity elements sigma2=0.01809, nu2=4.536, max_bias=0.2865 \n", "sensitivity contour grid 20 x 20; cf_d range [0, 0.15]; cf_y range [0, 0.15]; value range [0.1107, 0.1573]; inspect the retained frame \n", "sensitivity evalue point=3.092, limit=2.936, source scale=mean difference; approximate conversion \n", "\n", "Checks\n", "------\n", "status count operations \n", "------ ----- --------------------------\n", "passed 1 validation.score_equations\n", "\n", "Not run\n", "-------\n", "status count operations \n", "-------------- ----- ---------------------------------\n", "deferred 4 diagnostics.truncation_curve \n", " diagnostics.refute \n", " sensitivity.benchmark \n", " sensitivity.simulated_confounding\n", "not_applicable 3 diagnostics.corrections \n", " sensitivity.missingness \n", " sensitivity.tipping_gamma \n", "\n", "Full ledger: call to_frame(). Next steps: call next_steps().\n", "Retained payloads: call report(...).\n", "needs attention: ()\n", "\n", "Positivity / overlap diagnostics\n", "--------------------------------\n", "n = 3000; propensity truncated to [0.0114, 0.9886]\n", "\n", "g(W) quantile 0% 1% 5% 25% 50% 75% 95% 99% 100% \n", "------------- ------ ------ ------ ------ ------ ------ ------ ------ ------\n", "overall 0.0239 0.1185 0.1930 0.3521 0.4663 0.5683 0.7164 0.7987 0.8649\n", "treated 0.0579 0.1711 0.2548 0.4145 0.5193 0.6091 0.7649 0.8107 0.8649\n", "control 0.0239 0.0954 0.1662 0.3095 0.4227 0.5242 0.6543 0.7703 0.8510\n", "\n", "arm n effective n ESS / n top 1% weight top 5% weight\n", "------- ---- ----------- ------- ------------- -------------\n", "treated 1379 1137.7 0.825 0.037 0.122 \n", "control 1621 1454.9 0.898 0.030 0.102 \n", "\n", "threshold P(g < t) P(g > 1-t)\n", "--------- -------- ----------\n", "0.01 0.0000 0.0000 \n", "0.025 0.0003 0.0000 \n", "0.05 0.0010 0.0000 \n", "0.1 0.0063 0.0000 \n", "\n", "truncated: 0 unit(s) (0.00%); most extreme untruncated g(W) = 0.02391\n", "\n", "group equation target rows Kish-equivalent rows Kish / target Kish / total top 1% load top 5% load max |w h| zero load g bound clipped \n", "----- -------- ----------- -------------------- ------------- ------------ ----------- ----------- --------- --------- ---------------- ---------\n", "mean h1 3000/3000 1137.7 0.379 0.379 0.066 0.214 17.28 1621 [0.0114, 0.9886] 0 (0.00%)\n", "(absolute residual-multiplier load from the fitted equation; Kish values describe residual-multiplier concentration, not residual contributions or estimator information, influence, or precision; target rows are score-mask rows and include structural zeros)\n", "max |clever covariate| (mean): 17.28\n", "\n", "VERDICT: no truncation-driven fragility detected (0.0% of units clipped at the marginal treatment-mechanism bound [0.0114, 0.9886]). The narrowest arm has a Kish-equivalent weight count of 83% of its nominal rows. This is concentration, not estimator information, and no threshold is applied because none is derived. Absolute-load concentration is greatest for group 'mean', equation 'h1': 1137.7 Kish-equivalent rows out of 3000 score-mask rows (38%; 38% of 3000 total rows). This is a descriptive residual-multiplier concentration index. It excludes the residual and is not estimator effective sample size, information, actual influence, precision, or a positivity test.\n", "\n", "Nuisance model diagnostics\n", "----------------------------------------\n", "model auc brier log_loss r2 mse cal_slope cal_se reg_slope\n", "---------- ------ ------ -------- ------ ------ --------- ------ ---------\n", "propensity 0.6737 0.2259 0.6427 - - 0.9611 0.0624 - \n", "outcome - - - 0.6653 0.0181 - - 0.9650 \n", "Calibration slope: logistic recalibration of the out-of-fold predictions with one intercept per fold.\n", "A finding needs the 95.0% interval to lie above 0 and exclude 1, a Bonferroni level over 1 tested model(s).\n", "\n", "VERDICT: nuisance fits look reasonable.\n", "\n", "Score-equation check\n", "--------------------\n", "target kind |score| before threshold ratio ok \n", "------ --------------- --------- --------- --------- -------- ---\n", "mean fluctuation 0.000e+00 4.084e-03 9.939e-08 0.00e+00 yes\n", "ate influence curve 0.000e+00 - 9.939e-08 0.00e+00 yes\n", "\n", "PASS: the targeting step solved the estimated efficient score equation.\n", "\n", "smallest estimated g(W), before truncation: 0.0239\n", "truncated: 0 units (0.00%)\n", "treated: 1379 rows, effective sample size 1137.7 (82.5%)\n", "control: 1621 rows, effective sample size 1454.9 (89.8%)\n", "\n", "in-sample support: maximum truncated fraction 0.0%; minimum effective-sample-size ratio 89.6%; group load: mean:h1 1235.0/3000 Kish-equivalent mask rows (41.2%; 41.2% all; draw 01 of 01); not estimator ESS\n" ] } ], "source": [ "assessment = cross_fitted.assess()\n", "print(assessment.summary())\n", "print(\"needs attention:\", tuple(item.name for item in assessment.attention))\n", "support = assessment.report(\"support\")\n", "nuisance = assessment.report(\"nuisance_models\")\n", "scores = assessment.report(\"score_equations\")\n", "print()\n", "print(support.summary())\n", "print()\n", "print(nuisance.summary())\n", "print()\n", "print(scores.summary())\n", "print()\n", "smallest = support.propensity_quantiles[\"overall\"][0.0]\n", "print(f\"smallest estimated g(W), before truncation: {smallest:.4f}\")\n", "truncated = support.truncated\n", "print(f\"truncated: {truncated['count']:.0f} units ({truncated['fraction']:.2%})\")\n", "for arm, ess in support.effective_sample_size.items():\n", " ratio = f\"{ess['effective']:.1f} ({ess['ratio']:.1%})\"\n", " print(f\"{arm}: {ess['n']:.0f} rows, effective sample size {ratio}\")\n", "print()\n", "print(\"in-sample support:\", in_sample.diagnostics.run_all()[\"support\"].detail)" ] }, { "cell_type": "markdown", "id": "assessment-reading", "metadata": {}, "source": [ "**What this output tells you.** The `Returned results` rows are calculations that ran, not\n", "passes.\n", "\n", "| output part | what it shows on this draw |\n", "| --- | --- |\n", "| `Checks` and `needs attention` | the score-equation check passed, and no row needs attention |\n", "| nuisance model diagnostics | the out-of-fold calibration slope of the propensity is 0.9611 |\n", "| support fields | the smallest fitted $g(W)$ is 0.0239. No unit sits at the truncation bound. The treated arm keeps an effective-sample-size ratio of 82.5% |\n", "| in-sample support | the in-sample fit reports a minimum effective-sample-size ratio of 89.6% |\n", "\n", "The calibration slope comes from a logistic recalibration of the out-of-fold propensities. A slope\n", "of 1 means that their spread matches the observed offer rates. On 90% of the probe draws, it lay\n", "between 0.90 and 1.01. The effective-sample-size ratio is the share of rows that the weights are\n", "worth. `cleverly` derives no threshold for it, so the report gives 82.5% no grade.\n", "\n", "These reports describe the fitted propensity model, not the treatment law. The fitted model puts a\n", "share of 0.0063 of the rows below 0.1, in the `P(g < t)` column. In the law, 1.9% of patients have\n", "a true propensity below 0.1 (review probe `truth.py`). The fitted share fell below the law's share\n", "on 196 of the 200 probe draws. A calibration slope near 1 does not show that the model sees the\n", "tail.\n", "\n", "The in-sample fit reports more even weights, from the same assignment law. Its minimum ratio was\n", "higher on all 200 probe draws. The overfitted\n", "propensities sit closer to the observed arms. A better in-sample diagnostic is therefore no reason\n", "to prefer the in-sample fit." ] }, { "cell_type": "markdown", "id": "sensitivity-heading", "metadata": {}, "source": [ "## Step 10: sensitivity to unmeasured confounding\n", "\n", "No fold construction can check the no-unmeasured-confounding assumption. The omitted-variable\n", "analysis asks how strong a hidden confounder would have to be to move the estimate to zero.\n", "\n", "| term | plain meaning |\n", "| --- | --- |\n", "| `cf_y` | the share of the remaining outcome variation that a hidden confounder explains |\n", "| `cf_d` | the share of the Riesz representer's second moment that only the hidden confounder adds |\n", "| `rho` | how closely the confounder's two effects align. `rho=1` is the worst case |\n", "| robustness value | the equal `cf_y` and `cf_d` that moves the point estimate to zero at `rho=1` |\n", "| $\\nu^2$ | the second moment of the Riesz representer. For the ATE it is $E[1 / \\{g(1 - g)\\}]$ |\n", "\n", "The bound scales with $\\sqrt{\\sigma^2 \\nu^2}$, where $\\sigma^2$ is the residual outcome variance.\n", "$\\nu^2$ is finite because the true propensity lies between 5% and 95%. In this law it is about\n", "4.83 (review probe `truth.py`). The code uses the default doubly robust estimate of $\\nu^2$ and the\n", "default strengths, `cf_y = cf_d = 0.03`. Those strengths are not calibrated to this program. The\n", "[point-treatment tutorial](point-treatment-tmle.ipynb#step-10-sensitivity-what-the-fit-cannot-check)\n", "calibrates them against a recorded covariate." ] }, { "cell_type": "code", "execution_count": 10, "id": "sensitivity", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T01:16:57.645957Z", "iopub.status.busy": "2026-10-16T01:16:57.645694Z", "iopub.status.idle": "2026-10-16T01:16:57.649235Z", "shell.execute_reply": "2026-10-16T01:16:57.648809Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sigma^2 (residual outcome variance): 0.018093\n", "nu^2 (doubly robust): 4.5363\n", "robustness value: 0.419\n", "robustness value at the 95% limit: 0.398\n", "\n", "Omitted-variable sensitivity for 'ate'\n", "--------------------------------------------\n", "estimate 0.15731; bias scale sqrt(sigma^2 nu^2) = 0.28648\n", "assumed confounding: cf_y = 0.03, cf_d = 0.03, rho = 1 -> bias <= 0.0087264\n", "bias-adjusted bounds: [0.14859, 0.16604]\n", "with 95% one-sided CIs: [0.13962, 0.17499] (the sign of the effect survives)\n", "\n", "robustness value RV = 0.4187: a confounder with cf_y = cf_d = 41.9% (the share of the residual outcome variation it explains, and the share of the Riesz representer's second moment only it adds) would move the estimate to 0.\n", "robustness value RVa = 0.3980: the same, for the 95% confidence bound.\n" ] } ], "source": [ "sensitivity = cross_fitted.sensitivity\n", "elements = sensitivity.elements(estimand=\"ate\")\n", "robustness = sensitivity.robustness_value()\n", "bounds = sensitivity.omitted_confounding()\n", "print(f\"sigma^2 (residual outcome variance): {elements.sigma2:.6f}\")\n", "print(f\"nu^2 (doubly robust): {elements.nu2:.4f}\")\n", "print(f\"robustness value: {robustness['rv']:.3f}\")\n", "print(f\"robustness value at the 95% limit: {robustness['rva']:.3f}\")\n", "print()\n", "print(bounds.summary())" ] }, { "cell_type": "markdown", "id": "sensitivity-reading", "metadata": {}, "source": [ "**What this output tells you.** The robustness value is 0.419. A confounder with `cf_y` and\n", "`cf_d` both equal to 0.419 would move the point estimate to zero. The value at the 95% limit is\n", "0.398. At the default strengths, the one-sided 95% limits of the bias-adjusted bounds are\n", "[0.13962, 0.17499], and they exclude zero.\n", "\n", "The fitted $\\nu^2$ is 4.5363, below the law's 4.83. The expected value of the doubly robust estimate is\n", "the law's $\\nu^2$ minus the mean squared error of the fitted Riesz representer. It therefore\n", "reads low when the fitted propensity has an error, and Step 9 shows that it misses part of the\n", "tail. On the 200 probe\n", "draws, its mean was 4.50. A $\\nu^2$ that is 7% low makes the bias scale about 3% low, so the\n", "robustness value reads slightly high. At the law's $\\nu^2$, this draw gives a robustness value of\n", "0.409 rather than 0.419 (review probe `summarize.py`).\n", "\n", "These numbers hold under the omitted-variable model, its worst-case alignment, and the fitted\n", "propensity. They do not show that no stronger confounder exists. The\n", "[omitted-variable bounds](../technical-reference/validation-methods.md#omitted-variable-bounds-robustness-value-benchmark-and-contours)\n", "section defines each quantity." ] }, { "cell_type": "markdown", "id": "method-evidence", "metadata": {}, "source": [ "## Method evidence\n", "\n", "The fits on this page use one draw of a synthetic law. The registered studies in the table\n", "measure each method over repeated draws. Each study page gives its law, its nuisance models,\n", "and the result of each cell.\n", "\n", "| fit on this page | registered study | law of the study |\n", "| --- | --- | --- |\n", "| Steps 5 and 6, cross-fitted and in-sample fits | [Stacked point-treatment CV-TMLE](../technical-reference/method-evidence/stacked-point-treatment-cv-tmle.md) | this page's Step 2 law |\n", "| Step 7, navigator teams | [Clustered point-treatment CV-TMLE](../technical-reference/method-evidence/clustered-point-treatment-cv-tmle.md) | the Step 7 team law |\n", "\n", "No registered study uses the boosted learners or the five folds of this page.\n", "\n", "A review probe is a script under `reviews/notebook-review/probes/`. The step readings quote its\n", "results. A review probe is not a registered study.\n", "\n", "The [technical entry](../technical-reference/cv-tmle.md#validation-issues-special-to-this-method) lists the evidence and the limits of the method." ] }, { "cell_type": "markdown", "id": "interpret", "metadata": {}, "source": [ "## Interpret the result\n", "\n", "| question | answer |\n", "| --- | --- |\n", "| what the estimate means | in the Step 2 law, the estimated effect of an offer to every eligible discharge is 0.15731 of the maximum score, against usual support for every discharge |\n", "| population and intervention | adults with a discharge-home order at a participating hospital. The intervention is the offer, analyzed regardless of completed contacts. The estimate is not the effect of completed navigation |\n", "| uncertainty | the 95% Wald interval [0.14664, 0.16798], from the cross-fitted influence curve. It treats patients as independent, which holds only because the Step 2 law has no teams |\n", "| with teams | the program's patients share teams, so its primary analysis must declare them. In the Step 7 law, the team-level interval of the top-box risk difference is [0.0667, 0.1845]. Without the declaration, the registered study's interval covered 0.84125 of the time |\n", "| evidence for the interpretation | the registered studies of [Method evidence](#method-evidence), the review probe of Steps 5 and 6, the diagnostics of Step 9, and the sensitivity analysis of Step 10 |\n", "| assumptions that stay unverified | consistency, no interference, no unmeasured confounding, positivity in the population, and the product-rate condition on the learners |\n", "| conclusions the analysis does not support | that every patient or team benefits. In the Step 7 law, about 46% of teams have a negative mean effect. Also, that this interval covers at 95% for other learners or laws, or that no stronger hidden confounder exists |" ] }, { "cell_type": "markdown", "id": "where-next", "metadata": {}, "source": [ "## Where to go next\n", "\n", "Cross-fitting addresses only data reuse. If you doubt one nuisance model, read the\n", "[product-rate condition](../technical-reference/cv-tmle.md#what-this-solves). The interval needs\n", "it. A declared known assignment mechanism removes it. [DR-TMLE](../technical-reference/dr-tmle/index.md) addresses it only\n", "in [the listed cases](../technical-reference/dr-tmle/theorem.md#the-conditioning-terms-are-linear-in-one-error).\n", "If you doubt which baseline variables belong in the assignment model, read\n", "[collaborative TMLE](collaborative-tmle.ipynb).\n", "\n", "The [examples index](index.md#the-program) lists every tutorial in the program." ] } ], "metadata": { "cleverly_execution": { "command": [ "python", "scripts/execute_notebook.py", "docs/examples/cross-fitting.ipynb" ], "gated": { "code_source_sha256": "49caaea5d88314261f9f4306de7ff84b71ca5bb51026cb0f0eb1b3768376fd78", "execution_payload_sha256": "0ee8379a29e4113460a90fd8ab112b761fee1191e500ed83860ff34b7ddb0382" }, "recorded": { "cleverly_commit": "4cea0d1ad32abd00580cd9f3a27d84a8c294db5f", "cleverly_version": "0.1.3", "cleverly_worktree_clean": false, "dependency_lock_sha256": "2e4f7ad3264ff9b35cf7588e5c93a0e20fc5c0852092ac1da365491543836682", "generator_files": [ "scripts/execute_notebook.py", "tests/notebooks.py" ], "generator_sha256": "f247771ab390b23e7e092d7eb35794e241c2ac84fc0526695df1cee1fc457a83", "package_source_sha256": "5fce085883c189351f5a0d142b216264208c189d01a46bb372a5e8fc68eed390" }, "schema_version": 5 }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.16" } }, "nbformat": 4, "nbformat_minor": 5 }