{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "title",
   "metadata": {},
   "source": [
    "# Survey non-response: the patients who never answered\n",
    "\n",
    "A hospital program offers transition navigation at discharge. On day 30 it sends a survey that\n",
    "scores how well the transition went, and the survey closes on day 45. About a quarter of patients\n",
    "never return it. The program sponsor asks for the effect of navigation on the score of **every\n",
    "eligible patient**, including the patients who did not answer.\n",
    "\n",
    "This notebook builds that analysis from the question to the interpretation. It reuses the\n",
    "navigation program of the [point-treatment tutorial](point-treatment-tmle.ipynb) and adds one\n",
    "problem: the outcome is missing for some patients.\n",
    "\n",
    "## The situation\n",
    "\n",
    "| element | in this program |\n",
    "| --- | --- |\n",
    "| population | adults with a discharge-home order at a participating hospital |\n",
    "| time zero | the discharge-home order, after the baseline record and before the offer |\n",
    "| baseline variables $W$ | `discharge_risk`, `age`, and `prior_utilization`, recorded before time zero |\n",
    "| treatment $A$ | `transition_navigation`: 1 for the navigation offer, 0 for usual discharge support |\n",
    "| response indicator $\\Delta$ | `responded`: 1 when the survey comes back by day 45 |\n",
    "| outcome $Y$ | `transition_score`, a share of the maximum score, between 0 and 1. The data hold it only when $\\Delta = 1$ |\n",
    "| counterfactual outcome $Y(a)$ | the score that the patient would report under arm $a$. No row records it under both arms, and no row records it when $\\Delta = 0$ |\n",
    "\n",
    "The data are synthetic, so the true effect is known. A real program has no such truth. Each\n",
    "comparison with the truth below is a teaching device, not a check that a real analysis can run.\n",
    "\n",
    "## The causal graph\n",
    "\n",
    "The graph lists every arrow of the synthetic law, `missing_outcome_bounded_dgp()` in\n",
    "`cleverly.datasets`. W1 is `discharge_risk`, W2 is `age`, and W3 is `prior_utilization`.\n",
    "\n",
    "```text\n",
    "W1 -> A     W2 -> A\n",
    "W1 -> Y     W2 -> Y     W3 -> Y     A -> Y\n",
    "W1 -> Δ     W3 -> Δ     A -> Δ\n",
    "```\n",
    "\n",
    "| arrow | what it asserts | what the analysis does with it |\n",
    "| --- | --- | --- |\n",
    "| `W1 -> A`, `W2 -> A` | discharge risk and age change the chance of an offer | W1 and W2 confound the effect of A on Y. The treatment model $g(W) = P(A = 1 \\mid W)$ adjusts for them |\n",
    "| `W1 -> Y`, `W2 -> Y`, `W3 -> Y` | each baseline variable changes the score | the outcome regression $Q(A, W)$ uses all three |\n",
    "| `A -> Y` | navigation changes the score | this is the effect to estimate. Its size falls as W1 rises, so W1 also modifies the effect |\n",
    "| `W1 -> Δ`, `W3 -> Δ` | response falls as discharge risk rises, and it rises with prior utilization | W3 does not confound A and Y, but the adjustment set must contain it. Without W3, the response model omits a common cause of $\\Delta$ and Y |\n",
    "| `A -> Δ` | navigation changes who answers | the response model $\\pi(A, W) = P(\\Delta = 1 \\mid A, W)$ conditions on the arm |\n",
    "\n",
    "An arrow that the graph leaves out is also an assumption.\n",
    "\n",
    "| absent arrow | the assumption that it encodes | can the data check it? |\n",
    "| --- | --- | --- |\n",
    "| `Y -> Δ`, or a hidden common cause of $\\Delta$ and Y | missingness at random (MAR): given A and W, response carries no information about the score | no |\n",
    "| a hidden common cause of A and Y | no unmeasured confounding: given W, the offer is independent of $Y(a)$ | no |\n",
    "| an arrow from one patient to another | no interference: one patient's offer does not change another patient's score | no |\n",
    "| `W3 -> A`, `W2 -> Δ` | prior utilization does not change the offer, and age does not change response. Both hold in this law | identification does not use them, because every model adjusts for all of W |\n",
    "\n",
    "The graph has no death node, because the synthetic law draws no deaths. Step 4 states what deaths\n",
    "would change."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "plan",
   "metadata": {},
   "source": [
    "## The causal question\n",
    "\n",
    "The sponsor decides whether to offer navigation to every eligible patient. The question is how\n",
    "much the mean transition score changes if every eligible patient gets the offer, compared with\n",
    "none.\n",
    "\n",
    "| element | definition |\n",
    "| --- | --- |\n",
    "| target population | every eligible patient, whether or not the patient returns the survey |\n",
    "| intervention | the navigation offer for all ($a = 1$), against usual support for all ($a = 0$) |\n",
    "| counterfactual outcome | $Y(a)$, the score under arm $a$ |\n",
    "| estimand | the average treatment effect (ATE), $\\psi = E[Y(1)] - E[Y(0)]$, in units of the score scale |\n",
    "\n",
    "The decision applies to every eligible patient, so the estimand averages over all of them. An\n",
    "average over respondents only is a different quantity: the conditional effect standardized to the\n",
    "respondents' covariates (Step 7). It is also not an effect in one fixed subgroup. Response occurs\n",
    "after the offer, and navigation changes it (`A -> Δ`).\n",
    "\n",
    "Under the assumptions of the next table, the observed data identify the ATE:\n",
    "\n",
    "$$\\psi = E_W\\big[E(Y \\mid A = 1, \\Delta = 1, W) - E(Y \\mid A = 0, \\Delta = 1, W)\\big].$$\n",
    "\n",
    "The inner regression uses respondents only. The outer mean runs over **every** patient.\n",
    "\n",
    "| assumption | statement | support |\n",
    "| --- | --- | --- |\n",
    "| consistency | $Y = Y(a)$ when $A = a$ | the protocol defines one version of each arm. Scientific judgment |\n",
    "| no interference | one patient's offer does not change another patient's score | reserved navigator capacity. Scientific judgment |\n",
    "| no unmeasured confounding | $Y(a)$ is independent of A given W | the recorded baseline variables. Scientific judgment |\n",
    "| missingness at random | $Y(a)$ is independent of $\\Delta$ given $A = a$ and W | scientific judgment. Step 9 measures how far it must fail to change the conclusion |\n",
    "| treatment positivity | $P(A = a \\mid W) > 0$ for each arm | partly checkable. Step 8 reads the fitted probabilities |\n",
    "| response positivity | $P(\\Delta = 1 \\mid A = a, W) > 0$ for each arm | partly checkable. Step 8 reads the fitted probabilities |\n",
    "\n",
    "## Why this method\n",
    "\n",
    "| feature of the problem | the method feature that it calls for |\n",
    "| --- | --- |\n",
    "| one decision at time zero, with baseline covariates only | point-treatment TMLE. No time-varying treatment or confounding needs a longitudinal fit |\n",
    "| missing scores, with response that depends on A and W | the design declares `missingness=\"responded\"`, and the fit adds the response model $\\pi(A, W)$ |\n",
    "| two models that can each be wrong | double robustness: the estimate is consistent when Q is correct, **or** when the product $g \\cdot \\pi$ is correct for each arm |\n",
    "| an outcome surface with curvature and an effect that changes with W1 | a flexible learner for Q, a boosted tree ensemble |\n",
    "| a flexible learner | cross-fitting: each row's nuisance predictions come from models fitted without that row |\n",
    "| a score with a known range of 0 to 1 | `q_bounds=(0, 1)`, which a cross-fitted fit of a continuous outcome must declare |\n",
    "\n",
    "Targeting updates the initial outcome regression along the clever covariate\n",
    "\n",
    "$$H(A, \\Delta, W) = \\frac{\\Delta}{\\pi(A, W)}\\Big(\\frac{A}{g(W)} - \\frac{1 - A}{1 - g(W)}\\Big).$$\n",
    "\n",
    "The factor $1/\\pi$ reweights the respondents toward the whole population. After the update, the\n",
    "estimate solves the estimated efficient influence-curve equation. The interval needs the product of\n",
    "the errors of Q and of $g \\cdot \\pi$ to shrink faster than $n^{-1/2}$.\n",
    "[Missing outcomes](../technical-reference/point-treatment-tmle.md#missing-outcomes-and-controlled-direct-effects)\n",
    "gives the influence curve, the remainder, and the stacked CV-TMLE that this page fits.\n",
    "\n",
    "| term | plain meaning |\n",
    "| --- | --- |\n",
    "| [nuisance](../technical-reference/point-treatment-tmle.md) | a model that the estimate needs but the question does not ask about. Here Q, g, and $\\pi$ |\n",
    "| [targeting](../user-guide/methods-learners.md#targeting-and-bounds) | the update of Q along H that removes the first-order bias of the initial fit |\n",
    "| [influence curve](../technical-reference/inference.md) | how much each row moves the estimate. Its variance gives the standard error |\n",
    "| [cross-fitting](../technical-reference/cv-tmle.md) | each row's nuisance prediction comes from models fitted without that row |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup-heading",
   "metadata": {},
   "source": [
    "## Step 1: set up\n",
    "\n",
    "Every fit below passes its learners, its fold policy, and its random seed, so a rerun reproduces\n",
    "the stored outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "setup",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:15.012662Z",
     "iopub.status.busy": "2026-10-16T01:17:15.012461Z",
     "iopub.status.idle": "2026-10-16T01:17:16.615823Z",
     "shell.execute_reply": "2026-10-16T01:17:16.614854Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cleverly 0.1.3\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import HistGradientBoostingRegressor\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "import cleverly\n",
    "\n",
    "pd.set_option(\"display.width\", 130)\n",
    "pd.set_option(\"display.max_columns\", 20)\n",
    "print(\"cleverly\", cleverly.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** The stored outputs came from the version named above. A different\n",
    "version can print different numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "data-heading",
   "metadata": {},
   "source": [
    "## Step 2: the data\n",
    "\n",
    "The code draws 4,000 patients from `make_missing_outcome_bounded()` and renames the columns to the\n",
    "program's names. The default `strength` of 2 gives the outcome curvature, an effect that falls with\n",
    "discharge risk, and a strong link from discharge risk to response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "data",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:16.618105Z",
     "iopub.status.busy": "2026-10-16T01:17:16.617792Z",
     "iopub.status.idle": "2026-10-16T01:17:16.931708Z",
     "shell.execute_reply": "2026-10-16T01:17:16.930854Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rows and columns: (4000, 6)\n",
      "['transition_score', 'transition_navigation', 'discharge_risk', 'age', 'prior_utilization', 'responded']\n",
      "   transition_score  transition_navigation  discharge_risk    age  prior_utilization  responded\n",
      "0             0.707                    1.0           0.049  0.518             -0.110        1.0\n",
      "1             0.567                    1.0          -0.089  0.503              0.511        1.0\n",
      "2             0.560                    0.0           0.345 -1.421             -0.570        1.0\n",
      "3             0.685                    1.0          -0.307  0.740              0.045        1.0\n",
      "4             0.174                    0.0          -0.394  0.378              0.326        1.0\n",
      "\n",
      "response rate:            0.763\n",
      "missing transition score: 949 rows\n",
      "population ATE:           0.1466\n"
     ]
    }
   ],
   "source": [
    "from cleverly.datasets import make_missing_outcome_bounded\n",
    "\n",
    "program_names = {\n",
    "    \"Y\": \"transition_score\",\n",
    "    \"A\": \"transition_navigation\",\n",
    "    \"W1\": \"discharge_risk\",\n",
    "    \"W2\": \"age\",\n",
    "    \"W3\": \"prior_utilization\",\n",
    "    \"Delta\": \"responded\",\n",
    "}\n",
    "frame, truth = make_missing_outcome_bounded(n=4_000, seed=71)\n",
    "frame = frame.rename(columns=program_names)\n",
    "print(\"rows and columns:\", frame.shape)\n",
    "print(list(frame.columns))\n",
    "print(frame.head().round(3))\n",
    "print()\n",
    "print(f\"response rate:            {frame['responded'].mean():.3f}\")\n",
    "print(f\"missing transition score: {int(frame['transition_score'].isna().sum())} rows\")\n",
    "print(f\"population ATE:           {truth['ate']:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "data-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** Each row is one discharge. The response rate is 0.763, so 949 of\n",
    "the 4,000 scores are missing. A missing score is `NaN`. The true ATE for every eligible patient is\n",
    "0.1466 on the 0 to 1 scale."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "association-heading",
   "metadata": {},
   "source": [
    "## Step 3: who answers the survey\n",
    "\n",
    "Before any model, compare the respondents with the patients who did not answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "association",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:16.933706Z",
     "iopub.status.busy": "2026-10-16T01:17:16.933056Z",
     "iopub.status.idle": "2026-10-16T01:17:16.940579Z",
     "shell.execute_reply": "2026-10-16T01:17:16.939914Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean baseline covariate by response status:\n",
      "           discharge_risk    age  prior_utilization\n",
      "responded                                          \n",
      "0.0                 0.781  0.012             -0.215\n",
      "1.0                -0.229 -0.012              0.056\n"
     ]
    }
   ],
   "source": [
    "covariates = [\"discharge_risk\", \"age\", \"prior_utilization\"]\n",
    "by_response = frame.groupby(\"responded\")[covariates].mean()\n",
    "print(\"mean baseline covariate by response status:\")\n",
    "print(by_response.round(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "association-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** Patients who did not respond have a mean `discharge_risk` of\n",
    "0.781. Respondents have a mean of -0.229. Respondents also have more prior utilization, 0.056\n",
    "against -0.215. Both gaps follow the arrows `W1 -> Δ` and `W3 -> Δ`.\n",
    "\n",
    "The effect falls as discharge risk rises, and respondents have lower discharge risk. The average\n",
    "effect among respondents is therefore larger than the ATE. Step 7 measures that difference."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "protocol-heading",
   "metadata": {},
   "source": [
    "## Step 4: write the protocol\n",
    "\n",
    "A `StudyProtocol` records the scientific design before any model runs. The page starts from\n",
    "`navigation_protocol()`, the protocol of the\n",
    "[shared study design](index.md#the-shared-study-design), and changes three fields.\n",
    "\n",
    "| field | what changes here |\n",
    "| --- | --- |\n",
    "| `target_population` | it states that non-respondents belong to the population |\n",
    "| `outcome` | it names the survey window and the 0 to 1 range of the score |\n",
    "| `assumption_rationale` | it adds missingness at random for the survey response |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "protocol",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:16.941929Z",
     "iopub.status.busy": "2026-10-16T01:17:16.941769Z",
     "iopub.status.idle": "2026-10-16T01:17:16.951259Z",
     "shell.execute_reply": "2026-10-16T01:17:16.950880Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "causal study protocol: schema 1; 1f8351d839cc8afe\n",
      "target population: Adults with a discharge-home order at a participating hospital during the enrollment period, whether or not they return the survey\n",
      "eligibility: ['Age 18 years or older', 'Discharge home ordered at a participating hospital']\n",
      "time zero: Discharge-home order, after baseline measurement and before the navigation offer\n",
      "treatment strategies: ['Offer standard transition navigation', 'Provide usual discharge support']\n",
      "treatment versions: ['Bedside transition plan and two scheduled navigator contacts within 30 days', 'No access to the transition-navigation offer']\n",
      "outcome: Patient-reported transition score, as a share of the maximum score, from the survey sent on day 30 and closed on day 45; the score lies between 0 and 1\n",
      "horizon: 30 days after discharge\n",
      "intercurrent-event handling: ['Use the transition score regardless of readmission', 'Analyze the offer regardless of completed contacts', 'The protocol scores death before day 30 as the worst transition score (composite strategy)']\n",
      "interference unit: Individual patient\n",
      "assumption rationale: ['The recorded baseline variables cover the measured common causes', 'Given the arm and the baseline variables, survey response carries no further information about the unobserved score (missingness at random)', 'The standardized offer and version records support consistency', 'Reserved navigator capacity and access controls support no interference']\n"
     ]
    }
   ],
   "source": [
    "from dataclasses import replace\n",
    "\n",
    "from cleverly.datasets import navigation_protocol\n",
    "\n",
    "program = navigation_protocol()\n",
    "protocol = replace(\n",
    "    program,\n",
    "    target_population=f\"{program.target_population}, whether or not they return the survey\",\n",
    "    outcome=(\n",
    "        f\"{program.outcome}, from the survey sent on day 30 and closed on day 45; \"\n",
    "        \"the score lies between 0 and 1\"\n",
    "    ),\n",
    "    assumption_rationale=(\n",
    "        program.assumption_rationale[0],\n",
    "        \"Given the arm and the baseline variables, survey response carries no further \"\n",
    "        \"information about the unobserved score (missingness at random)\",\n",
    "        *program.assumption_rationale[1:],\n",
    "    ),\n",
    ")\n",
    "print(\"\\n\".join(protocol.summary_lines()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "protocol-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** The first line gives the fingerprint `1f8351d839cc8afe`. The fit\n",
    "of Step 6 carries the same fingerprint.\n",
    "\n",
    "The program protocol scores a death before day 30 as the worst score (a composite strategy). This\n",
    "synthetic law draws no deaths, so the strategy does not change the estimate here. In a real\n",
    "program, a death row has an observed score of 0. Response then depends on survival, and MAR given\n",
    "A and W alone fails. That analysis needs a regression on survival before the response model. The\n",
    "point-treatment fit of this page has one response model on A and W, and it does not compute that\n",
    "regression.\n",
    "\n",
    "Non-response is not an intercurrent event. It hides a score that exists. `StudyProtocol` has no\n",
    "field for a missing-data rule. The page records the MAR argument in `assumption_rationale`, and\n",
    "Step 5 declares the response column as a design role. The program adjustment set also names\n",
    "medication burden and site factors. A real protocol keeps them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "identify-heading",
   "metadata": {},
   "source": [
    "## Step 5: design and identification\n",
    "\n",
    "The design declares `responded` as the response indicator with `missingness=`. The rows with a\n",
    "missing score stay in the data, because they belong to the target population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "identify",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:16.952651Z",
     "iopub.status.busy": "2026-10-16T01:17:16.952528Z",
     "iopub.status.idle": "2026-10-16T01:17:16.958682Z",
     "shell.execute_reply": "2026-10-16T01:17:16.958295Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average treatment effect, E[Y^a] - E[Y^reference]\n",
      "identified by explicit-adjustment: E_W[E(transition_score | transition_navigation=a, responded=1, W)] - E_W[E(transition_score | transition_navigation=0, responded=1, W)] for a in [1]\n",
      "adjustment/history: ['discharge_risk', 'age', 'prior_utilization']\n",
      "required nuisances: ['outcome_regression', 'treatment_mechanism', 'missingness_mechanism']\n",
      "assumptions:\n",
      "  - consistency: Y = Y^a when A = a\n",
      "  - no interference: one unit's potential outcome does not depend on other units' treatment assignments\n",
      "  - no unmeasured confounding: Y^a is independent of A given W\n",
      "  - positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]\n",
      "  - missingness at random for responded: transition_score is independent of responded given (transition_navigation, W)\n",
      "  - response positivity for responded: P(responded = 1 | transition_navigation = a, W) > 0 almost surely wherever the target functional evaluates arm a\n",
      "causal study protocol: schema 1; 1f8351d839cc8afe\n"
     ]
    }
   ],
   "source": [
    "from cleverly import ATE, CausalStudy, PointTreatment\n",
    "\n",
    "study = CausalStudy(\n",
    "    frame,\n",
    "    design=PointTreatment(\n",
    "        outcome=\"transition_score\",\n",
    "        treatment=\"transition_navigation\",\n",
    "        adjustment=tuple(covariates),\n",
    "        missingness=\"responded\",\n",
    "    ),\n",
    "    protocol=protocol,\n",
    ")\n",
    "effect = study.identify(ATE(reference=0))\n",
    "\n",
    "print(effect.summary(protocol=\"fingerprint\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "identify-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** The formula is the identification formula of\n",
    "[the causal question](#the-causal-question). It fits the outcome regression among respondents\n",
    "(`responded=1`) and averages it over all patients. The required nuisances are Q, g, and $\\pi$\n",
    "(`missingness_mechanism`). The assumptions match the table of the causal question. No output of\n",
    "this page can check missingness at random or no unmeasured confounding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "estimate-heading",
   "metadata": {},
   "source": [
    "## Step 6: estimate the ATE\n",
    "\n",
    "| setting | value | why |\n",
    "| --- | --- | --- |\n",
    "| `outcome_learner` | boosted trees of depth 3, with early stopping | the outcome surface has curvature and an `A`-by-`discharge_risk` interaction |\n",
    "| `treatment_learner` | main-effects logistic regression | the offer depends on discharge risk and age through a smooth score |\n",
    "| `missingness_learner` | main-effects logistic regression on A and W | response depends on the arm, discharge risk, and prior utilization through a smooth score |\n",
    "| `CrossFitting(n_folds=5)` | five outer folds | the boosted trees are flexible, so in-sample predictions can overfit the rows they score |\n",
    "| `Targeting(q_bounds=(0.0, 1.0))` | the known range of the score | the logistic fluctuation works on this range. The bounds come from the instrument, not from the observed scores |\n",
    "| `Runtime(random_state=71, n_jobs=1)` | fixed seed, one process | the fit is reproducible |\n",
    "\n",
    "In this synthetic law the two logistic models have the true form. A real analysis cannot know\n",
    "that. A flexible learner for g and $\\pi$ is the safer default when the sample allows it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "estimate",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:16.960358Z",
     "iopub.status.busy": "2026-10-16T01:17:16.960145Z",
     "iopub.status.idle": "2026-10-16T01:17:18.046263Z",
     "shell.execute_reply": "2026-10-16T01:17:18.045377Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Targeted maximum likelihood estimation\n",
      "======================================\n",
      "n = 4000 (3051 with an observed outcome); covariates = 3; P(A=1) = 0.4993\n",
      "causal estimand: average treatment effect, E[Y^a] - E[Y^reference]\n",
      "identification: explicit-adjustment; E_W[E(transition_score | transition_navigation=a, responded=1, W)] - E_W[E(transition_score | transition_navigation=0, responded=1, W)] for a in [1]\n",
      "required nuisances: outcome_regression, treatment_mechanism, missingness_mechanism\n",
      "identification assumptions: consistency: Y = Y^a when A = a; no interference: one unit's potential outcome does not depend on other units' treatment assignments; no unmeasured confounding: Y^a is independent of A given W; positivity: P(transition_navigation = a | W) > 0 almost surely for every supported treatment level a in [0, 1]; missingness at random for responded: transition_score is independent of responded given (transition_navigation, W); response positivity for responded: P(responded = 1 | transition_navigation = a, W) > 0 almost surely wherever the target functional evaluates arm a\n",
      "causal study protocol: schema 1; 1f8351d839cc8afe\n",
      "outcome family: gaussian; fluctuation: logistic (iterative)\n",
      "stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled\n",
      "propensity truncated to [0.009532, 0.9905]; ATT/ATC to [0.025, 0.975]\n",
      "P(Delta=1|A,W) truncated to [0.01, 1]\n",
      "data 20a917c8c326d554 | folds b3d93054673ed5ae | cleverly 0.1.3 | 2026-10-16T01:17:18+00:00\n",
      "\n",
      "estimand  psi      std_err   95% CI              p_value\n",
      "--------  -------  --------  ------------------  -------\n",
      "ate       0.14782  0.005219  [0.13759, 0.15805]  <1e-4  \n",
      "\n",
      "estimate:        0.1478\n",
      "standard error:  0.0052\n",
      "95% CI:          (0.1376, 0.1581)\n",
      "population ATE:  0.1466\n"
     ]
    }
   ],
   "source": [
    "from cleverly import CrossFitting, ModelSpec, Runtime, Targeting, TMLEMethod\n",
    "\n",
    "method = TMLEMethod(\n",
    "    models=ModelSpec(\n",
    "        outcome_learner=HistGradientBoostingRegressor(\n",
    "            max_iter=200, learning_rate=0.05, max_depth=3, early_stopping=True, random_state=71\n",
    "        ),\n",
    "        treatment_learner=LogisticRegression(max_iter=1000, random_state=71),\n",
    "        missingness_learner=LogisticRegression(max_iter=1000, random_state=71),\n",
    "    ),\n",
    "    cross_fitting=CrossFitting(n_folds=5),\n",
    "    targeting=Targeting(q_bounds=(0.0, 1.0)),\n",
    "    runtime=Runtime(random_state=71, n_jobs=1),\n",
    ")\n",
    "full = effect.estimate(method=method)\n",
    "\n",
    "estimate = full[\"ate\"]\n",
    "print(full.summary(protocol=\"fingerprint\"))\n",
    "print()\n",
    "print(f\"estimate:        {estimate.psi:.4f}\")\n",
    "print(f\"standard error:  {estimate.std_error:.4f}\")\n",
    "print(f\"95% CI:          ({estimate.ci[0]:.4f}, {estimate.ci[1]:.4f})\")\n",
    "print(f\"population ATE:  {truth['ate']:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "estimate-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** The header reports 4000 rows, of which 3051 have an observed\n",
    "outcome. The line `stacked CV-TMLE (Levy): nuisances cross-fitted over 5 folds; targeting: pooled`\n",
    "confirms the fold layer: every nuisance prediction is out of fold, and one fluctuation targets\n",
    "all rows. The estimator clips each fitted probability into its bound before it divides by it:\n",
    "[0.009532, 0.9905] for the propensity and [0.01, 1] for the response probability.\n",
    "\n",
    "The estimate is 0.1478 with a standard error of 0.0052. The 95% interval (0.1376, 0.1581) contains\n",
    "the true ATE of 0.1466. One covered interval is not evidence of coverage.\n",
    "The review probe `reviews/notebook-review/probes/sn-bounded/probe.py` repeats this fit on 500\n",
    "draws. The interval covered the ATE on 478 of 500 draws."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "failure-heading",
   "metadata": {},
   "source": [
    "## Step 7: what a complete-case fit estimates\n",
    "\n",
    "A common shortcut drops the patients who did not answer and runs the ordinary analysis. The two fits\n",
    "target different quantities. With $\\tau(W) = E[Y(1) - Y(0) \\mid W]$ for the conditional effect:\n",
    "\n",
    "| fit | rows | target |\n",
    "| --- | --- | --- |\n",
    "| Step 6 | all 4,000, with `responded` declared | $\\psi = E[\\tau(W)]$, the ATE of every eligible patient |\n",
    "| complete case | respondents only, with no response model | $E[\\tau(W) \\mid \\Delta = 1]$, the conditional effect averaged over the respondents' covariates |\n",
    "\n",
    "The two targets differ only when a predictor of response also modifies the effect. In this law,\n",
    "discharge risk does both. The complete-case fit drops the response model, but it still needs MAR.\n",
    "It conditions on $\\Delta = 1$, and response depends on the arm. Under the same assumptions, it\n",
    "answers a different question.\n",
    "\n",
    "The code fits the complete-case analysis with the Step 6 method. It then computes the\n",
    "complete-case target for the respondents of this draw from the law's outcome means. The study\n",
    "carries no protocol, because its rows no longer match the protocol's target population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "failure-mode",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:18.048029Z",
     "iopub.status.busy": "2026-10-16T01:17:18.047677Z",
     "iopub.status.idle": "2026-10-16T01:17:18.948645Z",
     "shell.execute_reply": "2026-10-16T01:17:18.948184Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "complete cases only    psi=0.1722  se=0.0052  CI=(0.1621, 0.1823)  covers the ATE=False\n",
      "missingness declared   psi=0.1478  se=0.0052  CI=(0.1376, 0.1581)  covers the ATE=True\n",
      "population ATE: 0.1466\n",
      "rows used by the complete-case fit: 3051 of 4000\n",
      "complete-case target for these respondents: 0.1711\n",
      "complete-case estimate minus that target, in standard errors: 0.2\n"
     ]
    }
   ],
   "source": [
    "from cleverly.datasets import missing_outcome_bounded_dgp\n",
    "\n",
    "\n",
    "def complete_case_fit(data):\n",
    "    \"\"\"Drop the non-respondents, forget the indicator, and fit the Step 6 method.\"\"\"\n",
    "    kept = data[data[\"responded\"] == 1].drop(columns=[\"responded\"])\n",
    "    design = PointTreatment(\n",
    "        outcome=\"transition_score\",\n",
    "        treatment=\"transition_navigation\",\n",
    "        adjustment=tuple(covariates),\n",
    "    )\n",
    "    fit = CausalStudy(kept, design=design).identify(ATE(reference=0)).estimate(method=method)\n",
    "    return fit, kept\n",
    "\n",
    "\n",
    "def show(label, result, target):\n",
    "    point = result[\"ate\"]\n",
    "    low, high = point.ci\n",
    "    covered = low <= target <= high\n",
    "    print(\n",
    "        f\"{label:22s} psi={point.psi:.4f}  se={point.std_error:.4f}  \"\n",
    "        f\"CI=({low:.4f}, {high:.4f})  covers the ATE={covered}\"\n",
    "    )\n",
    "\n",
    "\n",
    "complete_case, respondents = complete_case_fit(frame)\n",
    "show(\"complete cases only\", complete_case, truth[\"ate\"])\n",
    "show(\"missingness declared\", full, truth[\"ate\"])\n",
    "print(f\"population ATE: {truth['ate']:.4f}\")\n",
    "print(\"rows used by the complete-case fit:\", len(respondents), \"of\", len(frame))\n",
    "\n",
    "# The law's outcome means give the true conditional effect at each respondent's covariates.\n",
    "law = missing_outcome_bounded_dgp()\n",
    "w = respondents[covariates].to_numpy()\n",
    "respondent_effect = float(np.mean(law.outcome_mean(w, 1.0, None) - law.outcome_mean(w, 0.0, None)))\n",
    "distance = (complete_case[\"ate\"].psi - respondent_effect) / complete_case[\"ate\"].std_error\n",
    "print(f\"complete-case target for these respondents: {respondent_effect:.4f}\")\n",
    "print(f\"complete-case estimate minus that target, in standard errors: {distance:.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "failure-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** The complete-case fit uses 3051 of 4000 rows. Its estimate is\n",
    "0.1722, and its interval (0.1621, 0.1823) excludes the ATE of 0.1466.\n",
    "\n",
    "The complete-case target is 0.1711 for the respondents of this draw, and 0.173 in the\n",
    "population. The complete-case estimate lies 0.2 standard errors from the target of this draw. So\n",
    "the complete-case fit estimates its own target well. That target is the wrong one for the\n",
    "sponsor's question.\n",
    "\n",
    "A review probe repeated both fits on 500 draws of this law. The complete-case interval contained\n",
    "the ATE on 0 of 500 draws. It contained the complete-case target of 0.173 on 480 draws (0.960).\n",
    "The mean complete-case estimate lay 0.00043 above that target, with a Monte Carlo SE of 0.00023.\n",
    "\n",
    "The comparison does not show that a complete-case fit is always biased. When the effect does not\n",
    "vary with the predictors of response, the two targets are equal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assessment-heading",
   "metadata": {},
   "source": [
    "## Step 8: diagnostics, what the data can check\n",
    "\n",
    "`assess()` collects the validation, diagnostic, and sensitivity operations for the Step 6 fit.\n",
    "`include_retargets=True` adds the operations that reuse the cached nuisance predictions without a\n",
    "refit. The `arguments` mapping configures the two missingness operations that Step 9 reads.\n",
    "\n",
    "The code prints each row that ran or that runs on request. It then prints the support report and\n",
    "the nuisance report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "assessment",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:18.950048Z",
     "iopub.status.busy": "2026-10-16T01:17:18.949773Z",
     "iopub.status.idle": "2026-10-16T01:17:19.190069Z",
     "shell.execute_reply": "2026-10-16T01:17:19.189535Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    surface            check    status\n",
      " validation  score_equations    passed\n",
      " validation          support completed\n",
      " validation  nuisance_models completed\n",
      "diagnostics truncation_curve completed\n",
      "diagnostics           refute  deferred\n",
      "sensitivity      missingness completed\n",
      "sensitivity    tipping_gamma completed\n",
      "needs attention: ()\n",
      "\n",
      "Positivity / overlap diagnostics\n",
      "--------------------------------\n",
      "n = 4000; propensity truncated to [0.009532, 0.9905]\n",
      "\n",
      "g(W) quantile  0%      1%      5%      25%     50%     75%     95%     99%     100%  \n",
      "-------------  ------  ------  ------  ------  ------  ------  ------  ------  ------\n",
      "overall        0.1807  0.2516  0.3126  0.4210  0.5003  0.5807  0.6837  0.7529  0.8336\n",
      "treated        0.1807  0.2680  0.3332  0.4490  0.5271  0.6019  0.7018  0.7691  0.8336\n",
      "control        0.2012  0.2429  0.2985  0.3980  0.4745  0.5491  0.6546  0.7213  0.8145\n",
      "\n",
      "arm      n     effective n  ESS / n  top 1% weight  top 5% weight\n",
      "-------  ----  -----------  -------  -------------  -------------\n",
      "treated  1997  1875.0       0.939    0.021          0.087        \n",
      "control  2003  1894.1       0.946    0.021          0.084        \n",
      "\n",
      "threshold  P(g < t)  P(g > 1-t)\n",
      "---------  --------  ----------\n",
      "0.01       0.0000    0.0000    \n",
      "0.025      0.0000    0.0000    \n",
      "0.05       0.0000    0.0000    \n",
      "0.1        0.0000    0.0000    \n",
      "\n",
      "truncated: 0 unit(s) (0.00%); most extreme untruncated g(W) = 0.16643\n",
      "\n",
      "group  equation  target rows  Kish-equivalent rows  Kish / target  Kish / total  top 1% load  top 5% load  max |w h|  zero load  g bound             clipped  \n",
      "-----  --------  -----------  --------------------  -------------  ------------  -----------  -----------  ---------  ---------  ------------------  ---------\n",
      "mean   h0        3051/4000    930.0                 0.305          0.232         0.098        0.249        41.19      1531       [0.009532, 0.9905]  0 (0.00%)\n",
      "(absolute residual-multiplier load from the fitted equation; Kish values describe residual-multiplier concentration, not residual contributions or estimator information, influence, or precision; target rows are score-mask rows and include structural zeros)\n",
      "max |clever covariate| (mean): 81.76\n",
      "\n",
      "mechanism         min     1%      5%      median  ESS / n  top 1% weight  top 5% weight  clipped  \n",
      "----------------  ------  ------  ------  ------  -------  -------------  -------------  ---------\n",
      "P(Delta=1|A,W)    0.0382  0.1302  0.2905  0.8285  0.852    0.038          0.115          0 (0.00%)\n",
      "P(A=a,Delta=1|W)  0.0081  0.0457  0.1223  0.3883  0.739    0.049          0.131          0 (0.00%)\n",
      "(P(Delta=1|A,W) truncated to [0.01, 1]; P(A=a,Delta=1|W) truncated to [0.009532, 0.9905] x [0.01, 1], factor by factor; each row counts both arms)\n",
      "\n",
      "VERDICT: no truncation-driven fragility detected (0.0% of units clipped at the marginal treatment-mechanism bound [0.009532, 0.9905]). The narrowest arm has a Kish-equivalent weight count of 94% of its nominal rows. This is concentration, not estimator information, and no threshold is applied because none is derived. Absolute-load concentration is greatest for group 'mean', equation 'h0': 930.0 Kish-equivalent rows out of 3051 score-mask rows (30%; 23% of 4000 total rows). This is a descriptive residual-multiplier concentration index. It excludes the residual and is not estimator effective sample size, information, actual influence, precision, or a positivity test.\n",
      "\n",
      "Nuisance model diagnostics\n",
      "----------------------------------------\n",
      "model        auc     brier   log_loss  r2      mse     cal_slope  cal_se  reg_slope\n",
      "-----------  ------  ------  --------  ------  ------  ---------  ------  ---------\n",
      "propensity   0.6265  0.2379  0.6684    -       -       0.9861     0.0724  -        \n",
      "missingness  0.8055  0.1373  0.4294    -       -       0.9972     0.0398  -        \n",
      "outcome      -       -       -         0.7421  0.0160  -          -       1.0126   \n",
      "Calibration slope: logistic recalibration of the out-of-fold predictions with one intercept per fold.\n",
      "A finding needs the 97.5% interval to lie above 0 and exclude 1, a Bonferroni level over 2 tested model(s).\n",
      "\n",
      "VERDICT: nuisance fits look reasonable.\n"
     ]
    }
   ],
   "source": [
    "assessment = full.assess(\n",
    "    include_retargets=True,\n",
    "    arguments={\n",
    "        \"missingness\": {\n",
    "            \"gamma\": (0.0, 1.0, 2.0, 4.0, 8.0),\n",
    "            \"arm_gamma\": {0: 0.0, 1: -1.0},\n",
    "        },\n",
    "        \"tipping_gamma\": {\n",
    "            \"arm_gamma\": {0: 0.0, 1: -1.0},\n",
    "        },\n",
    "    },\n",
    ")\n",
    "ledger = assessment.to_frame()\n",
    "ran = ledger[~ledger[\"status\"].isin([\"unavailable\", \"not_applicable\"])]\n",
    "print(ran[[\"surface\", \"check\", \"status\"]].to_string(index=False))\n",
    "print(\"needs attention:\", tuple(item.name for item in assessment.attention))\n",
    "\n",
    "support = assessment.report(\"support\")\n",
    "nuisance = assessment.report(\"nuisance_models\")\n",
    "print()\n",
    "print(support.summary())\n",
    "print()\n",
    "print(nuisance.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assessment-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** Read the parts in order.\n",
    "\n",
    "| output part | what it shows on this draw |\n",
    "| --- | --- |\n",
    "| the status table | the score-equation check `passed`: targeting solved the estimated efficient score equation. A `completed` row ran, and is not a pass. The `deferred` row, `refute`, runs on request |\n",
    "| the first support tables | the fitted propensity ranges from 0.1807 to 0.8336, and no unit is truncated |\n",
    "| the `mechanism` table | the fitted response probability has a minimum of 0.0382. The product of the arm and response probabilities, $P(A = a, \\Delta = 1 \\mid W)$, has a minimum of 0.0081 |\n",
    "| the nuisance report | out-of-fold measures: an outcome `r2` of 0.7421, and calibration slopes of 0.9861 for g and 0.9972 for $\\pi$, with no finding |\n",
    "\n",
    "Positivity here is a property of the product $g \\cdot \\pi$, not of either factor. A patient with a\n",
    "middle chance of the offer and a low chance of response still has a small product. The clever\n",
    "covariate divides by that product, and its largest absolute value here is 81.76. The table minimum\n",
    "of 0.0081 also counts each patient's other arm, so it can lie below 1/81.76.\n",
    "\n",
    "The nuisance measures are out of fold, so they describe prediction on rows each model did not see.\n",
    "They do not show that a model is correctly specified. The support report describes the fitted g\n",
    "and $\\pi$. It cannot show that they are the true mechanisms. These outputs are implementation\n",
    "checks. They are not evidence for the identification assumptions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sensitivity-heading",
   "metadata": {},
   "source": [
    "## Step 9: sensitivity, what the data cannot check\n",
    "\n",
    "The data cannot test missingness at random. This step asks instead how far one stated departure\n",
    "must go to change the conclusion. The missingness curve is a pattern-mixture analysis. Within each\n",
    "arm and covariate profile, it shifts the mean of the unobserved scores away from the respondents'\n",
    "mean.\n",
    "[Missingness tilt and tipping gamma](../technical-reference/validation-methods.md#missingness-tilt-and-tipping-gamma)\n",
    "defines both operations.\n",
    "\n",
    "| element | meaning |\n",
    "| --- | --- |\n",
    "| `gamma` | the shift added to the logit of the fitted respondents' mean. The declared `q_bounds` fix the scale, so one `gamma` has the same meaning in every sample |\n",
    "| `arm_gamma={0: 0.0, 1: -1.0}` | the usual-support arm stays at MAR. A positive `gamma` lowers the unobserved navigation-arm mean. This is the worry that patients whom navigation failed do not answer |\n",
    "| `gamma` away from 0 | each row reuses the targeted outcome model of `gamma=0` without a new targeting step, so it is a plug-in |\n",
    "| tipping gamma | the `gamma` nearest 0 at which the estimate reaches 0. With `use_ci=True`, the `gamma` at which an interval limit reaches 0 |\n",
    "| `ci_lower`, `ci_upper` | they reuse the standard error of the fit, so they ignore uncertainty about `gamma` |\n",
    "\n",
    "The tilt mixes the two means with the fitted response probability. The estimate therefore moves by\n",
    "the mean, over all patients, of $1 - \\hat\\pi(1, W)$ times the shift of the unobserved navigation-arm\n",
    "mean. The code divides each move by the mean of $1 - \\hat\\pi(1, W)$. The result is a mean in score\n",
    "units, weighted by the chance of non-response under navigation. A `gamma` of 30 puts every\n",
    "unobserved navigation-arm mean at nearly 0, the floor of the scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "sensitivity",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-16T01:17:19.193423Z",
     "iopub.status.busy": "2026-10-16T01:17:19.193172Z",
     "iopub.status.idle": "2026-10-16T01:17:19.354606Z",
     "shell.execute_reply": "2026-10-16T01:17:19.353998Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " gamma estimand    psi  std_err  ci_lower  ci_upper  is_mar  gamma[0]  gamma[1]\n",
      "   0.0      ate 0.1478   0.0052    0.1376    0.1581    True       0.0      -0.0\n",
      "   1.0      ate 0.1057   0.0052    0.0955    0.1159   False       0.0      -1.0\n",
      "   2.0      ate 0.0635   0.0052    0.0533    0.0737   False       0.0      -2.0\n",
      "   4.0      ate 0.0174   0.0052    0.0072    0.0277   False       0.0      -4.0\n",
      "   8.0      ate 0.0037   0.0052   -0.0065    0.0139   False       0.0      -8.0\n",
      "\n",
      "tipping gamma, the estimate reaches 0: None\n",
      "tipping gamma, the interval reaches 0: 4.921\n",
      "mean non-response probability under navigation:  0.200\n",
      "MAR mean of the unobserved navigation-arm scores: 0.727\n",
      "mean shift at the interval tipping gamma:        0.688\n",
      "estimate when those scores sit at the floor:      0.0025\n"
     ]
    }
   ],
   "source": [
    "from cleverly.sensitivity import missingness_tilt\n",
    "\n",
    "missingness_curve = assessment.report(\"missingness\")\n",
    "tipping_gamma = assessment.report(\"tipping_gamma\")\n",
    "ci_tipping_gamma = full.sensitivity.tipping_gamma(arm_gamma={0: 0.0, 1: -1.0}, use_ci=True)\n",
    "print(missingness_curve.round(4).to_string(index=False))\n",
    "print()\n",
    "print(\"tipping gamma, the estimate reaches 0:\", tipping_gamma)\n",
    "print(f\"tipping gamma, the interval reaches 0: {ci_tipping_gamma:.3f}\")\n",
    "\n",
    "# The fitted response probability under navigation, clipped as the fit clips it.\n",
    "response_under_navigation = full.nuisance.bounded_missingness(0.01)[:, full.nuisance.arms.index(1)]\n",
    "nonresponse = float(np.mean(1.0 - response_under_navigation))\n",
    "# Gamma = 30 moves every unobserved navigation-arm mean to nearly 0, the floor of the scale.\n",
    "at = missingness_tilt(\n",
    "    full, [ci_tipping_gamma, 30.0], estimands=[\"ate\"], arm_gamma={0: 0.0, 1: -1.0}\n",
    ")\n",
    "ci_move, floor_move = full[\"ate\"].psi - at[\"psi\"].to_numpy()\n",
    "ci_mean_shift = ci_move / nonresponse\n",
    "mar_mean = floor_move / nonresponse\n",
    "print(f\"mean non-response probability under navigation:  {nonresponse:.3f}\")\n",
    "print(f\"MAR mean of the unobserved navigation-arm scores: {mar_mean:.3f}\")\n",
    "print(f\"mean shift at the interval tipping gamma:        {ci_mean_shift:.3f}\")\n",
    "print(f\"estimate when those scores sit at the floor:      {at['psi'].iloc[1]:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sensitivity-reading",
   "metadata": {},
   "source": [
    "**What this output tells you.** The `gamma` = 0 row reproduces the Step 6 estimate, 0.1478. The\n",
    "estimate falls as `gamma` rises, to 0.0174 at `gamma` = 4 and 0.0037 at `gamma` = 8. It does not\n",
    "reach 0 inside the search range, so the point tipping gamma is `None`. The interval reaches 0 at a\n",
    "`gamma` of 4.921.\n",
    "\n",
    "| quantity | value |\n",
    "| --- | --- |\n",
    "| the MAR mean of the unobserved navigation-arm scores, weighted by non-response | 0.727 |\n",
    "| the mean shift at which the interval reaches 0 | 0.688 |\n",
    "| the estimate when every unobserved navigation-arm score sits at the floor of 0 | 0.0025 |\n",
    "\n",
    "The score is bounded, so this departure has a worst case. On this draw, even if every\n",
    "navigation-arm patient who did not answer scored 0, the estimate would stay just above 0. The interval reaches 0 only when those\n",
    "patients score, on average, 0.688 below their MAR prediction of 0.727. Each value comes from the\n",
    "fitted outcome model, so it carries that model's error.\n",
    "\n",
    "This tilt keeps usual support at MAR. A departure that also raised the unobserved usual-support\n",
    "scores would reach 0 sooner. The curve does not show why patients did not answer. The analyst must\n",
    "decide whether a departure of that size is plausible for this survey.\n",
    "\n",
    "A review probe repeated this fit on 500 draws. The interval reached 0 on 485 draws. Between the\n",
    "5th and 95th percentiles, its tipping gamma ran from 3.17 to 5.48. The estimate itself reached 0\n",
    "inside the search range on 358 draws. So the margin of 0.0025 at the floor is a property of this\n",
    "draw. On this law, the worst case lies near 0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "method-evidence",
   "metadata": {},
   "source": [
    "## Method evidence\n",
    "\n",
    "The fits on this page use one draw of a synthetic law. The registered study in the table\n",
    "measures the method over repeated draws. The study page gives its law, its nuisance models, and\n",
    "the result of each cell.\n",
    "\n",
    "| fit on this page | registered study | law of the study |\n",
    "| --- | --- | --- |\n",
    "| Step 6, CV-TMLE with missing outcomes | [Stacked arm-indexed missing-outcome CV-TMLE](../technical-reference/method-evidence/stacked-arm-indexed-missing-outcome-cvtmle.md) | another law |\n",
    "\n",
    "No registered study fits a misspecified learner on continuous covariates, as Step 6 does.\n",
    "\n",
    "A review probe is a script under `reviews/notebook-review/probes/`. The step readings quote its\n",
    "results. A review probe is not a registered study.\n",
    "\n",
    "The [technical entry](../technical-reference/cv-tmle.md#validation-issues-special-to-this-method) lists the evidence and the limits of the method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interpret",
   "metadata": {},
   "source": [
    "## What the result means\n",
    "\n",
    "| question | answer for the Step 6 fit |\n",
    "| --- | --- |\n",
    "| what the estimate means | if every eligible patient got the offer, the mean transition score would be 0.1478 higher on the 0 to 1 scale than under usual support for all |\n",
    "| population and intervention | every eligible patient of the protocol, respondent or not. The offer at the discharge-home order, against usual support |\n",
    "| uncertainty | the 95% Wald interval (0.1376, 0.1581), from the estimated influence curve of the cross-fitted fit |\n",
    "| evidence | the registered study of [Method evidence](#method-evidence), and the review probes of Steps 6, 7, and 9. The Step 8 outputs are implementation checks |\n",
    "| unverified assumptions | missingness at random, no unmeasured confounding, consistency, and no interference |\n",
    "| not supported | an effect among respondents, a claim that MAR holds, an estimate for a program with deaths before day 30, and a general claim about the estimator from this draw |\n",
    "\n",
    "The strongest assumption stays untestable. Non-response that the experience itself causes breaks\n",
    "MAR. An example is a patient who ignores the survey because navigation failed. Step 9 shows how\n",
    "large that departure must be: the interval reaches 0 only when such patients score, on average,\n",
    "0.688 below comparable respondents. Report the estimate as conditional on an argument about the\n",
    "survey process.\n",
    "\n",
    "**The response model reads only baseline variables.** The fit cannot condition response on a\n",
    "variable recorded after the offer, such as contact attempts, or on survival. Each needs a\n",
    "sequential regression over that variable, which this point-treatment fit does not compute."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "where-next",
   "metadata": {},
   "source": [
    "## Where to go next\n",
    "\n",
    "When the outcome is an event over time, censoring takes the place of non-response.\n",
    "[Time-to-event outcomes](longitudinal-survival.ipynb) asks a readmission question with death as a\n",
    "competing event and disenrollment as censoring.\n",
    "\n",
    "The [examples index](index.md#the-program) lists every tutorial in the program."
   ]
  }
 ],
 "metadata": {
  "cleverly_execution": {
   "command": [
    "python",
    "scripts/execute_notebook.py",
    "docs/examples/survey-nonresponse.ipynb"
   ],
   "gated": {
    "code_source_sha256": "7d9bc2097b5812fbe42380c34e54301eadeaa209d2e3c96e71299ba8c04517c2",
    "execution_payload_sha256": "1c9cd871fb4806a04220ec47facc794a2ad029f7d95efcad6e445a17160eb438"
   },
   "recorded": {
    "cleverly_commit": "4cea0d1ad32abd00580cd9f3a27d84a8c294db5f",
    "cleverly_version": "0.1.3",
    "cleverly_worktree_clean": false,
    "dependency_lock_sha256": "2e4f7ad3264ff9b35cf7588e5c93a0e20fc5c0852092ac1da365491543836682",
    "generator_files": [
     "scripts/execute_notebook.py",
     "tests/notebooks.py"
    ],
    "generator_sha256": "f247771ab390b23e7e092d7eb35794e241c2ac84fc0526695df1cee1fc457a83",
    "package_source_sha256": "5fce085883c189351f5a0d142b216264208c189d01a46bb372a5e8fc68eed390"
   },
   "schema_version": 5
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}