{ "cells": [ { "cell_type": "markdown", "id": "title", "metadata": {}, "source": [ "# Sequential offers: a weekly retail campaign\n", "\n", "A weekly offer changes who stays engaged, and engagement steers the next offer. This notebook\n", "estimates the value of offer policies with longitudinal TMLE, and it compares two policies on the\n", "joint influence curve. Each step shows its code, its output, and what the output tells you.\n", "[Longitudinal TMLE](../technical-reference/longitudinal-tmle.md) gives the sequential regression and\n", "the algorithm.\n", "\n", "## The situation\n", "\n", "A subscription retailer sends one offer to each customer in each of three weeks. The offer is no\n", "code, a 15% discount code, or a free-shipping code. Before each send, the CRM scores whether the\n", "customer is engaged that week. The CRM draws each offer at random, with logged probabilities that\n", "read that engagement. A customer who withdraws tracking consent is lost to follow-up.\n", "\n", "This page uses a synthetic law. [The examples index](index.md#the-retailer-program) states how it\n", "relates to the health program of the other tutorials.\n", "\n", "| element | on this page |\n", "| --- | --- |\n", "| data | a long panel of 20000 synthetic customers, one row for each customer and week |\n", "| time order | in weeks 1 to 3, the CRM scores engagement, sends the offer, and the customer keeps or withdraws tracking consent. In week 4, the customer orders or not |\n", "| treatment | the offer of each week: `none`, `discount`, or `shipping` |\n", "| outcome | a full-price order in week 4, an order placed without a code |\n", "| observed | the engagement scores of weeks 1 to 3, one offer path for each customer, and the order of each customer who kept tracking consent |\n", "| counterfactual | the week-4 order under an offer policy that differs from the offers the customer received, with tracking consent kept. The panel never shows it |\n", "\n", "### The causal graph\n", "\n", "The graph draws one week. The pattern of week $t$ repeats for $t = 1, 2, 3$.\n", "\n", "| node | meaning |\n", "| --- | --- |\n", "| $W$ | the tenure |\n", "| $L_t$ | the engagement score of week $t$, recorded before the offer. $L_4$ is the score of week 4 |\n", "| $A_t$ | the offer of week $t$ |\n", "| $C_t$ | the tracking consent after the send of week $t$ |\n", "| $D$ | the count of discounts sent to engaged customers |\n", "| $Y$ | the week-4 order |\n", "| $U$ | an unrecorded cause |\n", "\n", "```text\n", " W (tenure) --> L_1, L_2, L_3, L_4, C_1, C_2, C_3, and Y (no arrow W --> A_t)\n", "\n", " One week t (t = 1, 2, 3):\n", "\n", " C_t <-- L_t --> A_t --> L_{t+1} --> ... --> L_4 --> Y\n", " | | ^\n", " | v |\n", " +-----> D --------------------------------------+\n", " also: L_t --> L_{t+1}\n", "\n", " U (unrecorded cause) - - -> C_t and Y (assumed absent)\n", "```\n", "\n", "| arrow or absent arrow | what it asserts | role | where the law sets it |\n", "| --- | --- | --- | --- |\n", "| `W` to `L_1`, `L_2`, `L_3`, `L_4`, `C_t`, and `Y` | tenure shifts engagement, consent, and the order | baseline confounder | `e1_tenure`, `next_tenure`, `consent_tenure`, `y_tenure` |\n", "| `L_t` to `A_t` | the logged probabilities read that week's engagement | confounder of the offer | the logged probabilities |\n", "| `A_t` to `L_{t+1}` | a discount raises the next engagement, and free shipping raises it less | the treatment changes a later confounder | `next_discount`, `next_shipping` |\n", "| `L_t` to `L_{t+1}` | engagement persists | a path from a confounder to `Y` | `next_engaged` |\n", "| `L_t` to `C_t` | engaged customers keep tracking consent more often | censoring with a recorded cause | `consent_engaged` |\n", "| `L_t` and `A_t` to `D`, and `D` to `Y` | a discount sent to an engaged customer lowers the order | a direct offer path. Engagement modifies the discount effect | `y_fatigue` |\n", "| `L_4` to `Y` | week-4 engagement raises the order | a mediator after the last send. The panel does not record it | `y_engaged` |\n", "| no unrecorded cause of `A_t` | the CRM randomizes | encoded, and it holds by design | the logged probabilities |\n", "| no `U` with arrows into `C_t` and `Y` | consent loss depends only on the recorded history | encoded, and open. This is scientific judgment, and Step 11 probes it | none, an assumption |\n", "| no `W` to `A_t` | the offers ignore tenure | a feature of this law. Identification does not need it | the logged probabilities |\n", "| no `A_t` to `C_t` | a send does not change that week's consent | a feature of this law. Identification does not need it | `consent_intercept`, `consent_engaged`, `consent_tenure` |\n", "\n", "The graph has no competing event. The subscription term is fixed at time zero and runs through\n", "week 4, so no account closes before the outcome week.\n", "\n", "**Reading the graph.** Engagement $L_2$ lies on the path $A_1 \\to L_2 \\to \\dots \\to Y$, and it also\n", "causes $A_2$. One outcome regression that adjusts for $L_2$ blocks part of the effect of $A_1$. A\n", "regression that omits $L_2$ leaves $A_2$ confounded. The sequential g-formula handles both.\n", "\n", "The arrow $L_t \\to C_t$ and the path from $L_t$ to $Y$ make the customers who keep tracking consent\n", "a selected group. The node $D$ makes the discount effect depend on engagement, so a rule can beat\n", "every static policy. The two encoded absences are the exchangeability assumptions of the next\n", "section.\n", "\n", "## What you will learn\n", "\n", "| after this notebook you can | the step that shows it |\n", "| --- | --- |\n", "| state a policy estimand when engagement feeds back | the next section |\n", "| read from the graph why one regression fails and the sequential g-formula is needed | the graph and Step 3 |\n", "| say what declared offer probabilities buy in LTMLE | Steps 6 and 7 |\n", "| compare two policies on the joint influence curve, and state what the evidence does not establish | Steps 8 to 11 and the final section |" ] }, { "cell_type": "markdown", "id": "plan", "metadata": {}, "source": [ "## The question, the estimand, and the method\n", "\n", "The retention lead asks one question. Which offer policy gives the largest share of full-price\n", "orders in week 4, and which policy beats which? A policy is a plan of three offers, not the effect\n", "of one send.\n", "\n", "| part of the question | on this page |\n", "| --- | --- |\n", "| target population | active subscribers whose subscription term is fixed at the first weekly send and runs through week 4, as the protocol of Step 4 states |\n", "| intervention | policy $d$, one offer for each of weeks 1 to 3. Three policies are static: `none`, `discount`, and `shipping` every week. Two are rules that read the week's engagement: win-back and win-back then ship |\n", "| counterfactual outcome | $Y^{d,\\bar c=1}$, the week-4 full-price order if the customer had followed $d$ and kept tracking consent |\n", "| estimand | $\\psi_d = E[Y^{d,\\bar c=1}]$, the week-4 full-price share under $d$ |\n", "| contrasts | $\\psi_d - \\psi_{\\mathrm{none}}$ for the four other policies, and the difference of two such contrasts |\n", "\n", "A one-week offer effect does not fix the later offers, so it defines no policy. An \"ever discounted\"\n", "comparison has the same flaw. The discount effect also depends on engagement, so a rule that reads\n", "engagement can beat every static policy.\n", "\n", "| assumption | what it means here | checkable from data, or scientific judgment |\n", "| --- | --- | --- |\n", "| offer exchangeability | each offer is independent of the potential outcomes, given tenure, the engagement of that week and of earlier weeks, and the earlier offers | by design. The CRM randomizes with logged probabilities, and the data hold them |\n", "| consent exchangeability | consent loss depends only on the recorded history | scientific judgment. Step 11 probes it with a benchmark |\n", "| positivity | each history that a policy can reach has a chance of the policy's offer and of staying consenting | offers: by design, because the smallest logged probability is 0.25. Consent: empirical, read in Step 9 |\n", "| consistency | an offer is the declared code template, and a code expires at the end of its week | scientific judgment |\n", "| no interference | one customer's offers do not change another customer's order | scientific judgment. One account belongs to one customer |\n", "\n", "Each method feature answers one feature of the problem.\n", "\n", "| feature of the problem | feature of the method |\n", "| --- | --- |\n", "| an earlier offer changes engagement, and engagement confounds the next offer | sequential regression, which evaluates the g-formula with one regression for each week |\n", "| the CRM logged the offer probabilities | `treatment_probabilities=` declares them. The fit runs no offer model, and targeting uses the exact probabilities |\n", "| engagement drives consent loss | a consent model that reads the recorded history |\n", "| the nuisance models are main-terms parametric models | no cross-fitting. These models have few parameters, and the registered known-mechanism cells run in sample |\n", "| the question compares policies | the joint influence curve gives the interval of a difference of two contrasts |\n", "\n", "Three estimators could answer the question. The table states what each one reads and when it is\n", "consistent. [Known node\n", "mechanisms](../technical-reference/longitudinal-tmle.md#known-node-mechanisms) and [sequential\n", "g-computation and IPW](../technical-reference/sequential-baselines.md) give the conditions.\n", "\n", "| estimator | what it reads | consistent when |\n", "| --- | --- | --- |\n", "| g-computation | the outcome regressions | every outcome regression is correct |\n", "| IPW | the declared offer probabilities and the consent model | the consent model is correct |\n", "| LTMLE | all of them | every outcome regression is correct, or the consent model is correct, because the offer probabilities are exact. Both hold up to the truncation that Step 9 reads |\n", "\n", "The table defines the terms this page uses most.\n", "\n", "| term | plain meaning |\n", "| --- | --- |\n", "| node | one decision time. Here, the weekly send in weeks 1, 2, and 3 |\n", "| history | every column recorded before the send at a node |\n", "| policy | a rule for each node. An entry is one offer for everyone, or a rule that reads the history. The code calls a policy a plan |\n", "| win-back | the policy that sends the discount to a customer who is not engaged, and no offer otherwise |\n", "| declared offer mechanism | the logged probability of each offer, read from columns. The fit runs no offer model |\n", "| [nuisance](../technical-reference/point-treatment-tmle.md) | a model the estimate needs but the question does not ask about. Here, the outcome, pseudo-outcome, and consent models |\n", "| [targeting](../user-guide/methods-learners.md#targeting-and-bounds) | a small update to each node's regression, weighted by one over the cumulative probability of following the policy and staying consenting, that removes first-order bias |\n", "| [influence curve](../technical-reference/inference.md) | how much each customer moves the estimate. Its variance gives the standard error |\n", "| sequential positivity | at every node, each history that the policy can reach has some chance of the policy's offer and of staying consenting |" ] }, { "cell_type": "markdown", "id": "setup-heading", "metadata": {}, "source": [ "## Step 1: set up\n", "\n", "The setup imports the libraries and prints the installed `cleverly` version. Every fit below passes\n", "its learners, its cross-fitting setting, and its random seed explicitly, so a rerun reproduces the\n", "stored outputs." ] }, { "cell_type": "code", "execution_count": 1, "id": "setup", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:17.426791Z", "iopub.status.busy": "2026-10-16T19:55:17.426435Z", "iopub.status.idle": "2026-10-16T19:55:20.660381Z", "shell.execute_reply": "2026-10-16T19:55:20.658378Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cleverly 0.1.4\n" ] } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn.dummy import DummyClassifier, DummyRegressor\n", "from sklearn.linear_model import LinearRegression, LogisticRegression\n", "\n", "import cleverly\n", "\n", "pd.set_option(\"display.width\", 130)\n", "pd.set_option(\"display.max_columns\", 20)\n", "print(\"cleverly\", cleverly.__version__)" ] }, { "cell_type": "markdown", "id": "setup-reading", "metadata": {}, "source": [ "**What this output tells you.** The stored outputs in this notebook came from the version named\n", "above. A different version can print different numbers." ] }, { "cell_type": "markdown", "id": "data-heading", "metadata": {}, "source": [ "## Step 2: the data\n", "\n", "The generator `make_sequential_offers` returns a long panel with one row per customer and week. A\n", "customer who withdraws consent has no later rows. The code prints the shape and the column names. It\n", "counts the customers who withdrew consent, and it prints the true values of the synthetic law.\n", "\n", "The seed follows a fixed rule. The page uses 1000, the first seed of the range 1000 to 1099 that the\n", "review probe of Step 6 sweeps. The pilot of Step 10 uses that seed plus 100000. The rule was\n", "set before the page ran with this seed. No draw was chosen for its results, so an interval on this\n", "page can miss its true value." ] }, { "cell_type": "code", "execution_count": 2, "id": "data", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:20.664102Z", "iopub.status.busy": "2026-10-16T19:55:20.663497Z", "iopub.status.idle": "2026-10-16T19:55:21.530012Z", "shell.execute_reply": "2026-10-16T19:55:21.528213Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rows and columns: (57154, 10)\n", "customers: 20000\n", "['customer', 'week', 'tenure', 'engaged', 'offer', 'consent', 'p_none', 'p_discount', 'p_shipping', 'full_price']\n", " customer week tenure engaged offer consent p_none p_discount p_shipping full_price\n", "0 0 1 -0.321 1 none 1 0.4 0.25 0.35 1.0\n", "1 0 2 -0.321 0 none 1 0.3 0.45 0.25 1.0\n", "2 0 3 -0.321 0 shipping 1 0.3 0.45 0.25 1.0\n", "3 1 1 -0.486 1 shipping 1 0.4 0.25 0.35 NaN\n", "4 1 2 -0.486 1 none 1 0.4 0.25 0.35 NaN\n", "5 1 3 -0.486 0 shipping 0 0.3 0.45 0.25 NaN\n", "\n", "customers who withdrew consent before week 4: 2686 of 20000\n", "known values of the synthetic law:\n", " ey_regimen[none] 0.5275\n", " ey_regimen[discount] 0.4596\n", " ey_regimen[shipping] 0.5865\n", " ey_regimen[win_back] 0.5997\n", " ey_regimen[win_back_ship] 0.6268\n", " ate_regimen[discount vs none] -0.0680\n", " ate_regimen[shipping vs none] 0.0590\n", " ate_regimen[win_back vs none] 0.0722\n", " ate_regimen[win_back_ship vs none] 0.0993\n" ] } ], "source": [ "from cleverly.datasets import make_sequential_offers\n", "\n", "panel, truth = make_sequential_offers(20_000, seed=1000)\n", "print(\"rows and columns:\", panel.shape)\n", "print(\"customers:\", panel[\"customer\"].nunique())\n", "print(list(panel.columns))\n", "print(panel.head(6).round(3))\n", "print()\n", "left = (panel.groupby(\"customer\")[\"consent\"].min() == 0).sum()\n", "print(f\"customers who withdrew consent before week 4: {left} of {panel['customer'].nunique()}\")\n", "print(\"known values of the synthetic law:\")\n", "for name, value in truth.items():\n", " print(f\" {name:36s} {value:.4f}\")" ] }, { "cell_type": "markdown", "id": "data-reading", "metadata": {}, "source": [ "**What this output tells you.** The panel has 57154 rows and 10 columns for 20000 customers. Each\n", "customer has up to three rows.\n", "\n", "| column | role |\n", "| --- | --- |\n", "| `customer`, `week` | the unit and the decision time |\n", "| `tenure` | a baseline covariate, a standard-normal draw |\n", "| `engaged` | the engagement score of that week, recorded before the offer |\n", "| `offer` | the offer sent that week: `none`, `discount`, or `shipping` |\n", "| `consent` | 1 if the customer still allows tracking after that send, and 0 if the customer withdrew |\n", "| `p_none`, `p_discount`, `p_shipping` | the logged probabilities of the three offers, for that customer and week |\n", "| `full_price` | the week-4 outcome. It repeats on each row, and it is missing for a customer who withdrew |\n", "\n", "The CRM chose the probabilities, so they are known. Engaged customers have 0.40 for no offer and\n", "0.25 for a discount. Customers who are not engaged have 0.30 and 0.45.\n", "\n", "A total of 2686 customers withdrew consent before week 4, and they have no outcome. The estimator\n", "expects that shape. A complete-case frame would discard those customers before any model sees them.\n", "\n", "The true values come from `offers_truth()`, which enumerates the law and does not simulate it. The\n", "true full-price rate is 0.5275 with no offers, 0.4596 with a discount every week, and 0.5865 with\n", "free shipping every week. Win-back gives 0.5997, and win-back then ship gives 0.6268.\n", "\n", "A discount every week ranks last, and win-back then ship ranks first. The true contrast of win-back\n", "against no offers is 0.0722. A real program has no `truth`. Every comparison against it below is a\n", "teaching device." ] }, { "cell_type": "markdown", "id": "association-heading", "metadata": {}, "source": [ "## Step 3: association first\n", "\n", "A confounder is a variable that changes both the next offer and the outcome. Engagement in weeks 2\n", "and 3 is one, and the earlier offer also changes it. The code reads three arrows of the graph in the\n", "logs. It then compares the customers who happened to follow each policy in all three weeks with the\n", "customers who followed no offers. The comparison keeps only the customers who still consented after\n", "week 3. This crude contrast adjusts for no covariate, and it is a teaching device." ] }, { "cell_type": "code", "execution_count": 3, "id": "association", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:21.535495Z", "iopub.status.busy": "2026-10-16T19:55:21.535120Z", "iopub.status.idle": "2026-10-16T19:55:21.786977Z", "shell.execute_reply": "2026-10-16T19:55:21.784853Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "arrow A_1 -> L_2, share engaged in week 2, by the week-1 offer:\n", "offer\n", "discount 0.886\n", "none 0.551\n", "shipping 0.661\n", "\n", "arrow L_1 -> A_1, share of each week-1 offer, by week-1 engagement:\n", "offer discount none shipping\n", "engaged \n", "0 0.440 0.308 0.251\n", "1 0.246 0.399 0.355\n", "\n", "arrow L_1 -> C_1, share still consenting after the week-1 send, by week-1 engagement:\n", "engaged\n", "0 0.929\n", "1 0.969\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "customers still consenting after week 3: 17314 of 20000\n", "share engaged in week 1 among these customers: 0.562\n", " followers share engaged_week1 crude_vs_none true_vs_none\n", "plan \n", "none 805 0.555 0.646 0.000 0.000\n", "discount 436 0.445 0.404 -0.110 -0.068\n", "shipping 550 0.609 0.671 0.054 0.059\n", "win_back 1247 0.604 0.524 0.049 0.072\n", "win_back_ship 956 0.606 0.513 0.050 0.099\n" ] } ], "source": [ "from cleverly.datasets import OFFER_PLANS\n", "\n", "week_one = panel[panel[\"week\"] == 1]\n", "week_two = panel[panel[\"week\"] == 2][[\"customer\", \"engaged\"]]\n", "moved = week_one.merge(week_two, on=\"customer\", suffixes=(\"\", \"_week2\"))\n", "print(\"arrow A_1 -> L_2, share engaged in week 2, by the week-1 offer:\")\n", "print(moved.groupby(\"offer\")[\"engaged_week2\"].mean().round(3).to_string())\n", "print()\n", "print(\"arrow L_1 -> A_1, share of each week-1 offer, by week-1 engagement:\")\n", "print(pd.crosstab(week_one[\"engaged\"], week_one[\"offer\"], normalize=\"index\").round(3).to_string())\n", "print()\n", "print(\"arrow L_1 -> C_1, share still consenting after the week-1 send, by week-1 engagement:\")\n", "print(week_one.groupby(\"engaged\")[\"consent\"].mean().round(3).to_string())\n", "print()\n", "\n", "wide = panel.pivot(index=\"customer\", columns=\"week\")\n", "kept = wide[wide[(\"consent\", 3)] == 1]\n", "first_engaged = kept[(\"engaged\", 1)].astype(int).to_numpy()\n", "rows = []\n", "for name, rule in OFFER_PLANS.items():\n", " followed = np.ones(len(kept), dtype=bool)\n", " for week in (1, 2, 3):\n", " engaged = kept[(\"engaged\", week)].astype(int).to_numpy()\n", " followed &= kept[(\"offer\", week)].to_numpy() == np.asarray(rule(engaged), dtype=object)\n", " rows.append(\n", " {\n", " \"plan\": name,\n", " \"followers\": int(followed.sum()),\n", " \"share\": kept[(\"full_price\", 1)][followed].mean(),\n", " \"engaged_week1\": first_engaged[followed].mean(),\n", " }\n", " )\n", "crude = pd.DataFrame(rows).set_index(\"plan\")\n", "crude[\"crude_vs_none\"] = crude[\"share\"] - crude.loc[\"none\", \"share\"]\n", "crude[\"true_vs_none\"] = [truth.get(f\"ate_regimen[{name} vs none]\", 0.0) for name in crude.index]\n", "print(f\"customers still consenting after week 3: {len(kept)} of {len(wide)}\")\n", "print(f\"share engaged in week 1 among these customers: {first_engaged.mean():.3f}\")\n", "print(crude.round(3).to_string())" ] }, { "cell_type": "markdown", "id": "association-reading", "metadata": {}, "source": [ "**What this output tells you.** Each of the first three tables confirms one arrow of the graph.\n", "\n", "| arrow | what the table shows |\n", "| --- | --- |\n", "| `A_1` to `L_2` | after a discount, a share of 0.886 of the customers are engaged in week 2. After no offer the share is 0.551, and after free shipping it is 0.661 |\n", "| `L_1` to `A_1` | engaged customers receive no offer at a share of 0.399 and a discount at 0.246. Customers who are not engaged receive a discount at 0.440 |\n", "| `L_1` to `C_1` | the share still consenting is 0.969 for engaged customers and 0.929 for the others |\n", "\n", "Engagement is therefore a confounder of the later offers, and the earlier offer changes it. One\n", "outcome regression cannot condition on it without blocking part of the earlier effect. It cannot\n", "leave it out without confounding the later offer.\n", "\n", "The last table is the crude comparison. It keeps 17314 of 20000 customers, those who consented\n", "through week 3. The crude contrast of win-back against no offers is 0.049, and the true contrast is\n", "0.072. For a discount every week the crude contrast is -0.110 and the true contrast is -0.068.\n", "\n", "The crude contrast has a systematic bias for the rules. The CRM sends no offer to engaged customers\n", "more often than to others, 0.399 against 0.308 in week 1. So the customers who followed `none` are\n", "more often engaged in week 1, at 0.646 against 0.562 among all kept customers. Engaged customers\n", "order more. That lifts the reference group, and it lowers the crude contrast of each rule.\n", "\n", "Free shipping also favors engaged customers, at 0.355 against 0.251 in week 1. Its followers are\n", "engaged in week 1 at 0.671, against 0.646 for the followers of `none`. So the reference group is not\n", "more engaged than they are. On this draw, the crude contrast of 0.054 sits near the true 0.059.\n", "\n", "In the review probe, the crude win-back contrast fell below the truth on 96 of 100 draws, by 0.0314\n", "on average. The Monte Carlo standard error was 0.0019. The crude shipping contrast had a mean bias\n", "of -0.0061 (Monte Carlo standard error 0.0029). That bias is small, at about two standard errors,\n", "and the probe does not show that it is zero." ] }, { "cell_type": "markdown", "id": "protocol-heading", "metadata": {}, "source": [ "## Step 4: write the protocol\n", "\n", "A `StudyProtocol` records the scientific design before any model runs.\n", "[Point-treatment TMLE](point-treatment-tmle.ipynb#step-4-write-the-protocol) introduces its fields\n", "and its fingerprint. The helper `offers_protocol()` returns the record for this campaign. It names the five\n", "policies of the question, and it states how the design treats consent withdrawal and expired codes." ] }, { "cell_type": "code", "execution_count": 4, "id": "protocol", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:21.790639Z", "iopub.status.busy": "2026-10-16T19:55:21.790242Z", "iopub.status.idle": "2026-10-16T19:55:21.797497Z", "shell.execute_reply": "2026-10-16T19:55:21.795666Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "causal study protocol: schema 1; 91f189a988d70fec\n", "target population: Active subscribers at the start of the campaign window\n", "eligibility: ['Active subscription at the first weekly send', 'Marketing consent at the first weekly send', 'At least one prior order', 'One account per customer, so shared-household accounts are excluded', 'A subscription term that is fixed at the first weekly send and runs through week 4']\n", "time zero: The first weekly send, after the week-1 engagement score and before the first offer\n", "treatment strategies: ['Send no offer in weeks 1 to 3', 'Send the 15% discount code every week', 'Send the free-shipping code every week', 'Win-back: send the discount code when the customer is not engaged, otherwise send no offer', 'Win-back then ship: send the discount code when the customer is not engaged, otherwise send the free-shipping code']\n", "treatment versions: ['No code sent', 'The declared 15% code, one send per week, valid until the end of that week', 'The declared free-shipping code, one send per week, valid until the end of that week', 'The declared 15% code for customers who are not engaged at the send, no code otherwise', 'The declared 15% code for customers who are not engaged at the send, and the declared free-shipping code otherwise']\n", "outcome: A full-price order in week 4, an order placed without a code\n", "horizon: Week 4, the week after the third weekly send\n", "intercurrent-event handling: ['Tracking-consent withdrawal before week 4 censors the customer. The strategy is hypothetical: the order had tracking continued', 'Each code expires at the end of its week, so no code is redeemable in week 4', 'None: the term runs through week 4, so no account closes before the outcome week']\n", "interference unit: Individual customer\n", "assumption rationale: ['Randomization with logged probabilities gives sequential exchangeability for the offers, given tenure and the weekly engagement score', 'Tenure and weekly engagement are the recorded confounders, and they cover the measured causes of consent loss', 'Declared code templates, one send per week, support consistency', 'One account per customer and per-customer codes support no interference']\n" ] } ], "source": [ "from cleverly.datasets import offers_protocol\n", "\n", "protocol = offers_protocol()\n", "print(\"\\n\".join(protocol.summary_lines()))" ] }, { "cell_type": "markdown", "id": "protocol-reading", "metadata": {}, "source": [ "**What this output tells you.** The first line gives the schema version and the fingerprint\n", "`94024dfd011817af`. The other lines repeat each field. Read them as a checklist.\n", "\n", "| protocol field | the question it answers for this campaign |\n", "| --- | --- |\n", "| target population and eligibility | who the effect is about. Eligibility includes a subscription term that is fixed at the first weekly send and runs through week 4. Time zero fixes the population, so no later event defines it |\n", "| time zero | the first weekly send, after the week-1 engagement score and before the first offer |\n", "| treatment strategies and versions | the five policies, each with the declared code template. Each code is valid until the end of its week |\n", "| outcome and horizon | a full-price order in week 4 |\n", "| intercurrent-event handling | consent withdrawal censors the customer under a hypothetical strategy: the order had tracking continued. A code expires at the end of its week, so no code is redeemable in week 4 |\n", "| interference unit | the individual customer |\n", "| assumption rationale | randomization with logged probabilities gives sequential exchangeability for the offers, given tenure and the weekly engagement. Tenure and engagement are the recorded causes of consent loss |\n", "\n", "The eligibility entry `Marketing consent` and the `consent` column are two consents. Marketing\n", "consent lets the retailer send offers, and it is a condition of the population. The `consent` column\n", "records tracking consent. Its withdrawal censors the customer." ] }, { "cell_type": "markdown", "id": "identify-heading", "metadata": {}, "source": [ "## Step 5: design and identification\n", "\n", "`LongPanel` reads the long panel. It names the unit, the time, the treatment, and the baseline\n", "covariate. The argument `time_varying` lists the columns that change by week. The argument\n", "`censoring` names the consent column. The argument `treatment_probabilities` maps each offer level\n", "to its logged-probability column. That mapping declares the offer mechanism, so the fit runs no\n", "offer model.\n", "\n", "A plan is a tuple with one entry for each week, or a `DynamicRegimen`. The rule of a\n", "`DynamicRegimen` reads the history of that week. The declaration `rule_kind=\"known\"` says that the\n", "rule is a fixed function of each customer's own history, chosen before the fit. The `each_node`\n", "helper applies one function at every week. The typed `RegimeContrast` compares each plan with\n", "`none`." ] }, { "cell_type": "code", "execution_count": 5, "id": "identify", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:21.801206Z", "iopub.status.busy": "2026-10-16T19:55:21.800828Z", "iopub.status.idle": "2026-10-16T19:55:22.582673Z", "shell.execute_reply": "2026-10-16T19:55:22.580749Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "history_names(1): ('tenure', 'engaged@1')\n", "history_names(2): ('tenure', 'engaged@1', 'engaged@2')\n", "history_names(3): ('tenure', 'engaged@1', 'engaged@2', 'engaged@3')\n", "\n", "contrast of each regime against the reference\n", "identified by explicit-adjustment: sequential g-formula under the declared treatment regimen\n", "history at offer@1: ['tenure', 'engaged@1']\n", "history at offer@2: ['tenure', 'engaged@1', 'engaged@2', 'offer@1']\n", "history at offer@3: ['tenure', 'engaged@1', 'engaged@2', 'engaged@3', 'offer@1', 'offer@2']\n", "required nuisances: ['sequential_outcome_regressions', 'treatment_and_censoring_mechanisms']\n", "assumptions:\n", " - consistency: each observed history equals the potential history under its realized regimen\n", " - no interference: one unit's potential history does not depend on other units' regimens\n", " - sequential exchangeability given the recorded history at every node\n", " - sequential positivity for treatment and remaining under observation\n", "causal study protocol: schema 1; 91f189a988d70fec\n" ] } ], "source": [ "from cleverly import CausalStudy, LongPanel, RegimeContrast\n", "from cleverly.datasets import OFFER_WEEKS, win_back, win_back_then_ship\n", "from cleverly.longitudinal import DynamicRegimen, each_node\n", "\n", "LOGGED = {\"none\": \"p_none\", \"discount\": \"p_discount\", \"shipping\": \"p_shipping\"}\n", "\n", "\n", "def offers_design(*, time_varying=(\"engaged\",)):\n", " return LongPanel(\n", " unit=\"customer\",\n", " time=\"week\",\n", " treatment=\"offer\",\n", " baseline=[\"tenure\"],\n", " time_varying=list(time_varying),\n", " censoring=\"consent\",\n", " outcome=\"full_price\",\n", " treatment_probabilities=LOGGED,\n", " )\n", "\n", "\n", "def rule(label, offer_for):\n", " return DynamicRegimen(\n", " label,\n", " each_node(lambda history, week: offer_for(history[f\"engaged@{week}\"]), OFFER_WEEKS),\n", " rule_kind=\"known\",\n", " )\n", "\n", "\n", "study = CausalStudy(panel, design=offers_design(), protocol=protocol)\n", "for week in (1, 2, 3):\n", " print(f\"history_names({week}):\", study.data.history_names(week))\n", "print()\n", "plans = {\n", " \"none\": (\"none\",) * OFFER_WEEKS,\n", " \"discount\": (\"discount\",) * OFFER_WEEKS,\n", " \"shipping\": (\"shipping\",) * OFFER_WEEKS,\n", " \"win_back\": rule(\"win_back\", win_back),\n", " \"win_back_ship\": rule(\"win_back_ship\", win_back_then_ship),\n", "}\n", "effect = study.identify(RegimeContrast(plans, reference=\"none\"))\n", "print(effect.summary(protocol=\"fingerprint\"))" ] }, { "cell_type": "markdown", "id": "identify-reading", "metadata": {}, "source": [ "**What this output tells you.** The three `history_names` lines list the columns that the design\n", "puts in the history of each week. Week 1 holds `tenure` and `engaged@1`, the confounder of the first\n", "offer. Each later week adds that week's engagement. The earlier offers join the history through the\n", "policy, and the `history at` lines of the summary show them.\n", "\n", "The `history at offer@3` line lists `engaged@1`, `engaged@2`, `engaged@3`, `offer@1`, and `offer@2`,\n", "with `tenure`. No line lists the week-4 engagement, which no offer follows. The summary names the\n", "estimand and its identification, the sequential g-formula under the declared treatment regimen. The\n", "required nuisances are the sequential outcome regressions and the treatment and censoring\n", "mechanisms. The last line prints the fingerprint from Step 4.\n", "\n", "The four printed assumptions are those of the table in the plan. Offer exchangeability holds by\n", "design. The consent part of sequential exchangeability does not, and Step 11 probes it. Step 9 reads\n", "the empirical part of sequential positivity." ] }, { "cell_type": "markdown", "id": "ltmle-heading", "metadata": {}, "source": [ "## Step 6: estimate the contrasts with LTMLE\n", "\n", "The configuration is written out in full.\n", "\n", "| setting | value | what it does |\n", "| --- | --- | --- |\n", "| `outcome_learner` | logistic regression | fits the regression of the binary outcome at the last node |\n", "| `pseudo_learner` | linear regression | fits the intermediate regressions. Their targets are continuous, and the recursion clips their predictions to the unit interval |\n", "| `censoring_learner` | logistic regression | fits the probability of staying consenting at each node, given the history |\n", "| `treatment_learner` | none | the offer mechanism is declared with `treatment_probabilities=`, so the fit runs no offer model |\n", "| `CrossFitting(enabled=False)` | in-sample fit | the nuisance models are main-terms parametric models, and the registered known-mechanism cells run in sample |\n", "| `g_bounds` | the default `(0.01, 1)` | bounds each cumulative product of offer and consent probabilities from below, so no weight exceeds 100. Step 9 reads its effect |\n", "| `Runtime(random_state=42, n_jobs=1)` | fixed seed, one process | makes the fit reproducible |\n", "\n", "The code then compares each contrast with its true value. The helper `against_truth` returns the\n", "estimate, the standard error, the interval, and whether the interval contains the truth." ] }, { "cell_type": "code", "execution_count": 6, "id": "ltmle", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:22.594033Z", "iopub.status.busy": "2026-10-16T19:55:22.593720Z", "iopub.status.idle": "2026-10-16T19:55:24.153938Z", "shell.execute_reply": "2026-10-16T19:55:24.153229Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Longitudinal TMLE (3 time points, n = 20000)\n", "\n", "parameter estimate std. error 95% CI p-value support \n", "---------------------------------- -------- ---------- ------------------ ------- ------------\n", "ate_regimen[discount vs none] -0.0878 0.0300 [-0.1467, -0.0289] 0.0035 experimental\n", "ate_regimen[shipping vs none] 0.0569 0.0284 [0.0011, 0.1126] 0.0456 experimental\n", "ate_regimen[win_back vs none] 0.0754 0.0204 [0.0353, 0.1154] 0.0002 experimental\n", "ate_regimen[win_back_ship vs none] 0.0820 0.0237 [0.0355, 0.1285] 0.0005 experimental\n", "\n", "experimental: No registered study covers this configuration. Nearest row: G01 (canonical-ltmle), differing on treatment, qualifiers.\n", "\n", " time points: 3\n", " long panel: unit column 'customer', decision times 1, 2, 3 (column 'week')\n", " outcome family: binomial\n", " regimens: none=(none/none/none), discount=(discount/discount/discount), shipping=(shipping/shipping/shipping), win_back=(d:d@1/d:d@2/d:d@3), win_back_ship=(d:d@1/d:d@2/d:d@3)\n", " a 'd' is a rule d_t(H_t) read off [W, L_1, ..., L_t]; its followers are a covariate-dependent set, so the counts below describe this sample\n", " assigned arms, win_back: 6e5d6c23174182a2\n", " assigned arms, win_back_ship: 6ee945a698bd4b75\n", " reference: none\n", " cross-fitting: none -- nuisances fitted in sample\n", " g_bounds: fixed [0.01, 1] on each cumulative treatment-and-censoring probability (package default; R ltmle-compatible heuristic -- inspect truncation diagnostics)\n", " known factors (declared, no learner): offer@1, offer@2, offer@3; no truncation bound moved a declared value\n", " confidence level: 95%\n", " random_state: 42\n", " causal estimand: contrast of each regime against the reference\n", " identification: explicit-adjustment; sequential g-formula under the declared treatment regimen\n", " required nuisances: sequential_outcome_regressions, treatment_and_censoring_mechanisms\n", " identification assumptions: consistency: each observed history equals the potential history under its realized regimen; no interference: one unit's potential history does not depend on other units' regimens; sequential exchangeability given the recorded history at every node; sequential positivity for treatment and remaining under observation\n", " causal study protocol: schema 1; 91f189a988d70fec\n", " data 4ff95105cdca0278 | folds 738358eab9007cbb | cleverly 0.1.4 | 2026-10-16T19:55:23+00:00\n", " none: 805 of 20000 units followed it throughout; max weight 59.7, effective n 709; max truncated share 0.0% at t=1\n", " discount: 436 of 20000 units followed it throughout; max weight 80.9, effective n 379; max truncated share 0.0% at t=1\n", " shipping: 550 of 20000 units followed it throughout; max weight 100.0, effective n 475; max truncated share 0.2% at t=3\n", " win_back: 1247 of 20000 units followed it throughout; max weight 19.7, effective n 1242; max truncated share 0.0% at t=1\n", " win_back_ship: 956 of 20000 units followed it throughout; max weight 28.6, effective n 936; max truncated share 0.0% at t=1\n", "\n", " simultaneous 95% bands (multiplier bootstrap, critical value 2.411 vs 1.960 pointwise) [experimental]:\n", " ate_regimen[discount vs none] [-0.1602, -0.0154]\n", " ate_regimen[shipping vs none] [-0.0117, 0.1254]\n", " ate_regimen[win_back vs none] [0.0261, 0.1247]\n", " ate_regimen[win_back_ship vs none] [0.0248, 0.1392]\n", "\n", " estimate std_error ci_low ci_high truth covers\n", "contrast \n", "discount -0.088 0.030 -0.147 -0.029 -0.068 True\n", "shipping 0.057 0.028 0.001 0.113 0.059 True\n", "win_back 0.075 0.020 0.035 0.115 0.072 True\n", "win_back_ship 0.082 0.024 0.035 0.128 0.099 True\n" ] } ], "source": [ "from cleverly import CrossFitting, ModelSpec, Runtime, TMLEMethod\n", "\n", "logistic = LogisticRegression(max_iter=1000, random_state=42)\n", "ltmle = TMLEMethod(\n", " models=ModelSpec(\n", " outcome_learner=logistic,\n", " pseudo_learner=LinearRegression(),\n", " censoring_learner=logistic,\n", " ),\n", " cross_fitting=CrossFitting(enabled=False),\n", " runtime=Runtime(random_state=42, n_jobs=1),\n", ")\n", "result = effect.estimate(method=ltmle)\n", "print(result.summary(protocol=\"fingerprint\"))\n", "\n", "\n", "def against_truth(fit):\n", " rows = []\n", " for key, estimate in fit.estimates.items():\n", " low, high = estimate.ci\n", " rows.append(\n", " {\n", " \"contrast\": key.removeprefix(\"ate_regimen[\").removesuffix(\" vs none]\"),\n", " \"estimate\": estimate.psi,\n", " \"std_error\": estimate.std_error,\n", " \"ci_low\": low,\n", " \"ci_high\": high,\n", " \"truth\": truth[key],\n", " \"covers\": bool(low <= truth[key] <= high),\n", " }\n", " )\n", " return pd.DataFrame(rows).set_index(\"contrast\")\n", "\n", "\n", "print()\n", "print(against_truth(result).round(3).to_string())" ] }, { "cell_type": "markdown", "id": "ltmle-reading", "metadata": {}, "source": [ "**What this output tells you.** The table reports four contrasts against `none`. The win-back\n", "contrast is 0.0754, with a standard error of 0.0204 and a 95% interval of (0.0353, 0.1154). The\n", "interval contains the true contrast of 0.0722.\n", "\n", "The discount contrast is -0.0878 with an interval of (-0.1467, -0.0289), and it contains the true\n", "-0.0680. The shipping contrast is 0.0569 with an interval of (0.0011, 0.1126), and its p-value is\n", "0.0456. That interval excludes zero by a narrow margin. The win-back-then-ship contrast is 0.0820\n", "with an interval of (0.0355, 0.1285). All four intervals contain their true values on this draw.\n", "That is one draw, not a coverage result.\n", "\n", "The review probe\n", "`reviews/notebook-review/probes/sequential-offers/sweep.py` repeats the fits of this page on 100\n", "draws, seeds 1000 to 1099. The four intervals covered on 95, 95, 92, and 94 of 100 draws.\n", "\n", "The `support` column marks the four contrasts and the bands `experimental`, because no registered\n", "study covers this configuration. The [support label](../user-guide/results-assessment.md#read-the-support-label) makes no coverage claim.\n", "\n", "The simultaneous bands use a critical value of 2.411 in place of 1.960. They answer which policies\n", "differ from `none` for the four contrasts together. The bands of discount, win-back, and win-back\n", "then ship exclude zero. The band of shipping, (-0.0117, 0.1254), contains it." ] }, { "cell_type": "markdown", "id": "blind-outcome-heading", "metadata": {}, "source": [ "## Step 7: what the logged offer probabilities buy\n", "\n", "The offer probabilities are known, and the outcome regressions are not. The main-terms regressions\n", "of Step 6 are wrong. They omit the fatigue interaction of the offer and the engagement. The true\n", "regression also averages over the unrecorded week-4 engagement, which interacts with tenure. The\n", "Step 6 estimate therefore already rests on the consent model.\n", "\n", "This step asks what LTMLE gains when the outcome regressions are wrong. It compares g-computation\n", "and LTMLE under a deliberately wrong outcome model.\n", "\n", "| for the comparison | on this page |\n", "| --- | --- |\n", "| estimands | both estimators target the `RegimeContrast` of Step 5, each policy against `none`. The code prints whether the estimand keys match |\n", "| assumptions added | g-computation adds that every outcome regression is correct. LTMLE adds that the consent model is correct, and it uses the exact offer probabilities |\n", "| mechanism | the wrong outcome model returns the mean outcome of the customers who followed the policy and reads no history. The g-computation plug-in has no term that corrects it. The LTMLE targeting step regresses the residuals on a covariate built from the exact offer probabilities and the fitted consent model, and that update repairs the plug-in |\n", "| evidence | the registered cell `known_mechanism_accuracy/ate_regimen__ltmle_known_treatment__censoring_estimated` fits outcome regressions without `L1` and a fitted consent model. The review probe repeats this comparison on 100 draws, and the reading below gives its numbers. The test `tests/unit/test_known_node_mechanisms.py` (E10) shows that LTMLE with every factor declared returns the truth under a wrong outcome regression. [Known node mechanisms](../technical-reference/longitudinal-tmle.md#known-node-mechanisms) gives the theory |\n", "| reading | one draw shows no general property |\n", "\n", "The wrong outcome model is a deliberate mutation, and no analyst would choose it. A prior-only\n", "classifier fits the last node, and a mean regressor fits the earlier nodes. The consent model is the\n", "logistic regression of Step 6.\n", "\n", "G-computation reports a point estimate only with these learners, because its interval needs `GLM`\n", "outcome regressions. The code prints one row for each contrast. A row holds the truth, the Step 6\n", "estimate, and the crude contrast of Step 3. It also holds the wrong-model LTMLE estimate with its\n", "interval, and the wrong-model g-computation estimate." ] }, { "cell_type": "code", "execution_count": 7, "id": "blind-outcome", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:24.157641Z", "iopub.status.busy": "2026-10-16T19:55:24.157091Z", "iopub.status.idle": "2026-10-16T19:55:25.593299Z", "shell.execute_reply": "2026-10-16T19:55:25.588025Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "same estimands: True\n", " truth ltmle ltmle_se blind_ltmle blind_se ci_low ci_high covers blind_gcomp crude\n", "contrast \n", "discount -0.068 -0.088 0.030 -0.096 0.033 -0.160 -0.032 True -0.110 -0.110\n", "shipping 0.059 0.057 0.028 0.068 0.030 0.009 0.126 True 0.054 0.054\n", "win_back 0.072 0.075 0.020 0.080 0.022 0.038 0.123 True 0.049 0.049\n", "win_back_ship 0.099 0.082 0.024 0.088 0.025 0.038 0.138 True 0.050 0.050\n" ] } ], "source": [ "from cleverly import GComputationMethod\n", "\n", "blind = ModelSpec(\n", " outcome_learner=DummyClassifier(strategy=\"prior\"),\n", " pseudo_learner=DummyRegressor(),\n", " censoring_learner=logistic,\n", ")\n", "blind_ltmle = effect.estimate(\n", " method=TMLEMethod(models=blind, cross_fitting=ltmle.cross_fitting, runtime=ltmle.runtime)\n", ")\n", "blind_gcomp = effect.estimate(method=GComputationMethod(models=blind, runtime=ltmle.runtime))\n", "print(\n", " \"same estimands:\",\n", " blind_ltmle.estimates.keys() == blind_gcomp.estimates.keys() == result.estimates.keys(),\n", ")\n", "rows = []\n", "for key, estimate in blind_ltmle.estimates.items():\n", " low, high = estimate.ci\n", " name = key.removeprefix(\"ate_regimen[\").removesuffix(\" vs none]\")\n", " rows.append(\n", " {\n", " \"contrast\": name,\n", " \"truth\": truth[key],\n", " \"ltmle\": result[key].psi,\n", " \"ltmle_se\": result[key].std_error,\n", " \"blind_ltmle\": estimate.psi,\n", " \"blind_se\": estimate.std_error,\n", " \"ci_low\": low,\n", " \"ci_high\": high,\n", " \"covers\": bool(low <= truth[key] <= high),\n", " \"blind_gcomp\": blind_gcomp[key].psi,\n", " \"crude\": crude.loc[name, \"crude_vs_none\"],\n", " }\n", " )\n", "blind_table = pd.DataFrame(rows).set_index(\"contrast\")\n", "print(blind_table.round(3).to_string())" ] }, { "cell_type": "markdown", "id": "blind-outcome-reading", "metadata": {}, "source": [ "**What this output tells you.** The line `same estimands: True` confirms that the three fits report\n", "the same contrasts, so their numbers compare.\n", "\n", "On this draw the wrong-model g-computation estimate for win-back against `none` is 0.049, and the\n", "truth is 0.072. For win-back then ship it is 0.050 against 0.099. The `blind_gcomp` column equals\n", "the `crude` column on every row. The wrong model returns the mean outcome of the followers, which is\n", "the crude follower mean. So g-computation under this model reproduces the crude bias of Step 3.\n", "\n", "The wrong-model LTMLE estimate for win-back is 0.080, with an interval of (0.038, 0.123) that\n", "contains the truth. The Step 6 estimate is 0.075. All four wrong-model LTMLE intervals contain their\n", "true values on this draw. On this draw, the wrong-model LTMLE estimates sit nearer to the truth than\n", "the g-computation estimates for discount, win-back, and win-back then ship. The cost is a wider standard error, 0.022 against 0.020\n", "for win-back.\n", "\n", "One draw shows no general property. In the review probe, the wrong-model g-computation estimate for\n", "win-back fell below the truth by 0.0314 on average over 100 draws. The wrong-model LTMLE bias was\n", "-0.0060, and its interval covered the truth on 93 of 100 draws." ] }, { "cell_type": "markdown", "id": "failure-mode-heading", "metadata": {}, "source": [ "## Step 8: the failure mode, a comparison that ignores the shared sample\n", "\n", "The question asks which policy beats which. Both computations below target the same estimand, the\n", "difference of two policy contrasts, such as the win-back contrast minus the discount contrast. They\n", "differ only in the standard error.\n", "\n", "| computation | standard error of the difference |\n", "| --- | --- |\n", "| paired, `result.contrast(...)` | from the joint influence curve of the two contrasts, which includes their covariance |\n", "| unpaired | `sqrt(se_a^2 + se_b^2)`, which assumes that the two estimates are independent |\n", "\n", "The two contrasts share the `none` arm and every customer, so their estimates correlate positively.\n", "The unpaired formula adds an independence assumption that is false here. The code compares three\n", "pairs and prints both standard errors. The unpaired number is the failure mode." ] }, { "cell_type": "code", "execution_count": 8, "id": "failure-mode", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:25.598266Z", "iopub.status.busy": "2026-10-16T19:55:25.597946Z", "iopub.status.idle": "2026-10-16T19:55:25.614564Z", "shell.execute_reply": "2026-10-16T19:55:25.612327Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " estimate paired_se paired_low paired_high unpaired_se truth\n", "comparison \n", "win_back - discount 0.163 0.027 0.110 0.217 0.036 0.140\n", "win_back - shipping 0.018 0.026 -0.032 0.069 0.035 0.013\n", "win_back_ship - win_back 0.007 0.020 -0.033 0.046 0.031 0.027\n" ] } ], "source": [ "pairs = [\n", " (\"win_back\", \"discount\"),\n", " (\"win_back\", \"shipping\"),\n", " (\"win_back_ship\", \"win_back\"),\n", "]\n", "rows = []\n", "for better, other in pairs:\n", " keys = [f\"ate_regimen[{better} vs none]\", f\"ate_regimen[{other} vs none]\"]\n", " paired = result.contrast(\n", " lambda values: values[0] - values[1], keys, name=f\"{better} minus {other}\"\n", " )\n", " unpaired_se = float(np.hypot(result[keys[0]].std_error, result[keys[1]].std_error))\n", " low, high = paired.ci\n", " rows.append(\n", " {\n", " \"comparison\": f\"{better} - {other}\",\n", " \"estimate\": paired.psi,\n", " \"paired_se\": paired.std_error,\n", " \"paired_low\": low,\n", " \"paired_high\": high,\n", " \"unpaired_se\": unpaired_se,\n", " \"truth\": truth[keys[0]] - truth[keys[1]],\n", " }\n", " )\n", "comparison = pd.DataFrame(rows).set_index(\"comparison\")\n", "print(comparison.round(3).to_string())" ] }, { "cell_type": "markdown", "id": "failure-mode-reading", "metadata": {}, "source": [ "**What this output tells you.** Win-back against a discount every week has an estimate of 0.163. Its\n", "paired standard error is 0.027, and its 95% interval is (0.110, 0.217). The true difference is\n", "0.140. The unpaired standard error is 0.036. The paired interval excludes zero.\n", "\n", "Win-back against free shipping has an estimate of 0.018, a paired standard error of 0.026, and an\n", "interval of (-0.032, 0.069). Win-back then ship against win-back has an estimate of 0.007, a\n", "standard error of 0.020, and an interval of (-0.033, 0.046). The unpaired standard errors are 0.035\n", "and 0.031. Both paired intervals contain zero. The true differences are 0.013 and 0.027. This draw\n", "separates neither pair.\n", "\n", "The unpaired number is larger in every row. It treats two estimates that share every customer as\n", "independent, and it ignores their positive correlation. Report the paired interval." ] }, { "cell_type": "markdown", "id": "validation-heading", "metadata": {}, "source": [ "## Validation: what each check can and cannot show\n", "\n", "Steps 9 to 11 check the fit. None of them checks an identification assumption. Each reading names\n", "what its output does not show." ] }, { "cell_type": "markdown", "id": "assessment-heading", "metadata": {}, "source": [ "## Step 9: diagnostics, what the fit can check\n", "\n", "`result.assess()` runs the checks that apply to the fit. The code prints the operations that ran. It\n", "then prints the support report. The report shows how many customers followed each policy at each\n", "node, how heavy their weights are, and how many weights a bound replaced. A weight is one over the\n", "cumulative probability of following the policy and staying consenting. The code ends with the\n", "consent model of each node, measured on the rows it fit." ] }, { "cell_type": "code", "execution_count": 9, "id": "assessment", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:25.617402Z", "iopub.status.busy": "2026-10-16T19:55:25.617021Z", "iopub.status.idle": "2026-10-16T19:55:25.962172Z", "shell.execute_reply": "2026-10-16T19:55:25.959045Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " surface check status\n", "validation score_equations passed\n", "validation support completed\n", "validation nuisance_models completed\n", "\n", " regimen time n_followed max_weight effective_n share_truncated\n", " none 1 6814 4.007 6628.778 0.000\n", " none 2 2308 16.543 2140.520 0.000\n", " none 3 805 59.676 708.798 0.000\n", " discount 1 6333 4.316 5883.702 0.000\n", " discount 2 1630 18.759 1449.934 0.000\n", " discount 3 436 80.877 379.489 0.000\n", " shipping 1 5870 5.053 5663.046 0.000\n", " shipping 2 1766 26.343 1605.188 0.000\n", " shipping 3 550 100.000 475.304 0.002\n", " win_back 1 7953 2.728 7938.502 0.000\n", " win_back 2 3186 7.548 3177.376 0.000\n", " win_back 3 1247 19.651 1241.930 0.000\n", "win_back_ship 1 7499 3.108 7413.283 0.000\n", "win_back_ship 2 2665 9.424 2621.040 0.000\n", "win_back_ship 3 956 28.592 935.762 0.000\n", "minimum effective-sample-size ratio: 86.4%\n", "\n", " role time n evaluation auc calibration_slope\n", "censoring 1 20000 in_sample 0.647 1.003\n", "censoring 2 19017 in_sample 0.645 1.004\n", "censoring 3 18137 in_sample 0.650 1.000\n" ] } ], "source": [ "assessment = result.assess()\n", "ledger = assessment.to_frame()\n", "ran = ledger[~ledger[\"status\"].isin([\"unavailable\", \"not_applicable\", \"deferred\"])]\n", "print(ran[[\"surface\", \"check\", \"status\"]].to_string(index=False))\n", "support = assessment.report(\"support\").to_frame()\n", "columns = [\"regimen\", \"time\", \"n_followed\", \"max_weight\", \"effective_n\", \"share_truncated\"]\n", "print()\n", "print(support[columns].round(3).to_string(index=False))\n", "kish_ratio = (support[\"effective_n\"] / support[\"n_followed\"]).min()\n", "print(f\"minimum effective-sample-size ratio: {kish_ratio:.1%}\")\n", "print()\n", "nuisance = assessment.report(\"nuisance_models\").to_frame()\n", "consent = nuisance[nuisance[\"role\"] == \"censoring\"]\n", "print(\n", " consent[[\"role\", \"time\", \"n\", \"evaluation\", \"auc\", \"calibration_slope\"]]\n", " .round(3)\n", " .to_string(index=False)\n", ")" ] }, { "cell_type": "markdown", "id": "assessment-reading", "metadata": {}, "source": [ "**What this output tells you.** The first table lists the operations that ran. The score-equation\n", "check is `passed`. The support report and the nuisance report are `completed`. A completed result\n", "means the calculation ran. It is not a pass.\n", "\n", "The support table divides by the declared offer probabilities and the fitted consent model. It\n", "describes those models. It cannot see an unrecorded cause of consent loss.\n", "\n", "| policy | followers at week 3 | max weight | effective n | what it shows |\n", "| --- | --- | --- | --- | --- |\n", "| `none` | 805 | 59.676 | 708.798 | thin support |\n", "| `discount` | 436 | 80.877 | 379.489 | the thinnest support of the static policies |\n", "| `shipping` | 550 | 100.000 | 475.304 | the heaviest weight, at the cap |\n", "| `win_back` | 1247 | 19.651 | 1241.930 | the rule keeps the most customers |\n", "| `win_back_ship` | 956 | 28.592 | 935.762 | the rule keeps more customers than any static policy |\n", "\n", "A static policy sends the same offer to engaged and unengaged customers. The logged probability of\n", "that offer is lower in one of the two groups, so few customers follow the policy. Win-back sends the\n", "likeliest logged offer in each group, so it keeps the most customers.\n", "\n", "The minimum ratio of effective sample size to followers is 86.4%. One row has a truncated weight.\n", "The `shipping` policy at week 3 has a truncated share of 0.002, which is 1 of its 550 followers. The\n", "offer probabilities alone cannot reach the cap, because their product is at least 0.0156. The weight\n", "reached the cap of 100 through the consent probabilities, for a customer who has low tenure and was\n", "not engaged in any week.\n", "\n", "The consent models have an `auc` of 0.647, 0.645, and 0.650 at weeks 1, 2, and 3. They separate the\n", "customers poorly, because the chance of consent loss varies little with engagement and tenure. The\n", "calibration slopes are 1.003, 1.004, and 1.000. The models are fitted in sample, so a slope near 1\n", "does not show that a model is correct." ] }, { "cell_type": "markdown", "id": "deployment-heading", "metadata": {}, "source": [ "## Step 10: a synthetic deployment check\n", "\n", "A real campaign would run the win-back policy on a pilot cohort and compare its observed rate with\n", "the estimate. No applied data access exists for this tutorial, so the pilot is synthetic. The\n", "`policy=\"win_back\"` argument draws a fresh cohort of 10000 customers under that policy. In the\n", "cohort, every customer consents and stays tracked. This is a design condition of the pilot: a\n", "consented panel with no loss.\n", "\n", "The code fits the win-back level with `RegimeMean` and compares it with the pilot mean. The pilot is\n", "independent of the estimation sample, so the standard error of the difference is\n", "`sqrt(se_fit^2 + se_pilot^2)`." ] }, { "cell_type": "code", "execution_count": 10, "id": "deployment", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:25.966574Z", "iopub.status.busy": "2026-10-16T19:55:25.965847Z", "iopub.status.idle": "2026-10-16T19:55:27.474179Z", "shell.execute_reply": "2026-10-16T19:55:27.471379Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pilot customers: 10000; lost to consent: 0\n", "pilot share of full-price orders: 0.609 (standard error 0.0049)\n", "fitted win-back level: 0.596 (standard error 0.0133)\n", "difference, fit minus pilot: -0.0135 (standard error 0.0141)\n", "difference in standard errors: -0.95\n", "population win-back level: 0.600\n" ] } ], "source": [ "from cleverly import RegimeMean\n", "\n", "pilot, _ = make_sequential_offers(10_000, seed=101_000, policy=\"win_back\")\n", "pilot_outcome = pilot.loc[pilot[\"week\"] == 3, \"full_price\"]\n", "levels = study.identify(RegimeMean(plans)).estimate(method=ltmle)\n", "fitted = levels[\"ey_regimen[win_back]\"]\n", "pilot_mean = float(pilot_outcome.mean())\n", "pilot_se = float(pilot_outcome.std() / np.sqrt(len(pilot_outcome)))\n", "difference = fitted.psi - pilot_mean\n", "difference_se = float(np.hypot(fitted.std_error, pilot_se))\n", "print(\n", " f\"pilot customers: {len(pilot_outcome)}; lost to consent: {int((pilot['consent'] == 0).sum())}\"\n", ")\n", "print(f\"pilot share of full-price orders: {pilot_mean:.3f} (standard error {pilot_se:.4f})\")\n", "print(\n", " f\"fitted win-back level: {fitted.psi:.3f} (standard error {fitted.std_error:.4f})\"\n", ")\n", "print(f\"difference, fit minus pilot: {difference:+.4f} (standard error {difference_se:.4f})\")\n", "print(f\"difference in standard errors: {difference / difference_se:+.2f}\")\n", "print(f\"population win-back level: {truth['ey_regimen[win_back]']:.3f}\")" ] }, { "cell_type": "markdown", "id": "deployment-reading", "metadata": {}, "source": [ "**What this output tells you.** The pilot has 10000 customers and loses none to consent. Its\n", "full-price rate is 0.609, with a standard error of 0.0049. The fitted win-back level is 0.596, with\n", "a standard error of 0.0133. The difference is -0.0135 with a standard error of 0.0141, which is\n", "-0.95 standard errors. The population value is 0.600.\n", "\n", "The two numbers differ by less than 1.96 standard errors on this draw. On applied data, a failed\n", "exchangeability or consent assumption could move the fit away from a pilot. Here both draws come\n", "from one law in which the assumptions hold, so the check tests the estimator only. A small difference does not prove either assumption." ] }, { "cell_type": "markdown", "id": "sensitivity-heading", "metadata": {}, "source": [ "## Step 11: consent exchangeability, how much a recorded cause moves the estimate\n", "\n", "The offers hold exchangeability by design, because the CRM drew them at random with the logged\n", "probabilities. The assumption at risk is consent exchangeability. The fit assumes that consent loss\n", "depends only on the recorded history, which excludes the unrecorded cause $U$ of the graph.\n", "Engagement is a recorded cause of consent loss in this law, with `tenure`.\n", "\n", "The technical reference derives no omitted-variable bound for a longitudinal fit, so this step runs\n", "a benchmark. The code refits the two static contrasts without `engaged` in the history. The declared\n", "offer probabilities stay the same in both fits. The move then comes from the outcome and consent\n", "models. The win-back rules read `engaged`, so they cannot run without it. The covariate `tenure` is\n", "the only baseline covariate, so the code does not drop it." ] }, { "cell_type": "code", "execution_count": 11, "id": "sensitivity", "metadata": { "execution": { "iopub.execute_input": "2026-10-16T19:55:27.482130Z", "iopub.status.busy": "2026-10-16T19:55:27.481487Z", "iopub.status.idle": "2026-10-16T19:55:29.632885Z", "shell.execute_reply": "2026-10-16T19:55:29.631441Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " with_engagement without move move_in_se\n", "contrast \n", "discount -0.088 -0.095 -0.007 -0.239\n", "shipping 0.057 0.054 -0.003 -0.095\n" ] } ], "source": [ "static = {name: plans[name] for name in (\"none\", \"discount\", \"shipping\")}\n", "full_fit = study.identify(RegimeContrast(static, reference=\"none\")).estimate(method=ltmle)\n", "reduced_study = CausalStudy(panel, design=offers_design(time_varying=()), protocol=protocol)\n", "reduced_fit = reduced_study.identify(RegimeContrast(static, reference=\"none\")).estimate(\n", " method=ltmle\n", ")\n", "rows = []\n", "for key, estimate in full_fit.estimates.items():\n", " move = reduced_fit[key].psi - estimate.psi\n", " rows.append(\n", " {\n", " \"contrast\": key.removeprefix(\"ate_regimen[\").removesuffix(\" vs none]\"),\n", " \"with_engagement\": estimate.psi,\n", " \"without\": reduced_fit[key].psi,\n", " \"move\": move,\n", " \"move_in_se\": move / estimate.std_error,\n", " }\n", " )\n", "benchmark = pd.DataFrame(rows).set_index(\"contrast\")\n", "print(benchmark.round(3).to_string())" ] }, { "cell_type": "markdown", "id": "sensitivity-reading", "metadata": {}, "source": [ "**What this output tells you.** The full history gives -0.088 for the discount contrast and 0.057\n", "for the shipping contrast. Dropping `engaged` moves them to -0.095 and 0.054. The moves are -0.007\n", "and -0.003, which are -0.239 and -0.095 standard errors.\n", "\n", "Read the table as a benchmark, not as a bound. Dropping `engaged` hides a real cause of consent\n", "loss, so the move shows how much one recorded cause matters here. A move describes the dropped\n", "covariate only. An unrecorded cause of consent loss can act through other paths, and its move can be\n", "larger." ] }, { "cell_type": "markdown", "id": "method-evidence", "metadata": {}, "source": [ "## Method evidence\n", "\n", "The fits on this page use one draw of a synthetic law. The registered studies in the table\n", "measure each method over repeated draws. Each study page gives its law, its nuisance models,\n", "and the result of each cell.\n", "\n", "| fit on this page | registered study | law of the study |\n", "| --- | --- | --- |\n", "| Step 6, LTMLE | [LTMLE on declared known node mechanisms](../technical-reference/method-evidence/known-node-mechanisms-ltmle.md) | another law |\n", "| Step 7, g-computation | [Sequential g-computation and IPW](../technical-reference/method-evidence/sequential-baselines.md) | another law |\n", "\n", "No registered study covers three offer levels at three nodes.\n", "\n", "A review probe is a script under `reviews/notebook-review/probes/`. The step readings quote its\n", "results. A review probe is not a registered study.\n", "\n", "The [technical entry](../technical-reference/longitudinal-tmle.md#validation-issues-special-to-this-method) lists the evidence and the limits of the method." ] }, { "cell_type": "markdown", "id": "interpret", "metadata": {}, "source": [ "## What the analysis answers\n", "\n", "The question was which offer policy gives the largest share of full-price orders in week 4, and\n", "which policy beats which.\n", "\n", "| part of the answer | what the page supports |\n", "| --- | --- |\n", "| the estimate | if the assumptions below hold, win-back raises the week-4 full-price share by 0.0754 over no offers, with a 95% interval of (0.0353, 0.1154). The population is active subscribers whose subscription term runs through week 4. The intervention is the declared code templates, with tracking consent kept |\n", "| supported on this draw | win-back beats a discount every week. The paired interval (0.110, 0.217) excludes zero. A discount every week lowers the share, with an interval of (-0.1467, -0.0289) |\n", "| not resolved at this size | win-back against free shipping, and win-back then ship against win-back. Both paired intervals contain zero. In the review probe they excluded zero on 8 and 26 of 100 draws. The page does not compare win-back then ship with free shipping, so it names no best policy among the three |\n", "| evidence behind the reading | the exact offer probabilities, the support report, the synthetic deployment check, the benchmark of Step 11, and the review probes |\n", "| assumptions that stay unverified | consent exchangeability, consistency, and no interference |\n", "| conclusions the page does not support | a claim about applied data or another campaign, because the law is synthetic and no applied data access exists. A statement that the method is unbiased in general. A ranking of win-back and win-back then ship |" ] }, { "cell_type": "markdown", "id": "where-next", "metadata": {}, "source": [ "## Where to go next\n", "\n", "The [sequential pricing tutorial](sequential-pricing.ipynb) fits a continuous price at three monthly\n", "nodes in the same retailer program, with a learned density ratio. The [longitudinal user\n", "guide](../user-guide/longitudinal.md) shows how to read a long panel and how to report the\n", "sequential baselines. [Sequential g-computation and\n", "IPW](../technical-reference/sequential-baselines.md) states the g-computation estimator of Step 7.\n", "The [examples index](index.md#the-retailer-program) lists every tutorial in both programs." ] } ], "metadata": { "cleverly_execution": { "command": [ "python", "scripts/execute_notebook.py", "docs/examples/sequential-offers.ipynb" ], "gated": { "code_source_sha256": "8cf078b59925a4c9a88378b2335c78bbcaac7292004a862e6c3b3065bd77b2b9", "execution_payload_sha256": "333d651c5abec3a17e4aa5c82550da2ff341c9e7ac928a3f55235f9f8619325e" }, "recorded": { "cleverly_commit": "667928a10b7f09c2a203c19185536ee7327acdf4", "cleverly_version": "0.1.4", "cleverly_worktree_clean": false, "dependency_lock_sha256": "2e4f7ad3264ff9b35cf7588e5c93a0e20fc5c0852092ac1da365491543836682", "generator_files": [ "scripts/execute_notebook.py", "tests/notebooks.py" ], "generator_sha256": "f247771ab390b23e7e092d7eb35794e241c2ac84fc0526695df1cee1fc457a83", "package_source_sha256": "b9f057dd1103117f8f17907fe7c22b60f93d637e59f699ffcb35e658e8929998" }, "schema_version": 5 }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.16" } }, "nbformat": 4, "nbformat_minor": 5 }