Sequential offers: a weekly retail campaign#
A weekly offer changes who stays engaged, and engagement steers the next offer. This notebook estimates the value of offer policies with longitudinal TMLE, and it compares two policies on the joint influence curve. Each step shows its code, its output, and what the output tells you. Longitudinal TMLE gives the sequential regression and the algorithm.
The situation#
A subscription retailer sends one offer to each customer in each of three weeks. The offer is no code, a 15% discount code, or a free-shipping code. Before each send, the CRM scores whether the customer is engaged that week. The CRM draws each offer at random, with logged probabilities that read that engagement. A customer who withdraws tracking consent is lost to follow-up.
This page uses a synthetic law. The examples index states how it relates to the health program of the other tutorials.
element |
on this page |
|---|---|
data |
a long panel of 20000 synthetic customers, one row for each customer and week |
time order |
in weeks 1 to 3, the CRM scores engagement, sends the offer, and the customer keeps or withdraws tracking consent. In week 4, the customer orders or not |
treatment |
the offer of each week: |
outcome |
a full-price order in week 4, an order placed without a code |
observed |
the engagement scores of weeks 1 to 3, one offer path for each customer, and the order of each customer who kept tracking consent |
counterfactual |
the week-4 order under an offer policy that differs from the offers the customer received, with tracking consent kept. The panel never shows it |
The causal graph#
The graph draws one week. The pattern of week \(t\) repeats for \(t = 1, 2, 3\).
node |
meaning |
|---|---|
\(W\) |
the tenure |
\(L_t\) |
the engagement score of week \(t\), recorded before the offer. \(L_4\) is the score of week 4 |
\(A_t\) |
the offer of week \(t\) |
\(C_t\) |
the tracking consent after the send of week \(t\) |
\(D\) |
the count of discounts sent to engaged customers |
\(Y\) |
the week-4 order |
\(U\) |
an unrecorded cause |
W (tenure) --> L_1, L_2, L_3, L_4, C_1, C_2, C_3, and Y (no arrow W --> A_t)
One week t (t = 1, 2, 3):
C_t <-- L_t --> A_t --> L_{t+1} --> ... --> L_4 --> Y
| | ^
| v |
+-----> D --------------------------------------+
also: L_t --> L_{t+1}
U (unrecorded cause) - - -> C_t and Y (assumed absent)
arrow or absent arrow |
what it asserts |
role |
where the law sets it |
|---|---|---|---|
|
tenure shifts engagement, consent, and the order |
baseline confounder |
|
|
the logged probabilities read that week’s engagement |
confounder of the offer |
the logged probabilities |
|
a discount raises the next engagement, and free shipping raises it less |
the treatment changes a later confounder |
|
|
engagement persists |
a path from a confounder to |
|
|
engaged customers keep tracking consent more often |
censoring with a recorded cause |
|
|
a discount sent to an engaged customer lowers the order |
a direct offer path. Engagement modifies the discount effect |
|
|
week-4 engagement raises the order |
a mediator after the last send. The panel does not record it |
|
no unrecorded cause of |
the CRM randomizes |
encoded, and it holds by design |
the logged probabilities |
no |
consent loss depends only on the recorded history |
encoded, and open. This is scientific judgment, and Step 11 probes it |
none, an assumption |
no |
the offers ignore tenure |
a feature of this law. Identification does not need it |
the logged probabilities |
no |
a send does not change that week’s consent |
a feature of this law. Identification does not need it |
|
The graph has no competing event. The subscription term is fixed at time zero and runs through week 4, so no account closes before the outcome week.
Reading the graph. Engagement \(L_2\) lies on the path \(A_1 \to L_2 \to \dots \to Y\), and it also causes \(A_2\). One outcome regression that adjusts for \(L_2\) blocks part of the effect of \(A_1\). A regression that omits \(L_2\) leaves \(A_2\) confounded. The sequential g-formula handles both.
The arrow \(L_t \to C_t\) and the path from \(L_t\) to \(Y\) make the customers who keep tracking consent a selected group. The node \(D\) makes the discount effect depend on engagement, so a rule can beat every static policy. The two encoded absences are the exchangeability assumptions of the next section.
What you will learn#
after this notebook you can |
the step that shows it |
|---|---|
state a policy estimand when engagement feeds back |
the next section |
read from the graph why one regression fails and the sequential g-formula is needed |
the graph and Step 3 |
say what declared offer probabilities buy in LTMLE |
Steps 6 and 7 |
compare two policies on the joint influence curve, and state what the evidence does not establish |
Steps 8 to 11 and the final section |
The question, the estimand, and the method#
The retention lead asks one question. Which offer policy gives the largest share of full-price orders in week 4, and which policy beats which? A policy is a plan of three offers, not the effect of one send.
part of the question |
on this page |
|---|---|
target population |
active subscribers whose subscription term is fixed at the first weekly send and runs through week 4, as the protocol of Step 4 states |
intervention |
policy \(d\), one offer for each of weeks 1 to 3. Three policies are static: |
counterfactual outcome |
\(Y^{d,\bar c=1}\), the week-4 full-price order if the customer had followed \(d\) and kept tracking consent |
estimand |
\(\psi_d = E[Y^{d,\bar c=1}]\), the week-4 full-price share under \(d\) |
contrasts |
\(\psi_d - \psi_{\mathrm{none}}\) for the four other policies, and the difference of two such contrasts |
A one-week offer effect does not fix the later offers, so it defines no policy. An “ever discounted” comparison has the same flaw. The discount effect also depends on engagement, so a rule that reads engagement can beat every static policy.
assumption |
what it means here |
checkable from data, or scientific judgment |
|---|---|---|
offer exchangeability |
each offer is independent of the potential outcomes, given tenure, the engagement of that week and of earlier weeks, and the earlier offers |
by design. The CRM randomizes with logged probabilities, and the data hold them |
consent exchangeability |
consent loss depends only on the recorded history |
scientific judgment. Step 11 probes it with a benchmark |
positivity |
each history that a policy can reach has a chance of the policy’s offer and of staying consenting |
offers: by design, because the smallest logged probability is 0.25. Consent: empirical, read in Step 9 |
consistency |
an offer is the declared code template, and a code expires at the end of its week |
scientific judgment |
no interference |
one customer’s offers do not change another customer’s order |
scientific judgment. One account belongs to one customer |
Each method feature answers one feature of the problem.
feature of the problem |
feature of the method |
|---|---|
an earlier offer changes engagement, and engagement confounds the next offer |
sequential regression, which evaluates the g-formula with one regression for each week |
the CRM logged the offer probabilities |
|
engagement drives consent loss |
a consent model that reads the recorded history |
the nuisance models are main-terms parametric models |
no cross-fitting. These models have few parameters, and the registered known-mechanism cells run in sample |
the question compares policies |
the joint influence curve gives the interval of a difference of two contrasts |
Three estimators could answer the question. The table states what each one reads and when it is consistent. Known node mechanisms and sequential g-computation and IPW give the conditions.
estimator |
what it reads |
consistent when |
|---|---|---|
g-computation |
the outcome regressions |
every outcome regression is correct |
IPW |
the declared offer probabilities and the consent model |
the consent model is correct |
LTMLE |
all of them |
every outcome regression is correct, or the consent model is correct, because the offer probabilities are exact. Both hold up to the truncation that Step 9 reads |
The table defines the terms this page uses most.
term |
plain meaning |
|---|---|
node |
one decision time. Here, the weekly send in weeks 1, 2, and 3 |
history |
every column recorded before the send at a node |
policy |
a rule for each node. An entry is one offer for everyone, or a rule that reads the history. The code calls a policy a plan |
win-back |
the policy that sends the discount to a customer who is not engaged, and no offer otherwise |
declared offer mechanism |
the logged probability of each offer, read from columns. The fit runs no offer model |
a model the estimate needs but the question does not ask about. Here, the outcome, pseudo-outcome, and consent models |
|
a small update to each node’s regression, weighted by one over the cumulative probability of following the policy and staying consenting, that removes first-order bias |
|
how much each customer moves the estimate. Its variance gives the standard error |
|
sequential positivity |
at every node, each history that the policy can reach has some chance of the policy’s offer and of staying consenting |
Step 1: set up#
The setup imports the libraries and prints the installed cleverly version. Every fit below passes
its learners, its cross-fitting setting, and its random seed explicitly, so a rerun reproduces the
stored outputs.
import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier, DummyRegressor
from sklearn.linear_model import LinearRegression, LogisticRegression
import cleverly
pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 20)
print("cleverly", cleverly.__version__)
cleverly 0.1.4
What this output tells you. The stored outputs in this notebook came from the version named above. A different version can print different numbers.
Step 2: the data#
The generator make_sequential_offers returns a long panel with one row per customer and week. A
customer who withdraws consent has no later rows. The code prints the shape and the column names. It
counts the customers who withdrew consent, and it prints the true values of the synthetic law.
The seed follows a fixed rule. The page uses 1000, the first seed of the range 1000 to 1099 that the review probe of Step 6 sweeps. The pilot of Step 10 uses that seed plus 100000. The rule was set before the page ran with this seed. No draw was chosen for its results, so an interval on this page can miss its true value.
from cleverly.datasets import make_sequential_offers
panel, truth = make_sequential_offers(20_000, seed=1000)
print("rows and columns:", panel.shape)
print("customers:", panel["customer"].nunique())
print(list(panel.columns))
print(panel.head(6).round(3))
print()
left = (panel.groupby("customer")["consent"].min() == 0).sum()
print(f"customers who withdrew consent before week 4: {left} of {panel['customer'].nunique()}")
print("known values of the synthetic law:")
for name, value in truth.items():
print(f" {name:36s} {value:.4f}")
rows and columns: (57154, 10)
customers: 20000
['customer', 'week', 'tenure', 'engaged', 'offer', 'consent', 'p_none', 'p_discount', 'p_shipping', 'full_price']
customer week tenure engaged offer consent p_none p_discount p_shipping full_price
0 0 1 -0.321 1 none 1 0.4 0.25 0.35 1.0
1 0 2 -0.321 0 none 1 0.3 0.45 0.25 1.0
2 0 3 -0.321 0 shipping 1 0.3 0.45 0.25 1.0
3 1 1 -0.486 1 shipping 1 0.4 0.25 0.35 NaN
4 1 2 -0.486 1 none 1 0.4 0.25 0.35 NaN
5 1 3 -0.486 0 shipping 0 0.3 0.45 0.25 NaN
customers who withdrew consent before week 4: 2686 of 20000
known values of the synthetic law:
ey_regimen[none] 0.5275
ey_regimen[discount] 0.4596
ey_regimen[shipping] 0.5865
ey_regimen[win_back] 0.5997
ey_regimen[win_back_ship] 0.6268
ate_regimen[discount vs none] -0.0680
ate_regimen[shipping vs none] 0.0590
ate_regimen[win_back vs none] 0.0722
ate_regimen[win_back_ship vs none] 0.0993
What this output tells you. The panel has 57154 rows and 10 columns for 20000 customers. Each customer has up to three rows.
column |
role |
|---|---|
|
the unit and the decision time |
|
a baseline covariate, a standard-normal draw |
|
the engagement score of that week, recorded before the offer |
|
the offer sent that week: |
|
1 if the customer still allows tracking after that send, and 0 if the customer withdrew |
|
the logged probabilities of the three offers, for that customer and week |
|
the week-4 outcome. It repeats on each row, and it is missing for a customer who withdrew |
The CRM chose the probabilities, so they are known. Engaged customers have 0.40 for no offer and 0.25 for a discount. Customers who are not engaged have 0.30 and 0.45.
A total of 2686 customers withdrew consent before week 4, and they have no outcome. The estimator expects that shape. A complete-case frame would discard those customers before any model sees them.
The true values come from offers_truth(), which enumerates the law and does not simulate it. The
true full-price rate is 0.5275 with no offers, 0.4596 with a discount every week, and 0.5865 with
free shipping every week. Win-back gives 0.5997, and win-back then ship gives 0.6268.
A discount every week ranks last, and win-back then ship ranks first. The true contrast of win-back
against no offers is 0.0722. A real program has no truth. Every comparison against it below is a
teaching device.
Step 3: association first#
A confounder is a variable that changes both the next offer and the outcome. Engagement in weeks 2 and 3 is one, and the earlier offer also changes it. The code reads three arrows of the graph in the logs. It then compares the customers who happened to follow each policy in all three weeks with the customers who followed no offers. The comparison keeps only the customers who still consented after week 3. This crude contrast adjusts for no covariate, and it is a teaching device.
from cleverly.datasets import OFFER_PLANS
week_one = panel[panel["week"] == 1]
week_two = panel[panel["week"] == 2][["customer", "engaged"]]
moved = week_one.merge(week_two, on="customer", suffixes=("", "_week2"))
print("arrow A_1 -> L_2, share engaged in week 2, by the week-1 offer:")
print(moved.groupby("offer")["engaged_week2"].mean().round(3).to_string())
print()
print("arrow L_1 -> A_1, share of each week-1 offer, by week-1 engagement:")
print(pd.crosstab(week_one["engaged"], week_one["offer"], normalize="index").round(3).to_string())
print()
print("arrow L_1 -> C_1, share still consenting after the week-1 send, by week-1 engagement:")
print(week_one.groupby("engaged")["consent"].mean().round(3).to_string())
print()
wide = panel.pivot(index="customer", columns="week")
kept = wide[wide[("consent", 3)] == 1]
first_engaged = kept[("engaged", 1)].astype(int).to_numpy()
rows = []
for name, rule in OFFER_PLANS.items():
followed = np.ones(len(kept), dtype=bool)
for week in (1, 2, 3):
engaged = kept[("engaged", week)].astype(int).to_numpy()
followed &= kept[("offer", week)].to_numpy() == np.asarray(rule(engaged), dtype=object)
rows.append(
{
"plan": name,
"followers": int(followed.sum()),
"share": kept[("full_price", 1)][followed].mean(),
"engaged_week1": first_engaged[followed].mean(),
}
)
crude = pd.DataFrame(rows).set_index("plan")
crude["crude_vs_none"] = crude["share"] - crude.loc["none", "share"]
crude["true_vs_none"] = [truth.get(f"ate_regimen[{name} vs none]", 0.0) for name in crude.index]
print(f"customers still consenting after week 3: {len(kept)} of {len(wide)}")
print(f"share engaged in week 1 among these customers: {first_engaged.mean():.3f}")
print(crude.round(3).to_string())
arrow A_1 -> L_2, share engaged in week 2, by the week-1 offer:
offer
discount 0.886
none 0.551
shipping 0.661
arrow L_1 -> A_1, share of each week-1 offer, by week-1 engagement:
offer discount none shipping
engaged
0 0.440 0.308 0.251
1 0.246 0.399 0.355
arrow L_1 -> C_1, share still consenting after the week-1 send, by week-1 engagement:
engaged
0 0.929
1 0.969
customers still consenting after week 3: 17314 of 20000
share engaged in week 1 among these customers: 0.562
followers share engaged_week1 crude_vs_none true_vs_none
plan
none 805 0.555 0.646 0.000 0.000
discount 436 0.445 0.404 -0.110 -0.068
shipping 550 0.609 0.671 0.054 0.059
win_back 1247 0.604 0.524 0.049 0.072
win_back_ship 956 0.606 0.513 0.050 0.099
What this output tells you. Each of the first three tables confirms one arrow of the graph.
arrow |
what the table shows |
|---|---|
|
after a discount, a share of 0.886 of the customers are engaged in week 2. After no offer the share is 0.551, and after free shipping it is 0.661 |
|
engaged customers receive no offer at a share of 0.399 and a discount at 0.246. Customers who are not engaged receive a discount at 0.440 |
|
the share still consenting is 0.969 for engaged customers and 0.929 for the others |
Engagement is therefore a confounder of the later offers, and the earlier offer changes it. One outcome regression cannot condition on it without blocking part of the earlier effect. It cannot leave it out without confounding the later offer.
The last table is the crude comparison. It keeps 17314 of 20000 customers, those who consented through week 3. The crude contrast of win-back against no offers is 0.049, and the true contrast is 0.072. For a discount every week the crude contrast is -0.110 and the true contrast is -0.068.
The crude contrast has a systematic bias for the rules. The CRM sends no offer to engaged customers
more often than to others, 0.399 against 0.308 in week 1. So the customers who followed none are
more often engaged in week 1, at 0.646 against 0.562 among all kept customers. Engaged customers
order more. That lifts the reference group, and it lowers the crude contrast of each rule.
Free shipping also favors engaged customers, at 0.355 against 0.251 in week 1. Its followers are
engaged in week 1 at 0.671, against 0.646 for the followers of none. So the reference group is not
more engaged than they are. On this draw, the crude contrast of 0.054 sits near the true 0.059.
In the review probe, the crude win-back contrast fell below the truth on 96 of 100 draws, by 0.0314 on average. The Monte Carlo standard error was 0.0019. The crude shipping contrast had a mean bias of -0.0061 (Monte Carlo standard error 0.0029). That bias is small, at about two standard errors, and the probe does not show that it is zero.
Step 4: write the protocol#
A StudyProtocol records the scientific design before any model runs.
Point-treatment TMLE introduces its fields
and its fingerprint. The helper offers_protocol() returns the record for this campaign. It names the five
policies of the question, and it states how the design treats consent withdrawal and expired codes.
from cleverly.datasets import offers_protocol
protocol = offers_protocol()
print("\n".join(protocol.summary_lines()))
causal study protocol: schema 1; 91f189a988d70fec
target population: Active subscribers at the start of the campaign window
eligibility: ['Active subscription at the first weekly send', 'Marketing consent at the first weekly send', 'At least one prior order', 'One account per customer, so shared-household accounts are excluded', 'A subscription term that is fixed at the first weekly send and runs through week 4']
time zero: The first weekly send, after the week-1 engagement score and before the first offer
treatment strategies: ['Send no offer in weeks 1 to 3', 'Send the 15% discount code every week', 'Send the free-shipping code every week', 'Win-back: send the discount code when the customer is not engaged, otherwise send no offer', 'Win-back then ship: send the discount code when the customer is not engaged, otherwise send the free-shipping code']
treatment versions: ['No code sent', 'The declared 15% code, one send per week, valid until the end of that week', 'The declared free-shipping code, one send per week, valid until the end of that week', 'The declared 15% code for customers who are not engaged at the send, no code otherwise', 'The declared 15% code for customers who are not engaged at the send, and the declared free-shipping code otherwise']
outcome: A full-price order in week 4, an order placed without a code
horizon: Week 4, the week after the third weekly send
intercurrent-event handling: ['Tracking-consent withdrawal before week 4 censors the customer. The strategy is hypothetical: the order had tracking continued', 'Each code expires at the end of its week, so no code is redeemable in week 4', 'None: the term runs through week 4, so no account closes before the outcome week']
interference unit: Individual customer
assumption rationale: ['Randomization with logged probabilities gives sequential exchangeability for the offers, given tenure and the weekly engagement score', 'Tenure and weekly engagement are the recorded confounders, and they cover the measured causes of consent loss', 'Declared code templates, one send per week, support consistency', 'One account per customer and per-customer codes support no interference']
What this output tells you. The first line gives the schema version and the fingerprint
94024dfd011817af. The other lines repeat each field. Read them as a checklist.
protocol field |
the question it answers for this campaign |
|---|---|
target population and eligibility |
who the effect is about. Eligibility includes a subscription term that is fixed at the first weekly send and runs through week 4. Time zero fixes the population, so no later event defines it |
time zero |
the first weekly send, after the week-1 engagement score and before the first offer |
treatment strategies and versions |
the five policies, each with the declared code template. Each code is valid until the end of its week |
outcome and horizon |
a full-price order in week 4 |
intercurrent-event handling |
consent withdrawal censors the customer under a hypothetical strategy: the order had tracking continued. A code expires at the end of its week, so no code is redeemable in week 4 |
interference unit |
the individual customer |
assumption rationale |
randomization with logged probabilities gives sequential exchangeability for the offers, given tenure and the weekly engagement. Tenure and engagement are the recorded causes of consent loss |
The eligibility entry Marketing consent and the consent column are two consents. Marketing
consent lets the retailer send offers, and it is a condition of the population. The consent column
records tracking consent. Its withdrawal censors the customer.
Step 5: design and identification#
LongPanel reads the long panel. It names the unit, the time, the treatment, and the baseline
covariate. The argument time_varying lists the columns that change by week. The argument
censoring names the consent column. The argument treatment_probabilities maps each offer level
to its logged-probability column. That mapping declares the offer mechanism, so the fit runs no
offer model.
A plan is a tuple with one entry for each week, or a DynamicRegimen. The rule of a
DynamicRegimen reads the history of that week. The declaration rule_kind="known" says that the
rule is a fixed function of each customer’s own history, chosen before the fit. The each_node
helper applies one function at every week. The typed RegimeContrast compares each plan with
none.
from cleverly import CausalStudy, LongPanel, RegimeContrast
from cleverly.datasets import OFFER_WEEKS, win_back, win_back_then_ship
from cleverly.longitudinal import DynamicRegimen, each_node
LOGGED = {"none": "p_none", "discount": "p_discount", "shipping": "p_shipping"}
def offers_design(*, time_varying=("engaged",)):
return LongPanel(
unit="customer",
time="week",
treatment="offer",
baseline=["tenure"],
time_varying=list(time_varying),
censoring="consent",
outcome="full_price",
treatment_probabilities=LOGGED,
)
def rule(label, offer_for):
return DynamicRegimen(
label,
each_node(lambda history, week: offer_for(history[f"engaged@{week}"]), OFFER_WEEKS),
rule_kind="known",
)
study = CausalStudy(panel, design=offers_design(), protocol=protocol)
for week in (1, 2, 3):
print(f"history_names({week}):", study.data.history_names(week))
print()
plans = {
"none": ("none",) * OFFER_WEEKS,
"discount": ("discount",) * OFFER_WEEKS,
"shipping": ("shipping",) * OFFER_WEEKS,
"win_back": rule("win_back", win_back),
"win_back_ship": rule("win_back_ship", win_back_then_ship),
}
effect = study.identify(RegimeContrast(plans, reference="none"))
print(effect.summary(protocol="fingerprint"))
history_names(1): ('tenure', 'engaged@1')
history_names(2): ('tenure', 'engaged@1', 'engaged@2')
history_names(3): ('tenure', 'engaged@1', 'engaged@2', 'engaged@3')
contrast of each regime against the reference
identified by explicit-adjustment: sequential g-formula under the declared treatment regimen
history at offer@1: ['tenure', 'engaged@1']
history at offer@2: ['tenure', 'engaged@1', 'engaged@2', 'offer@1']
history at offer@3: ['tenure', 'engaged@1', 'engaged@2', 'engaged@3', 'offer@1', 'offer@2']
required nuisances: ['sequential_outcome_regressions', 'treatment_and_censoring_mechanisms']
assumptions:
- consistency: each observed history equals the potential history under its realized regimen
- no interference: one unit's potential history does not depend on other units' regimens
- sequential exchangeability given the recorded history at every node
- sequential positivity for treatment and remaining under observation
causal study protocol: schema 1; 91f189a988d70fec
What this output tells you. The three history_names lines list the columns that the design
puts in the history of each week. Week 1 holds tenure and engaged@1, the confounder of the first
offer. Each later week adds that week’s engagement. The earlier offers join the history through the
policy, and the history at lines of the summary show them.
The history at offer@3 line lists engaged@1, engaged@2, engaged@3, offer@1, and offer@2,
with tenure. No line lists the week-4 engagement, which no offer follows. The summary names the
estimand and its identification, the sequential g-formula under the declared treatment regimen. The
required nuisances are the sequential outcome regressions and the treatment and censoring
mechanisms. The last line prints the fingerprint from Step 4.
The four printed assumptions are those of the table in the plan. Offer exchangeability holds by design. The consent part of sequential exchangeability does not, and Step 11 probes it. Step 9 reads the empirical part of sequential positivity.
Step 6: estimate the contrasts with LTMLE#
The configuration is written out in full.
setting |
value |
what it does |
|---|---|---|
|
logistic regression |
fits the regression of the binary outcome at the last node |
|
linear regression |
fits the intermediate regressions. Their targets are continuous, and the recursion clips their predictions to the unit interval |
|
logistic regression |
fits the probability of staying consenting at each node, given the history |
|
none |
the offer mechanism is declared with |
|
in-sample fit |
the nuisance models are main-terms parametric models, and the registered known-mechanism cells run in sample |
|
the default |
bounds each cumulative product of offer and consent probabilities from below, so no weight exceeds 100. Step 9 reads its effect |
|
fixed seed, one process |
makes the fit reproducible |
The code then compares each contrast with its true value. The helper against_truth returns the
estimate, the standard error, the interval, and whether the interval contains the truth.
from cleverly import CrossFitting, ModelSpec, Runtime, TMLEMethod
logistic = LogisticRegression(max_iter=1000, random_state=42)
ltmle = TMLEMethod(
models=ModelSpec(
outcome_learner=logistic,
pseudo_learner=LinearRegression(),
censoring_learner=logistic,
),
cross_fitting=CrossFitting(enabled=False),
runtime=Runtime(random_state=42, n_jobs=1),
)
result = effect.estimate(method=ltmle)
print(result.summary(protocol="fingerprint"))
def against_truth(fit):
rows = []
for key, estimate in fit.estimates.items():
low, high = estimate.ci
rows.append(
{
"contrast": key.removeprefix("ate_regimen[").removesuffix(" vs none]"),
"estimate": estimate.psi,
"std_error": estimate.std_error,
"ci_low": low,
"ci_high": high,
"truth": truth[key],
"covers": bool(low <= truth[key] <= high),
}
)
return pd.DataFrame(rows).set_index("contrast")
print()
print(against_truth(result).round(3).to_string())
Longitudinal TMLE (3 time points, n = 20000)
parameter estimate std. error 95% CI p-value support
---------------------------------- -------- ---------- ------------------ ------- ------------
ate_regimen[discount vs none] -0.0878 0.0300 [-0.1467, -0.0289] 0.0035 experimental
ate_regimen[shipping vs none] 0.0569 0.0284 [0.0011, 0.1126] 0.0456 experimental
ate_regimen[win_back vs none] 0.0754 0.0204 [0.0353, 0.1154] 0.0002 experimental
ate_regimen[win_back_ship vs none] 0.0820 0.0237 [0.0355, 0.1285] 0.0005 experimental
experimental: No registered study covers this configuration. Nearest row: G01 (canonical-ltmle), differing on treatment, qualifiers.
time points: 3
long panel: unit column 'customer', decision times 1, 2, 3 (column 'week')
outcome family: binomial
regimens: none=(none/none/none), discount=(discount/discount/discount), shipping=(shipping/shipping/shipping), win_back=(d:d@1/d:d@2/d:d@3), win_back_ship=(d:d@1/d:d@2/d:d@3)
a 'd' is a rule d_t(H_t) read off [W, L_1, ..., L_t]; its followers are a covariate-dependent set, so the counts below describe this sample
assigned arms, win_back: 6e5d6c23174182a2
assigned arms, win_back_ship: 6ee945a698bd4b75
reference: none
cross-fitting: none -- nuisances fitted in sample
g_bounds: fixed [0.01, 1] on each cumulative treatment-and-censoring probability (package default; R ltmle-compatible heuristic -- inspect truncation diagnostics)
known factors (declared, no learner): offer@1, offer@2, offer@3; no truncation bound moved a declared value
confidence level: 95%
random_state: 42
causal estimand: contrast of each regime against the reference
identification: explicit-adjustment; sequential g-formula under the declared treatment regimen
required nuisances: sequential_outcome_regressions, treatment_and_censoring_mechanisms
identification assumptions: consistency: each observed history equals the potential history under its realized regimen; no interference: one unit's potential history does not depend on other units' regimens; sequential exchangeability given the recorded history at every node; sequential positivity for treatment and remaining under observation
causal study protocol: schema 1; 91f189a988d70fec
data 4ff95105cdca0278 | folds 738358eab9007cbb | cleverly 0.1.4 | 2026-10-16T19:55:23+00:00
none: 805 of 20000 units followed it throughout; max weight 59.7, effective n 709; max truncated share 0.0% at t=1
discount: 436 of 20000 units followed it throughout; max weight 80.9, effective n 379; max truncated share 0.0% at t=1
shipping: 550 of 20000 units followed it throughout; max weight 100.0, effective n 475; max truncated share 0.2% at t=3
win_back: 1247 of 20000 units followed it throughout; max weight 19.7, effective n 1242; max truncated share 0.0% at t=1
win_back_ship: 956 of 20000 units followed it throughout; max weight 28.6, effective n 936; max truncated share 0.0% at t=1
simultaneous 95% bands (multiplier bootstrap, critical value 2.411 vs 1.960 pointwise) [experimental]:
ate_regimen[discount vs none] [-0.1602, -0.0154]
ate_regimen[shipping vs none] [-0.0117, 0.1254]
ate_regimen[win_back vs none] [0.0261, 0.1247]
ate_regimen[win_back_ship vs none] [0.0248, 0.1392]
estimate std_error ci_low ci_high truth covers
contrast
discount -0.088 0.030 -0.147 -0.029 -0.068 True
shipping 0.057 0.028 0.001 0.113 0.059 True
win_back 0.075 0.020 0.035 0.115 0.072 True
win_back_ship 0.082 0.024 0.035 0.128 0.099 True
What this output tells you. The table reports four contrasts against none. The win-back
contrast is 0.0754, with a standard error of 0.0204 and a 95% interval of (0.0353, 0.1154). The
interval contains the true contrast of 0.0722.
The discount contrast is -0.0878 with an interval of (-0.1467, -0.0289), and it contains the true -0.0680. The shipping contrast is 0.0569 with an interval of (0.0011, 0.1126), and its p-value is 0.0456. That interval excludes zero by a narrow margin. The win-back-then-ship contrast is 0.0820 with an interval of (0.0355, 0.1285). All four intervals contain their true values on this draw. That is one draw, not a coverage result.
The review probe
reviews/notebook-review/probes/sequential-offers/sweep.py repeats the fits of this page on 100
draws, seeds 1000 to 1099. The four intervals covered on 95, 95, 92, and 94 of 100 draws.
The support column marks the four contrasts and the bands experimental, because no registered
study covers this configuration. The support label makes no coverage claim.
The simultaneous bands use a critical value of 2.411 in place of 1.960. They answer which policies
differ from none for the four contrasts together. The bands of discount, win-back, and win-back
then ship exclude zero. The band of shipping, (-0.0117, 0.1254), contains it.
Step 7: what the logged offer probabilities buy#
The offer probabilities are known, and the outcome regressions are not. The main-terms regressions of Step 6 are wrong. They omit the fatigue interaction of the offer and the engagement. The true regression also averages over the unrecorded week-4 engagement, which interacts with tenure. The Step 6 estimate therefore already rests on the consent model.
This step asks what LTMLE gains when the outcome regressions are wrong. It compares g-computation and LTMLE under a deliberately wrong outcome model.
for the comparison |
on this page |
|---|---|
estimands |
both estimators target the |
assumptions added |
g-computation adds that every outcome regression is correct. LTMLE adds that the consent model is correct, and it uses the exact offer probabilities |
mechanism |
the wrong outcome model returns the mean outcome of the customers who followed the policy and reads no history. The g-computation plug-in has no term that corrects it. The LTMLE targeting step regresses the residuals on a covariate built from the exact offer probabilities and the fitted consent model, and that update repairs the plug-in |
evidence |
the registered cell |
reading |
one draw shows no general property |
The wrong outcome model is a deliberate mutation, and no analyst would choose it. A prior-only classifier fits the last node, and a mean regressor fits the earlier nodes. The consent model is the logistic regression of Step 6.
G-computation reports a point estimate only with these learners, because its interval needs GLM
outcome regressions. The code prints one row for each contrast. A row holds the truth, the Step 6
estimate, and the crude contrast of Step 3. It also holds the wrong-model LTMLE estimate with its
interval, and the wrong-model g-computation estimate.
from cleverly import GComputationMethod
blind = ModelSpec(
outcome_learner=DummyClassifier(strategy="prior"),
pseudo_learner=DummyRegressor(),
censoring_learner=logistic,
)
blind_ltmle = effect.estimate(
method=TMLEMethod(models=blind, cross_fitting=ltmle.cross_fitting, runtime=ltmle.runtime)
)
blind_gcomp = effect.estimate(method=GComputationMethod(models=blind, runtime=ltmle.runtime))
print(
"same estimands:",
blind_ltmle.estimates.keys() == blind_gcomp.estimates.keys() == result.estimates.keys(),
)
rows = []
for key, estimate in blind_ltmle.estimates.items():
low, high = estimate.ci
name = key.removeprefix("ate_regimen[").removesuffix(" vs none]")
rows.append(
{
"contrast": name,
"truth": truth[key],
"ltmle": result[key].psi,
"ltmle_se": result[key].std_error,
"blind_ltmle": estimate.psi,
"blind_se": estimate.std_error,
"ci_low": low,
"ci_high": high,
"covers": bool(low <= truth[key] <= high),
"blind_gcomp": blind_gcomp[key].psi,
"crude": crude.loc[name, "crude_vs_none"],
}
)
blind_table = pd.DataFrame(rows).set_index("contrast")
print(blind_table.round(3).to_string())
same estimands: True
truth ltmle ltmle_se blind_ltmle blind_se ci_low ci_high covers blind_gcomp crude
contrast
discount -0.068 -0.088 0.030 -0.096 0.033 -0.160 -0.032 True -0.110 -0.110
shipping 0.059 0.057 0.028 0.068 0.030 0.009 0.126 True 0.054 0.054
win_back 0.072 0.075 0.020 0.080 0.022 0.038 0.123 True 0.049 0.049
win_back_ship 0.099 0.082 0.024 0.088 0.025 0.038 0.138 True 0.050 0.050
What this output tells you. The line same estimands: True confirms that the three fits report
the same contrasts, so their numbers compare.
On this draw the wrong-model g-computation estimate for win-back against none is 0.049, and the
truth is 0.072. For win-back then ship it is 0.050 against 0.099. The blind_gcomp column equals
the crude column on every row. The wrong model returns the mean outcome of the followers, which is
the crude follower mean. So g-computation under this model reproduces the crude bias of Step 3.
The wrong-model LTMLE estimate for win-back is 0.080, with an interval of (0.038, 0.123) that contains the truth. The Step 6 estimate is 0.075. All four wrong-model LTMLE intervals contain their true values on this draw. On this draw, the wrong-model LTMLE estimates sit nearer to the truth than the g-computation estimates for discount, win-back, and win-back then ship. The cost is a wider standard error, 0.022 against 0.020 for win-back.
One draw shows no general property. In the review probe, the wrong-model g-computation estimate for win-back fell below the truth by 0.0314 on average over 100 draws. The wrong-model LTMLE bias was -0.0060, and its interval covered the truth on 93 of 100 draws.
Validation: what each check can and cannot show#
Steps 9 to 11 check the fit. None of them checks an identification assumption. Each reading names what its output does not show.
Step 9: diagnostics, what the fit can check#
result.assess() runs the checks that apply to the fit. The code prints the operations that ran. It
then prints the support report. The report shows how many customers followed each policy at each
node, how heavy their weights are, and how many weights a bound replaced. A weight is one over the
cumulative probability of following the policy and staying consenting. The code ends with the
consent model of each node, measured on the rows it fit.
assessment = result.assess()
ledger = assessment.to_frame()
ran = ledger[~ledger["status"].isin(["unavailable", "not_applicable", "deferred"])]
print(ran[["surface", "check", "status"]].to_string(index=False))
support = assessment.report("support").to_frame()
columns = ["regimen", "time", "n_followed", "max_weight", "effective_n", "share_truncated"]
print()
print(support[columns].round(3).to_string(index=False))
kish_ratio = (support["effective_n"] / support["n_followed"]).min()
print(f"minimum effective-sample-size ratio: {kish_ratio:.1%}")
print()
nuisance = assessment.report("nuisance_models").to_frame()
consent = nuisance[nuisance["role"] == "censoring"]
print(
consent[["role", "time", "n", "evaluation", "auc", "calibration_slope"]]
.round(3)
.to_string(index=False)
)
surface check status
validation score_equations passed
validation support completed
validation nuisance_models completed
regimen time n_followed max_weight effective_n share_truncated
none 1 6814 4.007 6628.778 0.000
none 2 2308 16.543 2140.520 0.000
none 3 805 59.676 708.798 0.000
discount 1 6333 4.316 5883.702 0.000
discount 2 1630 18.759 1449.934 0.000
discount 3 436 80.877 379.489 0.000
shipping 1 5870 5.053 5663.046 0.000
shipping 2 1766 26.343 1605.188 0.000
shipping 3 550 100.000 475.304 0.002
win_back 1 7953 2.728 7938.502 0.000
win_back 2 3186 7.548 3177.376 0.000
win_back 3 1247 19.651 1241.930 0.000
win_back_ship 1 7499 3.108 7413.283 0.000
win_back_ship 2 2665 9.424 2621.040 0.000
win_back_ship 3 956 28.592 935.762 0.000
minimum effective-sample-size ratio: 86.4%
role time n evaluation auc calibration_slope
censoring 1 20000 in_sample 0.647 1.003
censoring 2 19017 in_sample 0.645 1.004
censoring 3 18137 in_sample 0.650 1.000
What this output tells you. The first table lists the operations that ran. The score-equation
check is passed. The support report and the nuisance report are completed. A completed result
means the calculation ran. It is not a pass.
The support table divides by the declared offer probabilities and the fitted consent model. It describes those models. It cannot see an unrecorded cause of consent loss.
policy |
followers at week 3 |
max weight |
effective n |
what it shows |
|---|---|---|---|---|
|
805 |
59.676 |
708.798 |
thin support |
|
436 |
80.877 |
379.489 |
the thinnest support of the static policies |
|
550 |
100.000 |
475.304 |
the heaviest weight, at the cap |
|
1247 |
19.651 |
1241.930 |
the rule keeps the most customers |
|
956 |
28.592 |
935.762 |
the rule keeps more customers than any static policy |
A static policy sends the same offer to engaged and unengaged customers. The logged probability of that offer is lower in one of the two groups, so few customers follow the policy. Win-back sends the likeliest logged offer in each group, so it keeps the most customers.
The minimum ratio of effective sample size to followers is 86.4%. One row has a truncated weight.
The shipping policy at week 3 has a truncated share of 0.002, which is 1 of its 550 followers. The
offer probabilities alone cannot reach the cap, because their product is at least 0.0156. The weight
reached the cap of 100 through the consent probabilities, for a customer who has low tenure and was
not engaged in any week.
The consent models have an auc of 0.647, 0.645, and 0.650 at weeks 1, 2, and 3. They separate the
customers poorly, because the chance of consent loss varies little with engagement and tenure. The
calibration slopes are 1.003, 1.004, and 1.000. The models are fitted in sample, so a slope near 1
does not show that a model is correct.
Step 10: a synthetic deployment check#
A real campaign would run the win-back policy on a pilot cohort and compare its observed rate with
the estimate. No applied data access exists for this tutorial, so the pilot is synthetic. The
policy="win_back" argument draws a fresh cohort of 10000 customers under that policy. In the
cohort, every customer consents and stays tracked. This is a design condition of the pilot: a
consented panel with no loss.
The code fits the win-back level with RegimeMean and compares it with the pilot mean. The pilot is
independent of the estimation sample, so the standard error of the difference is
sqrt(se_fit^2 + se_pilot^2).
from cleverly import RegimeMean
pilot, _ = make_sequential_offers(10_000, seed=101_000, policy="win_back")
pilot_outcome = pilot.loc[pilot["week"] == 3, "full_price"]
levels = study.identify(RegimeMean(plans)).estimate(method=ltmle)
fitted = levels["ey_regimen[win_back]"]
pilot_mean = float(pilot_outcome.mean())
pilot_se = float(pilot_outcome.std() / np.sqrt(len(pilot_outcome)))
difference = fitted.psi - pilot_mean
difference_se = float(np.hypot(fitted.std_error, pilot_se))
print(
f"pilot customers: {len(pilot_outcome)}; lost to consent: {int((pilot['consent'] == 0).sum())}"
)
print(f"pilot share of full-price orders: {pilot_mean:.3f} (standard error {pilot_se:.4f})")
print(
f"fitted win-back level: {fitted.psi:.3f} (standard error {fitted.std_error:.4f})"
)
print(f"difference, fit minus pilot: {difference:+.4f} (standard error {difference_se:.4f})")
print(f"difference in standard errors: {difference / difference_se:+.2f}")
print(f"population win-back level: {truth['ey_regimen[win_back]']:.3f}")
pilot customers: 10000; lost to consent: 0
pilot share of full-price orders: 0.609 (standard error 0.0049)
fitted win-back level: 0.596 (standard error 0.0133)
difference, fit minus pilot: -0.0135 (standard error 0.0141)
difference in standard errors: -0.95
population win-back level: 0.600
What this output tells you. The pilot has 10000 customers and loses none to consent. Its full-price rate is 0.609, with a standard error of 0.0049. The fitted win-back level is 0.596, with a standard error of 0.0133. The difference is -0.0135 with a standard error of 0.0141, which is -0.95 standard errors. The population value is 0.600.
The two numbers differ by less than 1.96 standard errors on this draw. On applied data, a failed exchangeability or consent assumption could move the fit away from a pilot. Here both draws come from one law in which the assumptions hold, so the check tests the estimator only. A small difference does not prove either assumption.
Step 11: consent exchangeability, how much a recorded cause moves the estimate#
The offers hold exchangeability by design, because the CRM drew them at random with the logged
probabilities. The assumption at risk is consent exchangeability. The fit assumes that consent loss
depends only on the recorded history, which excludes the unrecorded cause \(U\) of the graph.
Engagement is a recorded cause of consent loss in this law, with tenure.
The technical reference derives no omitted-variable bound for a longitudinal fit, so this step runs
a benchmark. The code refits the two static contrasts without engaged in the history. The declared
offer probabilities stay the same in both fits. The move then comes from the outcome and consent
models. The win-back rules read engaged, so they cannot run without it. The covariate tenure is
the only baseline covariate, so the code does not drop it.
static = {name: plans[name] for name in ("none", "discount", "shipping")}
full_fit = study.identify(RegimeContrast(static, reference="none")).estimate(method=ltmle)
reduced_study = CausalStudy(panel, design=offers_design(time_varying=()), protocol=protocol)
reduced_fit = reduced_study.identify(RegimeContrast(static, reference="none")).estimate(
method=ltmle
)
rows = []
for key, estimate in full_fit.estimates.items():
move = reduced_fit[key].psi - estimate.psi
rows.append(
{
"contrast": key.removeprefix("ate_regimen[").removesuffix(" vs none]"),
"with_engagement": estimate.psi,
"without": reduced_fit[key].psi,
"move": move,
"move_in_se": move / estimate.std_error,
}
)
benchmark = pd.DataFrame(rows).set_index("contrast")
print(benchmark.round(3).to_string())
with_engagement without move move_in_se
contrast
discount -0.088 -0.095 -0.007 -0.239
shipping 0.057 0.054 -0.003 -0.095
What this output tells you. The full history gives -0.088 for the discount contrast and 0.057
for the shipping contrast. Dropping engaged moves them to -0.095 and 0.054. The moves are -0.007
and -0.003, which are -0.239 and -0.095 standard errors.
Read the table as a benchmark, not as a bound. Dropping engaged hides a real cause of consent
loss, so the move shows how much one recorded cause matters here. A move describes the dropped
covariate only. An unrecorded cause of consent loss can act through other paths, and its move can be
larger.
Method evidence#
The fits on this page use one draw of a synthetic law. The registered studies in the table measure each method over repeated draws. Each study page gives its law, its nuisance models, and the result of each cell.
fit on this page |
registered study |
law of the study |
|---|---|---|
Step 6, LTMLE |
another law |
|
Step 7, g-computation |
another law |
No registered study covers three offer levels at three nodes.
A review probe is a script under reviews/notebook-review/probes/. The step readings quote its
results. A review probe is not a registered study.
The technical entry lists the evidence and the limits of the method.
What the analysis answers#
The question was which offer policy gives the largest share of full-price orders in week 4, and which policy beats which.
part of the answer |
what the page supports |
|---|---|
the estimate |
if the assumptions below hold, win-back raises the week-4 full-price share by 0.0754 over no offers, with a 95% interval of (0.0353, 0.1154). The population is active subscribers whose subscription term runs through week 4. The intervention is the declared code templates, with tracking consent kept |
supported on this draw |
win-back beats a discount every week. The paired interval (0.110, 0.217) excludes zero. A discount every week lowers the share, with an interval of (-0.1467, -0.0289) |
not resolved at this size |
win-back against free shipping, and win-back then ship against win-back. Both paired intervals contain zero. In the review probe they excluded zero on 8 and 26 of 100 draws. The page does not compare win-back then ship with free shipping, so it names no best policy among the three |
evidence behind the reading |
the exact offer probabilities, the support report, the synthetic deployment check, the benchmark of Step 11, and the review probes |
assumptions that stay unverified |
consent exchangeability, consistency, and no interference |
conclusions the page does not support |
a claim about applied data or another campaign, because the law is synthetic and no applied data access exists. A statement that the method is unbiased in general. A ranking of win-back and win-back then ship |
Where to go next#
The sequential pricing tutorial fits a continuous price at three monthly nodes in the same retailer program, with a learned density ratio. The longitudinal user guide shows how to read a long panel and how to report the sequential baselines. Sequential g-computation and IPW states the g-computation estimator of Step 7. The examples index lists every tutorial in both programs.