Example notebooks#

This page tells you how to write a tutorial under docs/examples/ as an executed notebook. Each step of a tutorial notebook shows its code, its output, and a plain reading of that output. The reader is a practitioner who is new to TMLE and new to cleverly.

Use point-treatment TMLE as the reference. It follows every rule on this page. The TWINS notebook analyzes public data, and it is not a program tutorial.

What a tutorial notebook consists of#

A tutorial has one stem, such as cross-fitting. The stem names both files you own.

file	what it holds
`docs/examples/<stem>.ipynb`	the notebook, with its stored outputs and the execution stamp
`tests/unit/tutorial_semantics/<module>.py`	the reviewed callback. `<module>` is the stem with each hyphen replaced by an underscore

The callback module defines check(namespace). The fast tier runs every code cell of the notebook, in order, in one namespace, and passes that namespace to check. The module can also define UNPRINTED_DECIMALS, a mapping from a decimal the prose writes to the reason no output prints it. The package docstring in tests/unit/tutorial_semantics/__init__.py states the convention.

The TWINS notebook downloads its data, so the offline fast tier does not execute it. Its module tests/unit/tutorial_semantics/twins_causal_inference.py defines no check. It defines UNPRINTED_DECIMALS and check_stored(), which read committed outputs only.

These fast tests read your notebook. Each one takes the notebook path as its test id, so the selection -k <stem> runs all of them.

test in `tests/unit/test_documentation_runtime.py`	what fails it
`test_every_notebook_has_an_internally_consistent_execution_artifact`	a code cell or an output that changed after the stamp, an error output, or non-contiguous execution counts
`test_every_notebook_records_its_repository_context`	a missing or malformed recorded half of the stamp
`test_every_published_figure_has_a_non_image_companion`	a figure with no printed values beside it
`test_tutorial_semantics_at_documented_size`	a cell that raises offline, or a claim in `check` that no longer holds
`test_every_narrated_decimal_matches_a_stored_output`	a decimal in the prose that no stored output prints at that precision. This test also reads the TWINS notebook
`test_every_static_only_notebook_keeps_its_pinned_outputs`	TWINS only: a stored output that `check_stored()` requires

The link test tests/unit/test_documentation_links.py also reads the markdown cells. Every relative link and every heading anchor in them must resolve.

The cell outline#

Every tutorial notebook follows this order. Use the cell ids in the table where the step applies. Each code cell is followed by a markdown cell that starts with **What this output tells you.**.

order	cell id	type	contents	the output it shows
1	`title`	markdown	the title as the only level-one heading, then the situation: the setting, the data, the treatment, the outcome, the time order, and which quantities are counterfactual. Add a compact causal graph where one applies, as the situation step describes. The learning table is optional. If you add one, list the decisions the problem teaches, not the features the page calls	none
2	`plan`	markdown	the question, the target population, the intervention, and the estimand, before any code. The identification assumptions, each marked as checkable from data or as scientific judgment. “Why this method”, with each method feature tied to a feature of the problem. A table of the terms the page uses most	none
3	`setup`	code	the imports and the installed `cleverly` version	the version
4	`data`	code	the data under the program’s column names, as the data step describes	the shape, the column names, the first rows, and the known truth of the synthetic law
5	`association`	code	an unadjusted comparison, where confounding is part of the lesson	the naive contrast beside the truth
6	`protocol`	code	the `StudyProtocol` for the question	`protocol.summary_lines()`, including the fingerprint
7	`identify`	code	the `CausalStudy`, the design, and `study.identify(...)`	`effect.summary(protocol="fingerprint")`: the formula, the nuisances, the assumptions, and the protocol fingerprint
8	one id per fit	code	each fit, with its method written out in full	`result.summary(protocol="fingerprint")` and the numbers the prose quotes
9	`failure-mode`	code	the failure mode the examples index names for this tutorial. Before the code, name the mechanism that causes the difference and the estimand of each method, as comparisons requires	the numbers that show the failure
10	`assessment`	code	`result.assess()` and the retained reports the page reads	the assessment summary and each report summary
11	`sensitivity`	code	the sensitivity analysis that is derived for the fitted estimator and runs on this fit. The sensitivity table names one for each kind of fit	the robustness value and the bounds, the E-value, the stress surface, the tipping point, or the benchmark refits
12	`method-evidence`	markdown	“Method evidence”: a table that links the registered study page of each method the page fits. The method-evidence step gives the rule	none
13	`interpret`	markdown	the answer to the question of row 2: what the estimate means, for which population and intervention, the reported interval, the evidence behind the reading, the assumptions that stay unverified, and the conclusions the analysis does not support	none
14	`where-next`	markdown	the next tutorial and the related reference entries	none

A tutorial can add steps between rows 8 and 11, such as a second population or a truncation curve. Give each added step a heading, a code cell, and its reading. Keep every scientific claim of the Markdown tutorial. A claim the output cannot show moves into the reading as a stated condition.

A tutorial notebook is an end-to-end example that works. Do not show a refusal as a lesson.

step	what to do
a step that would print a `CapabilityError`, a `DataError`, or a refusal message	change it to an analysis that `cleverly` supports, or remove it
a step that prints a capability list with unavailable rows	print only the rows that run
a statistical failure mode that runs and prints numbers, such as an interval that misses	keep it. Row 9 of the outline is for this step
a limit that the reading must name	write the requirement in one sentence, such as “Declare `q_bounds` for a cross-fitted continuous outcome”

The situation step#

Draw a causal graph only when it informs identification, data construction, the estimator, or the reading. Every node and every arrow must do one of these jobs. Under the graph, give a table with one row for each arrow or absent arrow and what it asserts. Name each confounder, mediator, competing event, censoring variable, and effect modifier. State which assumptions the graph encodes and which it leaves open. A text fence renders the graph without an extra build dependency.

Comparisons#

Keep a comparison only when it teaches a real distinction. Do not add an estimator only as a reference value.

for each comparison	what to write
the estimands	the estimand of each method. Compare numbers only for methods that target the same estimand, or say that the difference is the lesson
the assumptions	the assumptions that each method adds
the mechanism	the feature of the law or problem that causes the difference, named before the output
the evidence	the diagnostic, population quantity, or registered study that shows the mechanism
the reading	what the difference shows. One simulated sample does not show a general property

The data step#

Take the program’s column names from one place, so every page uses the same names.

generator	what to write
`make_nonlinear_bounded`	`frame, truth = navigation_data(n=..., seed=...)`, imported from `cleverly.datasets`. Do not rename its columns in the notebook
another generator	`frame.rename(columns=...)` with that generator’s mapping. Each generator gives its `W` columns other meanings, so no shared mapping fits them

Print the column names in the data step, such as print(list(frame.columns)). The printed names show the reader which columns the page uses. Name the generator in the reading.

The protocol step#

StudyProtocol in src/cleverly/protocol.py records the scientific design. Every result fitted from a study that carries it stores its fingerprint. Fill each field for the tutorial’s question. Then check each field and each unit name against the structural equations of the generator. For example, do not call an outcome “standardized” when the generator draws it on its own scale.

field	what to write
`target_population`	the population to which the causal question applies
`eligibility`	the inclusion and exclusion criteria, one entry each
`time_zero`	the event that aligns eligibility, treatment assignment, and follow-up
`treatment_strategies`	the strategies under comparison, one entry each
`treatment_versions`	the version of each strategy, in the same order
`outcome`	the outcome definition and how it is measured
`horizon`	the follow-up horizon
`intercurrent_event_handling`	the rule for each event after time zero that changes the outcome’s meaning
`interference_unit`	the unit within which one assignment can affect another outcome
`assumption_rationale`	why the design supports each identification assumption

The protocol does not store the contrast or the analysis. The typed estimand owns the contrast, and the method owns the analysis configuration. The shared study design gives the program values.

Do not copy the program values into a notebook. Start from the helper in cleverly.datasets that matches the design.

design	helper
one navigation decision at the discharge-home order	`navigation_protocol()`
navigation decisions at discharge and later, with time zero at discharge	`longitudinal_navigation_protocol()`

Pass the helper’s result to dataclasses.replace, and give only the fields your question changes. Reuse an unchanged entry of a sequence field by index, such as *program.assumption_rationale[1:]. To add a strategy, such as a dynamic rule, append it and its version: treatment_strategies=(*program.treatment_strategies, "<the rule>"). Write a new StudyProtocol only when neither helper fits the design.

Print each protocol record once on the page. The table gives where each notebook prints it.

record	where the notebook prints it	later summaries of its study
the record of the protocol step	the protocol step, with `"\n".join(protocol.summary_lines())`	pass `protocol="fingerprint"`, which prints only the fingerprint line
a record that a later step declares	the first summary of its study, with the default `protocol="full"`. A step can instead print the fingerprint and the fields that it changed	pass `protocol="fingerprint"`
no record	nothing to print. The summaries state `causal study protocol: absent`	keep the default

Every protocol that a notebook derives with dataclasses.replace needs two records. The reading names the changed fields. The callback asserts the exact set with changed_fields(protocol, base), where base is the protocol that replace received.

Rules for code cells#

rule	reason
Give every code cell visible output. Print text rather than rely on a bare last expression	the reading needs an output to interpret, and `--check` compares text
Print each number the prose quotes, rounded the way the prose writes it	the narration test compares prose decimals with stored outputs
Pass explicit, cheap learners to every fit. Prefer linear and logistic models where the lesson allows	the default learner library costs 30 to 120 seconds per fit
Do not pass a bare `HistGradientBoostingClassifier` as a treatment learner below 10,000 rows. Use a regularized or early-stopped booster, or a flexible parametric model, and check its calibration	scikit-learn stops the booster early only above 10,000 rows. Below that size, its default settings overfit the treatment model. The fitted propensities are then too extreme, so the standard error is too large and the default $\nu^2$ estimate can be negative
Print a score check as pass or fail and the rounded ratio to its threshold. Never print a raw residual below 1e-8	a residual below 1e-8 is machine noise. It changes between machines, and `--check` then fails
Set every effective seed: the generator seed, `Runtime(random_state=...)`, and each learner’s active `random_state`	a rerun must reproduce the stored outputs
Set `n_jobs=1` in `Runtime` and where a learner still uses that parameter. Do not pass a deprecated parameter that the installed learner ignores	several notebooks execute on one machine at once, and deprecated settings add warning noise without limiting work
Fit every learner through `cleverly`, and do not call `set_thread_limit`. Wrap any other fit in `with thread_limit():` from `cleverly.learners`	`HistGradientBoosting*` has no `n_jobs`, and its OpenMP pool uses every core. `cleverly` fits each learner under `thread_limit()`, which holds OpenMP and BLAS to one thread by default
Keep the whole notebook under 60 seconds with `scripts/execute_notebook.py`	the fast tier runs every tutorial. The stored cell timestamps of the reference notebook span about 11 seconds
Give each code cell a short, stable, kebab-case id. Never rename an id that a callback reads	`stored_output(path, cell_id)` finds a cell by its id
Pair every figure with printed values	an image cannot enter the `--check` comparison
Use no network, and write files only inside a `TemporaryDirectory`	the offline gate refuses a connection, and the kernel runs from the repository root
Write plain Python. Do not use IPython magics or shell escapes. `display(...)` is allowed	the fast tier runs the cells without a kernel
Run `ruff format` and `ruff check` on the notebook before you execute it	the formatter rewrites code cells, and a rewritten cell breaks the stamp

Rules for markdown cells#

The prose rules in CLAUDE.md apply to every markdown cell. Write one idea per sentence, and use a table where the content is parallel. python -m tests.prose --path <notebook> reports the findings.

Start each reading with **What this output tells you.**. Then name the part of the output you read, give its value, and say what it means for the program. Quote a number only when a stored output prints it.

Each reading also follows these rules.

rule	reason
Put a status word, a label, or a column name in backticks only when a stored output prints it. The assessment summary prints `Returned results`, not `completed`	the reader searches the output for the quoted token. No test catches this, because each program protocol prints “regardless of completed contacts”
Do not write that one seeded draw shows or establishes a property of the method. Cite a registered study for the property	one draw is one sample. The callback pins it as a relation, and a relation certifies no method
Before you call an interval that misses the true value sampling variation, run a seed sweep of the shown configuration with at least 60 draws. Use that explanation only when the sweep covers near the nominal level. Otherwise name the measured bias or under-coverage and the cause that a probe shows	a 95% interval misses on about one draw in 20 only when the configuration covers at the nominal level. A configuration with a bias can miss on most draws
A sentence that holds on fewer than 90% of the swept draws says “on this draw”. Alternatively, quote the printed numbers without a multiplier such as “about three times”	a relation of one seeded draw is a property of that draw. A multiplier reads as a property of the method
Name the source of each repeated-draw number that a reading quotes: a committed probe script or a registered study. List the number in the callback’s `UNPRINTED_DECIMALS` with that source. In `check`, assert the number against the committed file that holds it, such as the probe log or the study summary	no stored output prints a sweep number, so the narration test needs the entry. The narration test checks only that a reason exists. The assertion keeps the number equal to its source
A support or calibration reading names the fitted model that it comes from. It also says what that model cannot see	a support report of a wrong treatment model describes that model, not the population. An in-sample calibration slope near 1 does not show that a model is correct
Compare two estimates from one fit with `result.contrast(...)`, and read its interval. Do not compare their difference with the larger standard error	the two estimates share rows. `contrast` uses the joint influence curve, so it includes their correlation
Read a sensitivity output as robustness only when the technical reference derives it for the fitted estimator. Run the analysis that the sensitivity table names for the fit	an underived bound can be too narrow
Say that a quantity does not exist only when a source shows it	“not implemented” and “not defined” are two claims

Introduce each TMLE concept the first time the page uses it. Write one plain sentence, then link to the reference. The table gives a starting sentence and a link for the common concepts.

concept	one plain sentence	link
estimand	the number the question asks for, written before any model is chosen	estimands
nuisance	a model the estimate needs but the question does not ask about, such as Q or g	point-treatment TMLE
targeting	a small update to the outcome model, weighted by the treatment model, that removes first-order bias	targeting and bounds
influence curve	how much each row moves the estimate. Its variance gives the standard error	inference
cross-fitting	each row’s nuisance prediction comes from models fit without that row	CV-TMLE
positivity	every kind of patient has some chance of each arm	diagnostics
double robustness	the point estimate stays consistent when either nuisance model is consistent	point-treatment TMLE
sensitivity analysis	how strong an unmeasured confounder would need to be to change the conclusion	sensitivity analysis

Link to a sibling tutorial by its notebook file name, such as cross-fitting.ipynb. A heading anchor in a notebook includes its step number, such as #step-9-reuse-the-same-outer-split.

Sensitivity analysis for each kind of fit#

Run the sensitivity analysis that the table gives for the fit of the page. Each analysis runs on that fit and returns numbers. The sensitivity section of the technical reference describes each one and says which of them is a bound.

kind of fit	the sensitivity analysis to run	how to read it
point-treatment TMLE or CV-TMLE, on a law whose true propensity stays away from 0 and 1	`robustness_value()`, `benchmark()`, and `omitted_confounding()`, with the default $\nu^2$ estimator	a bound. $\nu^2$ is the second moment of the Riesz representer, and it is finite only when $1/g$ and $1/(1 - g)$ both have finite means
C-TMLE	`robustness_value()` and `omitted_confounding()` on the ordinary TMLE fit of the declared adjustment set only. A collaborative fit other than `strategy="derivative"` refuses the bound. Do not run `simulated_confounding()`	a bound for the ordinary fit. It does not cover the collaborative estimate
MSM projection	`robustness_value(estimand="ate[medium vs low]")` and its sibling on the multi-arm `ATE(reference="low")` fit, whose contrasts share the influence curves of the saturated MSM coefficients	a bound on each arm contrast, not on the MSM coefficient. The `msm` axis itself is refused
a fit with missing outcomes	`missingness()` and `tipping_gamma()`	the tilt of the unobserved outcomes at which the estimate reaches its null
longitudinal TMLE, with or without a time-to-event outcome	refit without one recorded covariate at a time, and print each move in standard errors	a benchmark, not a bound. Each move is signed and specific to the dropped covariate
a regime, shift, or incremental intervention	state that no omitted-variable bound exists for these contrasts (X9). If the page runs `simulated_confounding()`, a fit that declares `q_bounds` admits only a treatment-only grid, such as `ConfounderStrengthGrid(treatment=(0.0, 0.1), outcome=(0.0,))`. One flip strength is enough	a treatment-only cell moves the estimate by misclassification of the treatment, not by confounding. It is not a bound. Say so in the reading, or omit the step

The method-evidence step#

The method-evidence cell links the evidence for each method on the page. Do not copy study results, cell names, or citations into the notebook. The study page holds them, and a copy goes stale when the study is regenerated.

part of the cell	what to write
the table	one row for each fit: the step, a link to the registered study page in `docs/technical-reference/method-evidence/`, and whether the study uses the law of the page or another law
the gap	one sentence for each part of the page that no registered study covers, such as a learner, a fold count, or a fitted mechanism
the review probes	one sentence that a review probe is a script under `reviews/notebook-review/probes/` and is not a registered study. Quote a probe result in the step reading that uses it
the technical entry	a link to the “Validation issues special to this method” section of the method’s technical entry

Write the callback#

Start from the tutorial’s existing module in tests/unit/tutorial_semantics/. Keep each assertion that still describes the page. Rename a namespace key only when the notebook renames the variable.

Add an assertion for each seeded relation the new readings narrate. An example is “the interval contains the true value”. A relation is a claim about this draw, so it certifies no method. Use stored_output(NOTEBOOK, "<cell id>") to check a stored output that must stay present.

Use the shared helpers in tests/unit/tutorial_semantics/__init__.py. Do not define a local copy.

helper	what it checks
`covers(estimate, value)`	the estimate’s interval, or a `(low, high)` pair, contains the value
`assert_protocol_recorded(NOTEBOOK, "<cell id>", protocol, *results)`	the cell prints the protocol fingerprint, and each result carries it
`changed_fields(protocol, base)`	the set of fields in which `protocol` differs from the protocol it was derived from. Assert the exact set the reading names

Do not loosen a tolerance to make a relation pass. A relation that moves under a supported change is a sampling claim, and it belongs in a registered study.

Convert a tutorial#

Do these steps in order. Run each command from the repository root.

Read the Markdown tutorial, its callback module, and the reference notebook.
Write docs/examples/<stem>.ipynb with the cell outline above. Leave the readings short for now.

Format and lint the code cells.

ruff format docs/examples/<stem>.ipynb
ruff check docs/examples/<stem>.ipynb

Execute the notebook and write its stamp. Send the log to a file, and check the exit status. Do not pipe the run into tail or head, because the pipe hides the exit status.
```
python scripts/execute_notebook.py docs/examples/<stem>.ipynb > execute.log 2>&1
echo "exit status $?"
```
Read every stored output. Write each reading from those outputs.
Delete the Markdown source with git rm docs/examples/<stem>.md.
Update the callback module, as the previous section describes.

Confirm that a fresh run reproduces the stored outputs. The command writes nothing.

python scripts/execute_notebook.py docs/examples/<stem>.ipynb --check > check.log 2>&1
echo "exit status $?"

Confirm that the notebook has line-feed endings. The command prints False.

python -c "import sys; print(bytes([13, 10]) in open(sys.argv[1], 'rb').read())" docs/examples/<stem>.ipynb

Run the targeted checks, and read every finding.

ruff format --check docs/examples/<stem>.ipynb
python -m tests.prose --path docs/examples/<stem>.ipynb
pytest -q -p no:cacheprovider tests/unit/test_documentation_runtime.py -k <module>
pytest -q -p no:cacheprovider tests/unit/test_documentation_links.py -k <module>

Use the callback module name for <module>. It replaces hyphens with underscores, so the selection is valid in a pytest expression.

Report to the orchestrator. The report is in the last section.

A markdown edit does not change the stamp, so it needs no new execution. A code edit does. After a code edit, go back to step 3. A new execution can move a number, and the narration test then names each reading to correct.

Remove execute.log and check.log before you hand off.

Share one machine#

Several authors execute notebooks at the same time on one machine. Follow these limits.

limit	reason
Run `pytest` without `-n`, and never with `-n auto`	a worker pool per author makes the authors contend for the same cores
Run only the targeted checks above. Do not run the full suite or `nox -s docs`	the orchestrator runs both once, after the conversions
Execute one notebook at a time	each execution starts its own kernel
Set `n_jobs=1` everywhere	a parallel learner competes with the other authors

Do not edit shared files#

Edit only your notebook, your callback module, and the deletion of your Markdown source. Report any other change the conversion needs. The orchestrator applies shared edits one at a time, so parallel authors do not collide on one file.

shared file	what to report instead
`tests/prose-report.md`	the finding id, the sentence, and the `accepted: <reason>` you propose
`docs/examples/index.md`, another tutorial, or any page that links to your `.md` file	each link to rewrite. `git grep -n "<stem>.md"` lists them
`tests/unit/test_documentation_runtime.py`, `tests/unit/tutorial_semantics/__init__.py`, `tests/documents.py`, `tests/notebooks.py`, `tests/conftest.py`	the change and the failure it repairs
`scripts/execute_notebook.py`, `docs/conf.py`, `pyproject.toml`, `uv.lock`	the change and the failure it repairs

The report also gives the notebook’s execution time, the exit status of each command in step 10, and every claim of the Markdown tutorial that the notebook changed or removed.

kind of fit	the sensitivity analysis to run	how to read it
point-treatment TMLE or CV-TMLE, on a law whose true propensity stays away from 0 and 1	`robustness_value()`, `benchmark()`, and `omitted_confounding()`, with the default \(\nu^2\) estimator	a bound. \(\nu^2\) is the second moment of the Riesz representer, and it is finite only when \(1/g\) and \(1/(1 - g)\) both have finite means
C-TMLE	`robustness_value()` and `omitted_confounding()` on the ordinary TMLE fit of the declared adjustment set only. A collaborative fit other than `strategy="derivative"` refuses the bound. Do not run `simulated_confounding()`	a bound for the ordinary fit. It does not cover the collaborative estimate
MSM projection	`robustness_value(estimand="ate[medium vs low]")` and its sibling on the multi-arm `ATE(reference="low")` fit, whose contrasts share the influence curves of the saturated MSM coefficients	a bound on each arm contrast, not on the MSM coefficient. The `msm` axis itself is refused
a fit with missing outcomes	`missingness()` and `tipping_gamma()`	the tilt of the unobserved outcomes at which the estimate reaches its null
longitudinal TMLE, with or without a time-to-event outcome	refit without one recorded covariate at a time, and print each move in standard errors	a benchmark, not a bound. Each move is signed and specific to the dropped covariate
a regime, shift, or incremental intervention	state that no omitted-variable bound exists for these contrasts (X9). If the page runs `simulated_confounding()`, a fit that declares `q_bounds` admits only a treatment-only grid, such as `ConfounderStrengthGrid(treatment=(0.0, 0.1), outcome=(0.0,))`. One flip strength is enough	a treatment-only cell moves the estimate by misclassification of the treatment, not by confounding. It is not a bound. Say so in the reading, or omit the step