# Example notebooks

This page tells you how to write a tutorial under `docs/examples/` as an executed notebook. Each
step of a tutorial notebook shows its code, its output, and a plain reading of that output. The
reader is a practitioner who is new to TMLE and new to `cleverly`.

Use [point-treatment TMLE](../examples/point-treatment-tmle.ipynb) as the reference. It follows
every rule on this page. The [TWINS notebook](../examples/twins-causal-inference.ipynb) analyzes
public data, and it is not a program tutorial.

## What a tutorial notebook consists of

A tutorial has one stem, such as `cross-fitting`. The stem names both files you own.

| file | what it holds |
| --- | --- |
| `docs/examples/<stem>.ipynb` | the notebook, with its stored outputs and the execution stamp |
| `tests/unit/tutorial_semantics/<module>.py` | the reviewed callback. `<module>` is the stem with each hyphen replaced by an underscore |

The callback module defines `check(namespace)`. The fast tier runs every code cell of the notebook,
in order, in one namespace, and passes that namespace to `check`. The module can also define
`UNPRINTED_DECIMALS`, a mapping from a decimal the prose writes to the reason no output prints it.
The package docstring in `tests/unit/tutorial_semantics/__init__.py` states the convention.

The TWINS notebook downloads its data, so the offline fast tier does not execute it. Its module
`tests/unit/tutorial_semantics/twins_causal_inference.py` defines no `check`. It defines
`UNPRINTED_DECIMALS` and `check_stored()`, which read committed outputs only.

These fast tests read your notebook. Each one takes the notebook path as its test id, so the
selection `-k <stem>` runs all of them.

| test in `tests/unit/test_documentation_runtime.py` | what fails it |
| --- | --- |
| `test_every_notebook_has_an_internally_consistent_execution_artifact` | a code cell or an output that changed after the stamp, an error output, or non-contiguous execution counts |
| `test_every_notebook_records_its_repository_context` | a missing or malformed recorded half of the stamp |
| `test_every_published_figure_has_a_non_image_companion` | a figure with no printed values beside it |
| `test_tutorial_semantics_at_documented_size` | a cell that raises offline, or a claim in `check` that no longer holds |
| `test_every_narrated_decimal_matches_a_stored_output` | a decimal in the prose that no stored output prints at that precision. This test also reads the TWINS notebook |
| `test_every_static_only_notebook_keeps_its_pinned_outputs` | TWINS only: a stored output that `check_stored()` requires |

The link test `tests/unit/test_documentation_links.py` also reads the markdown cells. Every relative
link and every heading anchor in them must resolve.

## The cell outline

Every tutorial notebook follows this order. Use the cell ids in the table where the step applies.
Each code cell is followed by a markdown cell that starts with `**What this output tells you.**`.

| order | cell id | type | contents | the output it shows |
| --- | --- | --- | --- | --- |
| 1 | `title` | markdown | the title as the only level-one heading, then the situation: the setting, the data, the treatment, the outcome, the time order, and which quantities are counterfactual. Add a compact causal graph where one applies, as [the situation step](#the-situation-step) describes. The learning table is optional. If you add one, list the decisions the problem teaches, not the features the page calls | none |
| 2 | `plan` | markdown | the question, the target population, the intervention, and the estimand, before any code. The identification assumptions, each marked as checkable from data or as scientific judgment. "Why this method", with each method feature tied to a feature of the problem. A table of the terms the page uses most | none |
| 3 | `setup` | code | the imports and the installed `cleverly` version | the version |
| 4 | `data` | code | the data under the program's column names, as [the data step](#the-data-step) describes | the shape, the column names, the first rows, and the known truth of the synthetic law |
| 5 | `association` | code | an unadjusted comparison, where confounding is part of the lesson | the naive contrast beside the truth |
| 6 | `protocol` | code | the `StudyProtocol` for the question | `protocol.summary_lines()`, including the fingerprint |
| 7 | `identify` | code | the `CausalStudy`, the design, and `study.identify(...)` | `effect.summary(protocol="fingerprint")`: the formula, the nuisances, the assumptions, and the protocol fingerprint |
| 8 | one id per fit | code | each fit, with its method written out in full | `result.summary(protocol="fingerprint")` and the numbers the prose quotes |
| 9 | `failure-mode` | code | the failure mode the [examples index](../examples/index.md#the-program) names for this tutorial. Before the code, name the mechanism that causes the difference and the estimand of each method, as [comparisons](#comparisons) requires | the numbers that show the failure |
| 10 | `assessment` | code | `result.assess()` and the retained reports the page reads | the assessment summary and each report summary |
| 11 | `sensitivity` | code | the sensitivity analysis that is derived for the fitted estimator and runs on this fit. [The sensitivity table](#sensitivity-analysis-for-each-kind-of-fit) names one for each kind of fit | the robustness value and the bounds, the E-value, the stress surface, the tipping point, or the benchmark refits |
| 12 | `method-evidence` | markdown | "Method evidence": a table that links the registered study page of each method the page fits. [The method-evidence step](#the-method-evidence-step) gives the rule | none |
| 13 | `interpret` | markdown | the answer to the question of row 2: what the estimate means, for which population and intervention, the reported interval, the evidence behind the reading, the assumptions that stay unverified, and the conclusions the analysis does not support | none |
| 14 | `where-next` | markdown | the next tutorial and the related reference entries | none |

A tutorial can add steps between rows 8 and 11, such as a second population or a truncation curve.
Give each added step a heading, a code cell, and its reading. Keep every scientific claim of the
Markdown tutorial. A claim the output cannot show moves into the reading as a stated condition.

A tutorial notebook is an end-to-end example that works. Do not show a refusal as a lesson.

| step | what to do |
| --- | --- |
| a step that would print a `CapabilityError`, a `DataError`, or a refusal message | change it to an analysis that `cleverly` supports, or remove it |
| a step that prints a capability list with unavailable rows | print only the rows that run |
| a statistical failure mode that runs and prints numbers, such as an interval that misses | keep it. Row 9 of the outline is for this step |
| a limit that the reading must name | write the requirement in one sentence, such as "Declare `q_bounds` for a cross-fitted continuous outcome" |

### The situation step

Draw a causal graph only when it informs identification, data construction, the estimator, or the
reading. Every node and every arrow must do one of these jobs. Under the graph, give a table with
one row for each arrow or absent arrow and what it asserts. Name each confounder, mediator,
competing event, censoring variable, and effect modifier. State which assumptions the graph encodes
and which it leaves open. A `text` fence renders the graph without an extra build dependency.

### Comparisons

Keep a comparison only when it teaches a real distinction. Do not add an estimator only as a
reference value.

| for each comparison | what to write |
| --- | --- |
| the estimands | the estimand of each method. Compare numbers only for methods that target the same estimand, or say that the difference is the lesson |
| the assumptions | the assumptions that each method adds |
| the mechanism | the feature of the law or problem that causes the difference, named before the output |
| the evidence | the diagnostic, population quantity, or registered study that shows the mechanism |
| the reading | what the difference shows. One simulated sample does not show a general property |

### The data step

Take the program's column names from one place, so every page uses the same names.

| generator | what to write |
| --- | --- |
| `make_nonlinear_bounded` | `frame, truth = navigation_data(n=..., seed=...)`, imported from `cleverly.datasets`. Do not rename its columns in the notebook |
| another generator | `frame.rename(columns=...)` with that generator's mapping. Each generator gives its `W` columns other meanings, so no shared mapping fits them |

Print the column names in the data step, such as `print(list(frame.columns))`. The printed names
show the reader which columns the page uses. Name the generator in the reading.

### The protocol step

`StudyProtocol` in `src/cleverly/protocol.py` records the scientific design. Every result fitted
from a study that carries it stores its fingerprint. Fill each field for the tutorial's question.
Then check each field and each unit name against the structural equations of the generator. For
example, do not call an outcome "standardized" when the generator draws it on its own scale.

| field | what to write |
| --- | --- |
| `target_population` | the population to which the causal question applies |
| `eligibility` | the inclusion and exclusion criteria, one entry each |
| `time_zero` | the event that aligns eligibility, treatment assignment, and follow-up |
| `treatment_strategies` | the strategies under comparison, one entry each |
| `treatment_versions` | the version of each strategy, in the same order |
| `outcome` | the outcome definition and how it is measured |
| `horizon` | the follow-up horizon |
| `intercurrent_event_handling` | the rule for each event after time zero that changes the outcome's meaning |
| `interference_unit` | the unit within which one assignment can affect another outcome |
| `assumption_rationale` | why the design supports each identification assumption |

The protocol does not store the contrast or the analysis. The typed estimand owns the contrast, and
the method owns the analysis configuration. The
[shared study design](../examples/index.md#the-shared-study-design) gives the program values.

Do not copy the program values into a notebook. Start from the helper in `cleverly.datasets` that
matches the design.

| design | helper |
| --- | --- |
| one navigation decision at the discharge-home order | `navigation_protocol()` |
| navigation decisions at discharge and later, with time zero at discharge | `longitudinal_navigation_protocol()` |

Pass the helper's result to `dataclasses.replace`, and give only the fields your question changes.
Reuse an unchanged entry of a sequence field by index, such as `*program.assumption_rationale[1:]`.
To add a strategy, such as a dynamic rule, append it and its version:
`treatment_strategies=(*program.treatment_strategies, "<the rule>")`. Write a new `StudyProtocol`
only when neither helper fits the design.

Print each protocol record once on the page. The table gives where each notebook prints it.

| record | where the notebook prints it | later summaries of its study |
| --- | --- | --- |
| the record of the protocol step | the protocol step, with `"\n".join(protocol.summary_lines())` | pass `protocol="fingerprint"`, which prints only the fingerprint line |
| a record that a later step declares | the first summary of its study, with the default `protocol="full"`. A step can instead print the fingerprint and the fields that it changed | pass `protocol="fingerprint"` |
| no record | nothing to print. The summaries state `causal study protocol: absent` | keep the default |

Every protocol that a notebook derives with `dataclasses.replace` needs two records. The reading
names the changed fields. The callback asserts the exact set with `changed_fields(protocol, base)`,
where `base` is the protocol that `replace` received.

## Rules for code cells

| rule | reason |
| --- | --- |
| Give every code cell visible output. Print text rather than rely on a bare last expression | the reading needs an output to interpret, and `--check` compares text |
| Print each number the prose quotes, rounded the way the prose writes it | the narration test compares prose decimals with stored outputs |
| Pass explicit, cheap learners to every fit. Prefer linear and logistic models where the lesson allows | the default learner library costs 30 to 120 seconds per fit |
| Do not pass a bare `HistGradientBoostingClassifier` as a treatment learner below 10,000 rows. Use a regularized or early-stopped booster, or a flexible parametric model, and check its calibration | scikit-learn stops the booster early only above 10,000 rows. Below that size, its default settings overfit the treatment model. The fitted propensities are then too extreme, so the standard error is too large and the default $\nu^2$ estimate can be negative |
| Print a score check as pass or fail and the rounded ratio to its threshold. Never print a raw residual below 1e-8 | a residual below 1e-8 is machine noise. It changes between machines, and `--check` then fails |
| Set every effective seed: the generator seed, `Runtime(random_state=...)`, and each learner's active `random_state` | a rerun must reproduce the stored outputs |
| Set `n_jobs=1` in `Runtime` and where a learner still uses that parameter. Do not pass a deprecated parameter that the installed learner ignores | several notebooks execute on one machine at once, and deprecated settings add warning noise without limiting work |
| Fit every learner through `cleverly`, and do not call `set_thread_limit`. Wrap any other fit in `with thread_limit():` from `cleverly.learners` | `HistGradientBoosting*` has no `n_jobs`, and its OpenMP pool uses every core. `cleverly` fits each learner under `thread_limit()`, which holds OpenMP and BLAS to one thread by default |
| Keep the whole notebook under 60 seconds with `scripts/execute_notebook.py` | the fast tier runs every tutorial. The stored cell timestamps of the reference notebook span about 11 seconds |
| Give each code cell a short, stable, kebab-case id. Never rename an id that a callback reads | `stored_output(path, cell_id)` finds a cell by its id |
| Pair every figure with printed values | an image cannot enter the `--check` comparison |
| Use no network, and write files only inside a `TemporaryDirectory` | the offline gate refuses a connection, and the kernel runs from the repository root |
| Write plain Python. Do not use IPython magics or shell escapes. `display(...)` is allowed | the fast tier runs the cells without a kernel |
| Run `ruff format` and `ruff check` on the notebook before you execute it | the formatter rewrites code cells, and a rewritten cell breaks the stamp |

## Rules for markdown cells

The prose rules in `CLAUDE.md` apply to every markdown cell. Write one idea per sentence, and use a
table where the content is parallel. `python -m tests.prose --path <notebook>` reports the findings.

Start each reading with `**What this output tells you.**`. Then name the part of the output you
read, give its value, and say what it means for the program. Quote a number only when a stored
output prints it.

Each reading also follows these rules.

| rule | reason |
| --- | --- |
| Put a status word, a label, or a column name in backticks only when a stored output prints it. The assessment summary prints `Returned results`, not `completed` | the reader searches the output for the quoted token. No test catches this, because each program protocol prints "regardless of completed contacts" |
| Do not write that one seeded draw shows or establishes a property of the method. Cite a registered study for the property | one draw is one sample. The callback pins it as a relation, and a relation certifies no method |
| Before you call an interval that misses the true value sampling variation, run a seed sweep of the shown configuration with at least 60 draws. Use that explanation only when the sweep covers near the nominal level. Otherwise name the measured bias or under-coverage and the cause that a probe shows | a 95% interval misses on about one draw in 20 only when the configuration covers at the nominal level. A configuration with a bias can miss on most draws |
| A sentence that holds on fewer than 90% of the swept draws says "on this draw". Alternatively, quote the printed numbers without a multiplier such as "about three times" | a relation of one seeded draw is a property of that draw. A multiplier reads as a property of the method |
| Name the source of each repeated-draw number that a reading quotes: a committed probe script or a registered study. List the number in the callback's `UNPRINTED_DECIMALS` with that source. In `check`, assert the number against the committed file that holds it, such as the probe log or the study summary | no stored output prints a sweep number, so the narration test needs the entry. The narration test checks only that a reason exists. The assertion keeps the number equal to its source |
| A support or calibration reading names the fitted model that it comes from. It also says what that model cannot see | a support report of a wrong treatment model describes that model, not the population. An in-sample calibration slope near 1 does not show that a model is correct |
| Compare two estimates from one fit with `result.contrast(...)`, and read its interval. Do not compare their difference with the larger standard error | the two estimates share rows. `contrast` uses the joint influence curve, so it includes their correlation |
| Read a sensitivity output as robustness only when the technical reference derives it for the fitted estimator. Run the analysis that [the sensitivity table](#sensitivity-analysis-for-each-kind-of-fit) names for the fit | an underived bound can be too narrow |
| Say that a quantity does not exist only when a source shows it | "not implemented" and "not defined" are two claims |

Introduce each TMLE concept the first time the page uses it. Write one plain sentence, then link
to the reference. The table gives a starting sentence and a link for the common concepts.

| concept | one plain sentence | link |
| --- | --- | --- |
| estimand | the number the question asks for, written before any model is chosen | [estimands](../user-guide/estimands.md) |
| nuisance | a model the estimate needs but the question does not ask about, such as Q or g | [point-treatment TMLE](../technical-reference/point-treatment-tmle.md) |
| targeting | a small update to the outcome model, weighted by the treatment model, that removes first-order bias | [targeting and bounds](../user-guide/methods-learners.md#targeting-and-bounds) |
| influence curve | how much each row moves the estimate. Its variance gives the standard error | [inference](../technical-reference/inference.md) |
| cross-fitting | each row's nuisance prediction comes from models fit without that row | [CV-TMLE](../technical-reference/cv-tmle.md) |
| positivity | every kind of patient has some chance of each arm | [diagnostics](../user-guide/results-assessment.md#diagnostics) |
| double robustness | the point estimate stays consistent when either nuisance model is consistent | [point-treatment TMLE](../technical-reference/point-treatment-tmle.md) |
| sensitivity analysis | how strong an unmeasured confounder would need to be to change the conclusion | [sensitivity analysis](../user-guide/results-assessment.md#sensitivity-analysis) |

Link to a sibling tutorial by its notebook file name, such as `cross-fitting.ipynb`. A heading
anchor in a notebook includes its step number, such as `#step-9-reuse-the-same-outer-split`.

### Sensitivity analysis for each kind of fit

Run the sensitivity analysis that the table gives for the fit of the page. Each analysis runs on
that fit and returns numbers. The
[sensitivity section](../technical-reference/validation-methods.md#sensitivity-to-untestable-assumptions)
of the technical reference describes each one and says which of them is a bound.

| kind of fit | the sensitivity analysis to run | how to read it |
| --- | --- | --- |
| point-treatment TMLE or CV-TMLE, on a law whose true propensity stays away from 0 and 1 | `robustness_value()`, `benchmark()`, and `omitted_confounding()`, with the default $\nu^2$ estimator | a bound. $\nu^2$ is the second moment of the Riesz representer, and it is finite only when $1/g$ and $1/(1 - g)$ both have finite means |
| C-TMLE | `robustness_value()` and `omitted_confounding()` on the ordinary TMLE fit of the declared adjustment set only. A collaborative fit other than `strategy="derivative"` refuses the bound. Do not run `simulated_confounding()` | a bound for the ordinary fit. It does not cover the collaborative estimate |
| MSM projection | `robustness_value(estimand="ate[medium vs low]")` and its sibling on the multi-arm `ATE(reference="low")` fit, whose contrasts share the influence curves of the saturated MSM coefficients | a bound on each arm contrast, not on the MSM coefficient. The `msm` axis itself is refused |
| a fit with missing outcomes | `missingness()` and `tipping_gamma()` | the tilt of the unobserved outcomes at which the estimate reaches its null |
| longitudinal TMLE, with or without a time-to-event outcome | refit without one recorded covariate at a time, and print each move in standard errors | a benchmark, not a bound. Each move is signed and specific to the dropped covariate |
| a regime, shift, or incremental intervention | state that no omitted-variable bound exists for these contrasts ([X9](../roadmap.md#x9-omitted-variable-bounds-on-the-other-linear-functionals)). If the page runs `simulated_confounding()`, a fit that declares `q_bounds` admits only a treatment-only grid, such as `ConfounderStrengthGrid(treatment=(0.0, 0.1), outcome=(0.0,))`. One flip strength is enough | a treatment-only cell moves the estimate by misclassification of the treatment, not by confounding. It is not a bound. Say so in the reading, or omit the step |

### The method-evidence step

The `method-evidence` cell links the evidence for each method on the page. Do not copy study
results, cell names, or citations into the notebook. The study page holds them, and a copy goes
stale when the study is regenerated.

| part of the cell | what to write |
| --- | --- |
| the table | one row for each fit: the step, a link to the registered study page in `docs/technical-reference/method-evidence/`, and whether the study uses the law of the page or another law |
| the gap | one sentence for each part of the page that no registered study covers, such as a learner, a fold count, or a fitted mechanism |
| the review probes | one sentence that a review probe is a script under `reviews/notebook-review/probes/` and is not a registered study. Quote a probe result in the step reading that uses it |
| the technical entry | a link to the "Validation issues special to this method" section of the method's technical entry |

## Write the callback

Start from the tutorial's existing module in `tests/unit/tutorial_semantics/`. Keep each assertion
that still describes the page. Rename a namespace key only when the notebook renames the variable.

Add an assertion for each seeded relation the new readings narrate. An example is "the interval
contains the true value". A relation is a claim about this draw, so it certifies no method. Use
`stored_output(NOTEBOOK, "<cell id>")` to check a stored output that must stay present.

Use the shared helpers in `tests/unit/tutorial_semantics/__init__.py`. Do not define a local copy.

| helper | what it checks |
| --- | --- |
| `covers(estimate, value)` | the estimate's interval, or a `(low, high)` pair, contains the value |
| `assert_protocol_recorded(NOTEBOOK, "<cell id>", protocol, *results)` | the cell prints the protocol fingerprint, and each result carries it |
| `changed_fields(protocol, base)` | the set of fields in which `protocol` differs from the protocol it was derived from. Assert the exact set the reading names |

Do not loosen a tolerance to make a relation pass. A relation that moves under a supported change is
a sampling claim, and it belongs in a registered study.

## Convert a tutorial

Do these steps in order. Run each command from the repository root.

1. Read the Markdown tutorial, its callback module, and the reference notebook.
2. Write `docs/examples/<stem>.ipynb` with the cell outline above. Leave the readings short for now.
3. Format and lint the code cells.

   ```bash
   ruff format docs/examples/<stem>.ipynb
   ruff check docs/examples/<stem>.ipynb
   ```

4. Execute the notebook and write its stamp. Send the log to a file, and check the exit status.
   Do not pipe the run into `tail` or `head`, because the pipe hides the exit status.

   ```bash
   python scripts/execute_notebook.py docs/examples/<stem>.ipynb > execute.log 2>&1
   echo "exit status $?"
   ```

5. Read every stored output. Write each reading from those outputs.
6. Delete the Markdown source with `git rm docs/examples/<stem>.md`.
7. Update the callback module, as the previous section describes.
8. Confirm that a fresh run reproduces the stored outputs. The command writes nothing.

   ```bash
   python scripts/execute_notebook.py docs/examples/<stem>.ipynb --check > check.log 2>&1
   echo "exit status $?"
   ```

9. Confirm that the notebook has line-feed endings. The command prints `False`.

   ```bash
   python -c "import sys; print(bytes([13, 10]) in open(sys.argv[1], 'rb').read())" docs/examples/<stem>.ipynb
   ```

10. Run the targeted checks, and read every finding.

    ```bash
    ruff format --check docs/examples/<stem>.ipynb
    python -m tests.prose --path docs/examples/<stem>.ipynb
    pytest -q -p no:cacheprovider tests/unit/test_documentation_runtime.py -k <module>
    pytest -q -p no:cacheprovider tests/unit/test_documentation_links.py -k <module>
    ```

Use the callback module name for ``<module>``. It replaces hyphens with underscores, so the
selection is valid in a pytest expression.

11. Report to the orchestrator. The report is in the last section.

A markdown edit does not change the stamp, so it needs no new execution. A code edit does. After a
code edit, go back to step 3. A new execution can move a number, and the narration test then names
each reading to correct.

Remove `execute.log` and `check.log` before you hand off.

## Share one machine

Several authors execute notebooks at the same time on one machine. Follow these limits.

| limit | reason |
| --- | --- |
| Run `pytest` without `-n`, and never with `-n auto` | a worker pool per author makes the authors contend for the same cores |
| Run only the targeted checks above. Do not run the full suite or `nox -s docs` | the orchestrator runs both once, after the conversions |
| Execute one notebook at a time | each execution starts its own kernel |
| Set `n_jobs=1` everywhere | a parallel learner competes with the other authors |

## Do not edit shared files

Edit only your notebook, your callback module, and the deletion of your Markdown source. Report any
other change the conversion needs. The orchestrator applies shared edits one at a time, so parallel
authors do not collide on one file.

| shared file | what to report instead |
| --- | --- |
| `tests/prose-report.md` | the finding id, the sentence, and the `accepted: <reason>` you propose |
| `docs/examples/index.md`, another tutorial, or any page that links to your `.md` file | each link to rewrite. `git grep -n "<stem>.md"` lists them |
| `tests/unit/test_documentation_runtime.py`, `tests/unit/tutorial_semantics/__init__.py`, `tests/documents.py`, `tests/notebooks.py`, `tests/conftest.py` | the change and the failure it repairs |
| `scripts/execute_notebook.py`, `docs/conf.py`, `pyproject.toml`, `uv.lock` | the change and the failure it repairs |

The report also gives the notebook's execution time, the exit status of each command in step 10,
and every claim of the Markdown tutorial that the notebook changed or removed.