Fast tests and validation studies#

This page separates the checks that run before every pull request from statistical studies that run only when a relevant implementation changes. README.md gives the fast command. Method benchmarking gives the study design and registration rules.

The two checks#

check	command	what it establishes
fast tests	`pytest -q -n auto --dist loadgroup`	unit, integration, end-to-end, documentation, provenance, artifact, and published-verdict behavior
registered validation study	the affected study directory’s documented regeneration command	repeated-sampling, large-sample, flexible-learner, and reference-implementation claims

Run the fast tests before every pull request. They recompute every published verdict and negative control from committed study artifacts. They do not refit committed study replications.

Run a registered study only when a result-determining change can affect what that study computes. Choose a complete rerun or an audited targeted rerun under the conditions below. Both routes retain every declared replication and recompute each affected summary and verdict.

The repository has no pytest slow tier. Registered studies replace the deprecated repeated-sampling tests. An unrecorded partial probe cannot replace published study evidence.

What belongs in the fast suite#

Use a fast test when one compact sample, an exact law, or a deliberate mutation can observe the behavior. This includes these checks:

public behavior, validation, serialization, and presentation;
exact-law, Gateaux, remainder, identity, and mutation instruments;
targeting, cross-fitting, nuisance routing, and inference formulas on compact data;
artifact hashes, schemas, provenance, generated claims, and verdict recomputation;
documentation parsing, links, and executable examples.

Use tests.conftest.FAST_KWARGS and explicit parametric learners for estimator tests. Use a flexible learner only when flexible learning is the subject.

Do not add an expensive pytest marker. A claim that needs repeated sampling, large samples, many flexible fits, or an external implementation belongs in a registered validation study.

Choose affected studies#

A change affects a study when it can move that study’s inputs, fits, or verdicts. Typical examples include these changes:

change	action
an estimand, influence curve, variance, clustering, targeting, cross-fitting, nuisance prediction, or randomization path	regenerate every registered row that evaluates the path
a study law, margin, cell, seed, learner, estimator argument, schema, package pin, or reference runner	regenerate the affected evidence; use a targeted rerun only when its eligibility conditions hold
documentation, formatting, comments, type annotations, or presentation that preserves fitted arrays	run fast tests only
validation that exits before estimator construction	run fast tests only

Use the implementation validation grid to find the rows for a method. Read each row’s coverage limits before selecting it. If several rows evaluate the changed path, regenerate each one.

Record the selected study names and commands in the pull request. For regenerated evidence, also name the artifacts that moved and those that stayed byte-identical.

Targeted reruns#

Use a targeted rerun when an audit identifies every affected replication before fitting replacement samples. A seed collision repair can meet this condition without refitting samples whose seeds stay unchanged. When the conditions below hold, the study can reuse unchanged fits.

condition	required evidence
bounded change	identify affected replication keys from inputs or code, before reading replacement outcomes
fixed design	preserve the laws, learners, sample sizes, budgets, margins, pairing, and reading rules
reusable evidence	pin the original artifact commit and verify every inherited artifact hash
unchanged path	show that inherited samples and fitted quantities retain their result-determining inputs and code
independent fits	construct fresh estimators and learners for each replication; exclude shared training state and execution-order dependence
runtime	match the original runtime for replacement fits; independently investigate any unexplained replay difference
complete accounting	replace every affected key exactly once; retain every unaffected row and all declared counts
dependent fits	rerun both sides of an affected paired comparison and every dependent fit
analysis	recompute all affected summaries, bootstrap intervals, controls, and verdicts from the complete combined rows
provenance	distinguish inherited evidence from replacement fits and recomputed analyses in the manifest
verification	independently check the selection, preserved evidence, replacement fits, complete analysis, and manifest

Select replacement seeds from the declared allocation rule. Do not select samples from their fitted results.

Keep an unchanged replay witness when it can test the claimed reuse boundary. A replay witness supports the code audit. It does not establish that every inherited fit is unchanged. Declare any numerical replay tolerance before fitting. Require exact agreement for keys, counts, and discrete decisions. Passing a tolerance alone does not explain a runtime or source difference.

Retain primary and reference files as exact bytes when the change cannot affect them. Retain unaffected replication rows exactly, even when recompressing their complete file changes its hash. Publish failed or unresolved verdicts under the original acceptance rules. Do not repair a failed fit by dropping its row or drawing a more favorable sample.

Use a complete rerun when the audit cannot bound the change or verify the inherited evidence. Estimator changes that affect every fit usually require this route. Changing a runtime, learner, law, or reference can also invalidate the claimed reuse boundary.

Method benchmarking defines the provenance record. Record the targeted command, selected keys, baseline, preserved artifacts, and validation in the pull request.

Result-neutral study edits#

A study manifest records the Python modules that produced the run. A Python hash difference does not fail the fast tests because a comment or extracted helper can change the bytes without changing the result. Regenerate only when the edit changes what the study computes.

Reference Dockerfiles and R sources follow a stricter rule. The fast suite refuses an undeclared hash difference. Regenerate a result-determining reference edit. Record a result-neutral edit in tests/canonical/provenance-revisions.md without rewriting the recorded manifest hash.

Read method benchmarking for the complete provenance contract.

Resume a study run#

tests/canonical/regenerate.py keeps the scratch of each run outside the repository. An R reference stores the result of each group as a checkpoint. A rerun of the same command reuses the Python phase, the calibration and each checkpoint. The rerun prints what it reused.

A failure checkpoint is reused like a result: the rerun does not refit a group whose comparator call failed. The Python phase stores each replicate as a checkpoint too, so a rerun refits only the replicates that the stopped run did not finish. The property phase does the same, with one checkpoint for each fit of each property loop. The failure rule says which errors stop the phase.

A run shares its cache with a rerun only when the two have one resume key. The key holds the declared study without its output path, HEAD and the tree state. It also holds the reference sources, the R harness, the content of the samples and the truth, n, the replicate count and the Python versions. A change to any of these starts a new cache.

item	location or command
host scratch	`%LOCALAPPDATA%/cleverly/runs/<slug>/<key>/` on Windows, `~/.cache/cleverly/runs/<slug>/<key>/` elsewhere. `--cache DIR` replaces it
R checkpoints	`/cache/<slug>/<key>/<runner>__<output>/` in the Docker volume `cleverly-cache`
Python checkpoints	`checkpoints/<implementation>/<label>/` in the host scratch. The label is `primary`, `properties`, or `properties/<name>`
discard the cache of this key	add `--fresh`
refit the Python phase	add `--refresh-python`. It also discards the Python checkpoints
reuse the committed subject rows	add `--reuse-subject`. It applies to a study with no reference process to run, for example one whose comparator runs are pending
host lock	`_lock` in the host scratch. A second driver on one key refuses and names the holder. The driver removes the lock of a dead process
container lock	the container name `cleverly-<slug>-<key>-<leaf>`. Docker refuses a second live container of one name
the RM39 smoke	`python -m tests.canonical.harness_fixture.smoke`

A run that completes deletes its key directory in the volume and its default host scratch. It does not delete a --cache directory. A declared run keeps both until its pull request merges, for audit. The run prints both paths, and the implementer records them in the row’s progress file. Delete them after the merge with these commands:

docker run --rm -v cleverly-cache:/cache alpine rm -rf /cache/<slug>/<key>
rmdir /s /q "%LOCALAPPDATA%\cleverly\runs\<slug>\<key>"

The driver sizes the R workers from measured memory. A calibration run fits the first group of each scenario in a fresh fork. It measures the peak and the private memory of each fork. The harness then sets the workers to 0.85 of the available memory over 1.25 times the per-worker cost.

A runner can declare Reference(worker_memory_mb=...) as a floor, or Reference(memory_override=(factor, share, reason)). At its start, a phase logs the memory, the calibration, the plan and its binding reason. Each retry pass logs MemAvailable and the workers.

event	what the run does
a killed worker	the next pass halves the workers and fits the group again, alone
a second kill of one group	the phase stops with exit 3 and names the group. A rerun resumes
a pass that makes no progress	the phase stops with exit 3. A rerun resumes
a container killed whole (exit 137)	the driver runs the container once more with `CLEVERLY_R_WORKER_CAP` at half the last workers
a second container kill	the run stops. A rerun resumes

Give the study machine a memory limit in .wslconfig, so a memory overrun kills a container and not the host. The log line MemTotal ... MemAvailable ... cgroup limit ... shows what the container saw.

Run the RM39 smoke after an edit to tests/canonical/study_harness.R, to a fixture runner, or to the tmle3 Dockerfile. The smoke runs in Docker, because CI installs no R. It commits its results and tests/canonical/harness_fixture/fixture-manifest.json. The fast suite fails until the manifest names the current bytes of each of those files.

Check that a migration moved no byte#

tests/canonical/python_phase_identity.py runs each study in two fresh interpreters: one in a worktree of the base commit and one in this checkout. Use it when a change moves the fitting loops of a study and must leave every committed byte as it was. The primary phase compares the samples, the truth rows, and the subject rows. Add --phase properties to compare the property phase. The head runs with a systematic prefix of one replicate, so the two-call split of fit_replicates runs at two replicates. Add --prefix default --budget 21 to run the split as a study runs it.

git worktree add /tmp/base <base commit>
python -m tests.canonical.python_phase_identity --phase properties --mode budget \
    --base /tmp/base --out /tmp/identity

mode	what it compares
`budget`	the rows of each module whose `generate_property_rows` caps its replicates (`budget=`, `replicates=` or `cap=`), byte for byte. The wall-clock columns in `TIMING_COLUMNS` stay out of the comparison
`batches`	the rows of each module that declares `sampling_batches()`, with each cell cut to `--budget` replicates, byte for byte
`payload`	the call sequence of every property loop of every module. The base patches `map_parallel` and the head patches `fit_properties`. The payloads, the `n_jobs` value, and the worker source must agree, except for the modules in `WORKER_EXCEPTIONS`. A `run_cells` loop compares its cells and sample seeds through `CoverageStudy`. A module that re-keys its fits maps both sides to one form first (`KEY_PROJECTIONS`). The mode also compares the rows of the truncated run, and it records an error of the module’s post-processing in `note.txt`
`labels`	the labels of the head’s `fit_properties` calls: unique, and `properties` or `properties/<name>`
`--fail R`	the head only. Every fit of replicate `R` raises `LinAlgError` inside the driver’s collector. The rows must equal the rows of a run with no failure minus the recorded cells. Every other `noise_control` row must stay unchanged. The stamps must equal the records, and `reconcile` must pass

A study without a committed property artifact is covered by the same runs against the base code. Any difference stops the work. Do not record a difference as expected unless the change declares it.

The all-study property differential of the failure accounting runs as the property-differential job of .github/workflows/ci.yml. The property failure rules describe it.

Notebook artifacts#

A committed notebook stores the outputs of a run rather than recomputing them. The fast suite checks that its stamp still matches its code cells and stored outputs. This detects an edit after stamping, but the unkeyed hashes do not prove that one payload produced the other. Re-execute with python scripts/execute_notebook.py <path>. The command needs network access.

The stamp it writes has two halves. The gated half covers the notebook alone, and the fast suite asserts it equal. The recorded half fingerprints the repository context of the run. The fast suite asserts only that it is present and well formed. A library edit therefore fails no notebook check. tests/notebooks.py gives the reason, and it is the reason a study manifest does not gate its Python module hashes either.

A hash comparison cannot see a library change that moved a published number while every cell kept its bytes. Run python scripts/execute_notebook.py <path> --check to find one. The command re-executes the notebook, reports each cell whose printed text moved, and writes nothing.

Add a validation study#

Follow method benchmarking to register a new study. Pair each positive claim with a control that must fail the same instrument.