cleverly.validation.CoverageStudy

class cleverly.validation.CoverageStudy(dgp: DGP | Callable[..., tuple[Any, dict[str, float]]], estimator: Callable[[], TMLE | LTMLE], *, n: int = 1000, n_replicates: int = 100, estimands: Sequence[str] | None = None, fit_kwargs: dict[str, Any] | None = None, seed: int | None = None, sample_seeds: Sequence[int] | None = None, n_jobs: int = 1, truth_key: str = 'population', intermediate_value: float | None = None, label: str | None = None, checkpoint: ReplicationCheckpoint | None = None)

Repeatedly fit an estimator on fresh data and measure bias and coverage.

Parameters:

dgpDGP or callable()

A DGP, or a callable (n, seed) -> (frame, truth) following the convention of the generators in cleverly.datasets.

estimatorcallable()

A zero-argument factory returning a fresh estimator per replication. A factory rather than an instance, so replications cannot share fitted state.

nint

Sample size per replication.

n_replicatesint

How many replications to run.

estimandssequence of str or None

Which estimands to summarise; defaults to whatever the first fit reports.

fit_kwargsmapping or None

Passed to fit. Column names, delta=, id= and so on.

seedint or None

Root of the distinct per-replication sample seeds.

sample_seedssequence of int or None

Explicit distinct uint32 sample seeds, one per replication. Use these to coordinate separate studies. When supplied, these replace the sample stream from seed.

n_jobsint

Number of joblib workers across replications.

truth_key{“population”, “sample”}

"population" (default) compares against the population estimand, fixed across replications; "sample" compares against each replication’s realised sample estimand, which removes one source of variability but changes what coverage means.

intermediate_valueobject or None

The level of the intermediate variable to study, for a controlled direct effect. Required when fit_kwargs contains intermediate=, because such a fit returns one result per level and each level is a different parameter: the study picks that level out of the result set and asks the process for the truth at the same level, so the two cannot silently disagree. See cleverly.estimators.direct_effect.

labelstr

Name of the study, used in its reports.

checkpointReplicationCheckpoint or None

Storage for finished replications. With one, run() reuses every stored outcome and fits only the rest, so a stopped study resumes where it stopped.

Notes

A replication whose fit raises a numerical or fitting error is recorded as a ReplicationFailure and the study continues. These are recorded: DataError, LongitudinalError, numpy.linalg.LinAlgError, FloatingPointError, ZeroDivisionError, OverflowError, and a plain ValueError or RuntimeError that numpy, scipy, scikit-learn or statsmodels raised. Every other exception stops the study, including CapabilityError, MethodConfigurationError and programming errors such as TypeError. An error while drawing a sample always stops the study.

Examples

>>> from sklearn.linear_model import LinearRegression, LogisticRegression
>>> from cleverly.estimators import TMLE
>>> from cleverly.datasets import nonlinear_dgp
>>> from cleverly.validation import CoverageStudy
>>> study = CoverageStudy(
...     dgp=nonlinear_dgp(),
...     estimator=lambda: TMLE(
...         outcome_learner=LinearRegression(),
...         treatment_learner=LogisticRegression(max_iter=1000),
...     ),
...     n=500,
...     n_replicates=50,
...     seed=0,
... )
>>> study.n_replicates
50

Methods

CoverageStudy.run() → StudyResult

Execute the study.

Returns:

StudyResult

Per-replication records, per-estimand summaries, and the failures.

CoverageStudy.run_replication(replicate_index: int, seed: int) → tuple[ReplicationRecord, ...] | ReplicationFailure

Fit one declared replication from an explicit sample seed.

A fresh estimator is created for every call. A recordable fitting failure (see the class notes) returns an explicit record; any other exception propagates. This method does not aggregate a sampling distribution or change the study budget.

Parameters:

replicate_indexint

Original replication index within the declared budget.

seedint

Explicit uint32 seed supplied to the data generator.

Returns:

tuple of ReplicationRecord or ReplicationFailure

One record per estimand on success, or the fit’s explicit failure record.