cleverly.validation.DirectoryCheckpoint

class cleverly.validation.DirectoryCheckpoint(directory: str | PathLike[str])

Keep each finished replication of a CoverageStudy as a file in a directory.

One file per replication, written under a temporary name and then renamed, so a study stopped mid-write never reads a partial file. The directory belongs to one study. The load compares each stored seed with the study’s seed and refuses a mismatch. It compares nothing else, so a study with the same seed and another n, data-generating process or estimator reads the stored results. Give each configuration its own directory.

Parameters:

directorystr or os.PathLike

Where the files go. It is created when missing.

See also

ReplicationCheckpoint

The protocol this class implements.

Examples

>>> import tempfile
>>> from sklearn.linear_model import LinearRegression, LogisticRegression
>>> from cleverly.datasets import nonlinear_dgp
>>> from cleverly.estimators import TMLE
>>> from cleverly.validation import CoverageStudy, DirectoryCheckpoint
>>> def study(directory):
...     return CoverageStudy(
...         dgp=nonlinear_dgp(),
...         estimator=lambda: TMLE(
...             outcome_learner=LinearRegression(),
...             treatment_learner=LogisticRegression(),
...             cross_fit=False,
...         ),
...         n=200,
...         n_replicates=3,
...         seed=0,
...         checkpoint=DirectoryCheckpoint(directory),
...     )
>>> with tempfile.TemporaryDirectory() as directory:
...     first = study(directory).run()
...     resumed = study(directory).run()
>>> resumed.replications == first.replications
True

Methods

DirectoryCheckpoint.load(replicate: int) → tuple[ReplicationRecord, ...] | ReplicationFailure | None

Return the stored outcome of replicate, or None when there is none.

Parameters:

replicateint

Index of the replication.

Returns:

tuple of ReplicationRecord, ReplicationFailure or None

The stored outcome, or None.

DirectoryCheckpoint.save(replicate: int, outcome: tuple[ReplicationRecord, ...] | ReplicationFailure) → None

Store the outcome of replicate.

Parameters:

replicateint

Index of the replication.

outcometuple of ReplicationRecord or ReplicationFailure

What the replication returned.

Attributes

DirectoryCheckpoint.directory: str | PathLike[str]