cleverly.validation.NuisanceDiagnostics

class cleverly.validation.NuisanceDiagnostics(models: tuple[NuisanceModelReport, ...], evaluation: EvaluationBasis, n_repeats: int = 1, backend: str | None = None, selection: CTMLESelection | CTMLEOutcomeAdaptiveFit | CTMLEDerivativeFit | None = None, treatment_role: TreatmentModelRole | None = None, repeat_spread: tuple[RepeatSpreadRow, ...] = (), selection_omission: str | None = None, repeat_spread_omission: str | None = None, reported_repeat: int = 1, inference: InferenceStatus = 'influence_curve')

Fit quality for every nuisance model in a TMLE fit.

The metrics are out-of-fold when the fit cross-fitted its nuisances, and in-sample when it used one fold. evaluation records which.

Parameters:

modelstuple of NuisanceModelReport

One report per fitted nuisance model.

evaluation{“out_of_fold”, “in_sample”}

Whether the predictions the reports read were made out of fold. The calibration-slope rule tests out-of-fold predictions only, because an unpenalized logistic model with an intercept has an in-sample slope of exactly 1.

n_repeatsint

Cross-fitting draws the fit combined. The reports describe the first.

backendstr or None

Dataframe backend to_frame() returns when data is omitted.

selectionCTMLESelection or CTMLEOutcomeAdaptiveFit or CTMLEDerivativeFit or None

The fitted CTMLESelection, CTMLEOutcomeAdaptiveFit or CTMLEDerivativeFit. None means the method is not collaborative, or the stored artifact is unavailable.

treatment_rolestr or None

One of "estimated_treatment_law", "declared_treatment_law" and "collaborative_working_model", or None: the statistical role of the propensity predictions in models. A collaborative working mechanism is not the complete treatment law. A declared treatment law is the known mechanism the data declares, and its rows compare that declaration with the observed treatment rather than describe a fitted model.

repeat_spreadtuple of RepeatSpreadRow

One descriptive split-sensitivity row per reported parameter. A one-draw fit retains an empty tuple rather than a false zero spread.

selection_omissionstr or None

Machine-readable reason an expected C-TMLE artifact is absent.

repeat_spread_omissionstr or None

Machine-readable reason no split-spread row is available.

reported_repeatint

One-based draw described by the nuisance models and selection artifact.

inferencestr

The inference status the fit’s estimates declare. One of InferenceStatus, which the inference reference lists. Read here rather than off treatment_role, which reads "collaborative_working_model" for the outcome-adaptive path as well, and that path has a status of its own.

Methods

NuisanceDiagnostics.calibration_frame(name: str, data: Any = None) → Any

Binned observed-vs-predicted table for one model.

Parameters:

namestr

Which nuisance model to tabulate.

dataAny

A dataframe or fitted container whose backend to match. None uses the backend recorded on this object.

Returns:

dataframe

One row per bin, with the mean prediction and the observed rate in it.

NuisanceDiagnostics.repeat_spread_frame(data: Any = None) → Any

Return one split-sensitivity row per reported parameter.

Parameters:

dataAny

A dataframe or fitted container whose backend to match. None uses the backend recorded on this object.

Returns:

dataframe

Parameter aliases, draw counts, split standard deviations, reported standard errors, and their descriptive ratios. A fit that supplies no inference names the last two columns plugin_standard_error and ratio_to_plugin_standard_error, because the number is its plug-in diagnostic and not a standard error.

NuisanceDiagnostics.summary() → str

Return a printable summary.

Returns:

str

A printable table, one line per nuisance model.

NuisanceDiagnostics.to_frame(data: Any = None) → Any

Return tabular output in the input dataframe backend.

Parameters:

dataAny

A dataframe or fitted container whose backend to match. None uses the backend recorded on this object.

Returns:

dataframe

One row per nuisance model, with its fit metrics.

NuisanceDiagnostics.verdict() → str

A reading of the diagnostics that says what to do about them.

Returns:

str

A reading of the diagnostics that says what to do about them.

Attributes

NuisanceDiagnostics.backend: str | None = None

Name of the dataframe backend the fit’s data arrived in, so that to_frame() honours “results come back in the backend you passed in” without a caller having to thread the container back in by hand.

NuisanceDiagnostics.findings

Return findings that meet an existing diagnostic warning rule.

NuisanceDiagnostics.inference: InferenceStatus = 'influence_curve'

Keyed on what the estimates declare rather than on treatment_role, which cannot tell a selector path from the outcome-adaptive one.

NuisanceDiagnostics.inference_note

The fact a report adds when the fit’s estimates carry a labelled status.

The assessment note the status table records for a fit whose estimates declare a labelled status, shown or withheld, and None otherwise. Keyed on inference and not on treatment_role, which the outcome-adaptive path shares, and that path has a status of its own. summary() and the assessment’s nuisance-model row both read it here.

NuisanceDiagnostics.n_repeats: int = 1

How many cross-fitting draws the fit combined. The reports above describe the first draw, for the reason n_repeats gives: a model’s out-of-fold calibration is a property of that model, and one fitted under a different split is a different model rather than another measurement of the same one. Averaging their AUCs would report a number no model achieved.

NuisanceDiagnostics.repeat_spread: tuple[RepeatSpreadRow, ...] = ()

Unlike the nuisance models and selection, these rows read every retained draw.

NuisanceDiagnostics.repeat_spread_omission: str | None = None

NuisanceDiagnostics.reported_repeat: int = 1

Nuisance models and the selection artifact describe this draw. One is the only draw the fitted result retains method-specific extras for.

NuisanceDiagnostics.selection: CTMLESelection | CTMLEOutcomeAdaptiveFit | CTMLEDerivativeFit | None = None

The exact method object already retained by the fitted result. Selector paths keep every candidate and risk rather than flattening the selected index into prose.

NuisanceDiagnostics.selection_omission: str | None = None

NuisanceDiagnostics.treatment_role: TreatmentModelRole | None = None

The propensity report’s role. The role changes its interpretation, not its values.

NuisanceDiagnostics.models: tuple[NuisanceModelReport, ...]

NuisanceDiagnostics.evaluation: EvaluationBasis