cleverly.sensitivity.benchmark

cleverly.sensitivity.benchmark(result: TMLEResult, covariates: Any, *, estimand: str = 'ate', nu2_estimator: str = 'auto', random_state: int | None = None) → BenchmarkResult

Calibrate cf_y and cf_d against observed covariates.

Refits the whole model without the named covariates (the “short” model) and compares it with the full fit. The resulting gain statistics say how strong a confounder like the dropped ones would be, on the cf_y/cf_d scale, which turns an abstract sensitivity parameter into a concrete comparison.

Note this is a genuine refit, so it costs about as much as the original fit.

Parameters:

resultTMLEResult

A fitted result.

covariatessequence of str

Observed covariates to calibrate against. An encoded categorical covariate is named by its column, and the refit drops its whole block of indicator columns.

estimandstr

Alias to benchmark.

nu2_estimator{“auto”, “doubly_robust”, “plugin”}

Which estimator of the Riesz second moment to use.

random_stateint or None

Seed for the short refit. None uses the seed the fit was run with. A fit with no seed draws one. Either way the result records it under random_state, and passing that value back repeats the benchmark.

Returns:

BenchmarkResult

The sensitivity parameters a confounder as important as those covariates would need, and how far the estimate moved without them.

Raises:

DataError

If covariates is empty, names an indicator column of an encoded covariate, or names a covariate the fit does not adjust for. The argument is checked before any refusal.

ValueError

If nu2_estimator is not one of NU2_ESTIMATORS.

CapabilityError

If the result carries no fitted estimator, on every refusal sensitivity_elements() raises for the full fit or the short refit, and on the refusal of benchmark_refusal. The full fit and the covariates are checked before the refit runs.