Technical reference

This reference accounts for every statistical method cleverly ships, and for the evidence that each one is implemented correctly.

The project validates a method in two independent ways. It compares cleverly against a canonical implementation, where a maintained one exists. It also measures cleverly against the repeated-sampling properties the method’s own source theory predicts. Neither one replaces the scientific instruments that check the derivation itself: the exact-law, Gateaux, remainder, identity, and deliberate-mutation checks recorded in the evidence manifest.

Implementation matrix

The implementation validation grid covers the methods with a registered validation study, which has property cells. This matrix covers every implementation family cleverly ships, and names the correctness evidence for each.

implementation family	theory and citation	cleverly implementation	external provenance	correctness evidence
Counterfactual means; ATE; risk and odds ratios; multi-arm contrasts	Point-treatment TMLE; van der Laan & Rubin (2006)	estimators/tmle.py, targets/builtin.py	R tmle; pinned tmle3 parameter/update source	ate, ey1, ey0, rr, or oracle, Gateaux, remainder, and identity rows; gated fixed-weight and learned-nuisance weighted studies
ATT and ATC	Conditional-population effects; van der Laan (2010)	inference/influence.py	no numeric external oracle claimed	att, atc, including conditioning-event influence terms
Known treatment mechanism	Known treatment mechanism, known node mechanisms and the DR-TMLE contract; Moore & van der Laan (2009), van der Laan & Gruber (2012), Benkeser et al. (2017)	data/known_mechanism.py, estimators/_nuisance.py, longitudinal/sequential.py	R tmle 2.1.1 with g1W, R drtmle 1.1.2 with gn, and R ltmle 1.3-0 with a numeric gform	reporting-policy TMLE study, DR-TMLE study and LTMLE study; exact laws, witnesses, and mutations remain in estimator variants evidence
Natural-course mean, PAR, and PAF	Population interventions; Díaz Muñoz & van der Laan (2012) for complete outcomes; Díaz, Carone & van der Laan (2016) for the MAR mean; Zheng & van der Laan (2011) and Levy (2018) for its stacked CV-TMLE; the MAR stack for PAR and PAF	targets/builtin.py, estimators/tmle.py	R tmle 2.1.1 population-mean path with supplied predictions for the ordinary and stacked MAR targets, composed per arm for PAR and PAF; stochastic-intervention literature	ey_obs, par, paf exact-law and transformation checks; gated ordinary and stacked missing-outcome natural-course studies, and the gated missing-outcome attributable-effect study
Missing outcomes and controlled direct effects	Observed-data extensions; Díaz & van der Laan (2017) for the randomized arm-indexed missing-outcome construction	estimators/direct_effect.py, estimators/tmle.py	R tmle 2.1.1 for ordinary and stacked arm-indexed MAR and controlled direct effects; pinned R drtmle 1.1.2 for the randomized both-correct limit	gated ordinary arm-indexed MAR, stacked arm-indexed MAR, controlled direct-effect, and randomized DR-TMLE studies, plus the exact MAR and CDE laws, Gateaux checks, remainder identities, and deliberate mutations
Deterministic and dynamic regimes	Known regimes; Robins (2004), Díaz Muñoz & van der Laan (2012)	interventions/base.py, estimators/tmle.py	pinned R lmtp 1.5.4 at f04a2b4	gated deterministic-regime study, plus ey_regime, ate_regime, dynamic-rule and support mutations
Known stochastic regimes	Known regimes; Díaz & van der Laan (2013)	interventions/base.py, estimators/tmle.py	pinned R lmtp 1.5.4 at f04a2b4, reached through the point adapter	gated stochastic-regime study, including density replacement and targeting controls
Value of a learned treatment rule	Learned rules; van der Laan & Luedtke (2015), Section 7 and Appendix B; Montoya, van der Laan, Skeem & Petersen (2023)	interventions/learned.py, estimators/tmle.py	no numeric external oracle claimed. R tmle3mopttx weights the folds by their size and centres the curve at the pooled estimate	gated learned-rule study at two laws that meet the limiting-rule condition, with the declared logistic outcome learner, and a reporting boundary study: the interval under-covers at the exceptional and weak-blip laws that it reads; plus the ey_learned_rule oracle, Gateaux and identity rows
Continuous modified treatment policies	Modified treatment policies; Díaz Muñoz & van der Laan (2012), Haneuse & Rotnitzky (2013), Díaz et al. (2023)	interventions/policy.py, estimators/tmle.py	pinned R lmtp 1.5.4 at f04a2b4	reporting-policy continuous-policy study and point-policy study, plus ey_policy, ate_policy, continuous-law Gateaux and remainder checks
Modified treatment policies on a vector treatment	Vector treatments and vector nodes with a continuous component. Díaz, Williams, Hoffman & Schenck (2023), Section 2, Theorem 1, Equation (3), Section 5.4 and Theorem 3. Hoffman et al. (2024)	interventions/policy.py, interventions/joint_density.py, longitudinal/sequential.py	pinned R lmtp 1.5.4 at f04a2b4 with vector trt nodes. The comparator runs are pending	a reporting vector study, whose Python property cells are declared and not yet run, plus ey_policy, ate_policy, the mixed-law Gateaux checks and the mutation controls of the vector witnesses
Continuous policies at declared boundaries, with learned and tuned inputs	Modified treatment policies at a node and limits of the continuous-policy evidence. Díaz, Williams, Hoffman & Schenck (2023)	longitudinal/sequential.py, interventions/policy.py	pinned R lmtp 1.5.4 at f04a2b4 for the boundary and pricing studies, with comparator runs pending. No package fits a tuned cap	reporting boundary study, pricing study and tuned-cap study, each with Python property evidence
Incremental propensity-score interventions	Incremental interventions; Kennedy (2019)	interventions/incremental.py, estimators/tmle.py	R npcausal at 56a5ac1, the implementation Kennedy (2019) publishes	gated incremental-intervention study, plus ey_ipsi, ate_ipsi, including nonzero treatment-score and one-sided remainder witnesses
Incremental interventions over time	Incremental interventions over time; Kennedy (2019), Kim, Kennedy & Naimi (2021)	longitudinal/ipsi.py, longitudinal/estimator.py	R npcausal 0.1.0 at 56a5ac11 for the complete-data estimator, with comparator runs pending. No maintained package ships the dropout construction	gated complete-data study with Python property evidence, and a reporting dropout study with no comparator; exact laws, Gateaux checks and the npcausal transcription in longitudinal evidence
Categorical incremental interventions	Incremental interventions and over time; Kennedy (2019), Equation (1) and Theorems 1 and 2, with the general-level extension of the theory pages	interventions/incremental.py, longitudinal/ipsi.py	none. No maintained package ships a categorical odds tilt with its influence curve	gated categorical incremental study with Python property evidence, plus ey_ipsi, ate_ipsi and the mutation controls of tests/unit/test_categorical_incremental_mutations.py
Point and longitudinal MSM projections	Marginal structural models; Neugebauer & van der Laan (2007), Rosenblum & van der Laan (2010), Petersen et al. (2014)	msm.py, longitudinal/msm.py	pinned tmle3 Param_MSM; projected ordinary R ltmle and cross-fitted R lmtp regimen fits; raw R ltmleMSM differs in projection scale	gated point, ordinary longitudinal and cross-fitted longitudinal studies, plus the point msm and longitudinal MSM oracle rows
Longitudinal end-of-study TMLE	Sequential regression; Bang & Robins (2005), van der Laan & Gruber (2012); cross-fitting, Díaz et al. (2023)	longitudinal/estimator.py, longitudinal/sequential.py	R ltmle 1.3-0 for ordinary binary fitting; pinned lmtp 1.5.4 for cross-fitting and categorical treatment	a gated ordinary and a reporting cross-fitted binary study, plus gated ordinary and cross-fitted categorical studies, a reporting known-policy study, and a reporting modified-treatment-policy study; exact laws, Gateaux checks, and mutations remain in longitudinal evidence
Sequentially doubly robust longitudinal estimation	The sequentially doubly robust estimator; Díaz, Williams, Hoffman & Schenck (2023), Equation (4) and Section 5.3	longitudinal/sdr.py	pinned R lmtp 1.5.4 at f04a2b4, cf_sdr	a reporting SDR study; exact laws, the longhand recursion and mutations remain in longitudinal evidence
Sequential g-computation and IPW over time	Sequential g-computation and IPW; Bang & Robins (2005), Section 4; Stefanski & Boos (2002), Section 2	longitudinal/estimator.py, longitudinal/sequential.py	R ltmle 1.3-0 and R stremr, as parity tests	Python property evidence, a reporting sequential-baselines study; R comparator parity: ltmle (W9, tests/unit/test_sequential_baselines_ltmle_r.py) and stremr (tests/unit/test_sequential_baselines_stremr.py); the technical page lists the witnesses
Structural nested mean model	Structural dynamic effects; Lewis & Syrgkanis (2021), supplement Theorem 8 and arXiv Theorem 4	longitudinal/structural.py	Python property evidence, with comparator runs pending: EconML 0.17.0 DynamicDML in a pinned container. A fixture-level parity test checks the point estimates	reporting structural nested mean study; exact finite laws, influence-curve derivative checks and mutation controls on the technical page; parity with DynamicDML in tests/unit/test_structural_econml_parity.py
Parametric g-formula over time	Parametric g-formula; Robins (1986), eq. (4.7); McGrath et al. (2020); Pakes & Pollard (1989) and Newey & McFadden (1994) for the simulated-estimator step	longitudinal/gformula.py, longitudinal/estimator.py	pygformula 1.2.0 on four fixtures where pooled and node-specific models are equal. gfoRmula 1.1.1 and pygformula fit pooled models, so they are not a comparator on the study laws	Python property evidence, a registered parametric-gformula study (page docs/technical-reference/method-evidence/parametric-gformula.md); the technical page lists the witnesses
Longitudinal survival and competing risks	Event-process TMLE; Stitelman et al. (2012)	longitudinal/estimator.py	R ltmle 1.3-0 for ordinary survival; pinned lmtp 1.5.4 for cross-fitted survival and both competing-risk rows	a reporting ordinary survival study, and gated cross-fitted survival and ordinary and cross-fitted competing-risk studies, with a separate cross-fitted page for each; exact laws, Gateaux checks, and deliberate mutations remain in longitudinal evidence
Longitudinal repeated outcomes, IPSI	IPSI on a repeated outcome; Kim, Kennedy and Naimi (2021)	longitudinal/estimator.py, longitudinal/ipsi.py	none claimed. Python property evidence	gated repeated-outcome IPSI study; exact laws and Gateaux checks in longitudinal evidence
Longitudinal repeated outcomes	Repeated outcomes; Bang & Robins (2005), van der Laan & Gruber (2012)	longitudinal/estimator.py, longitudinal/sequential.py	Python property evidence, with comparator runs pending: R ltmle 1.3-0, fitted on the prefix through each horizon	gated repeated-outcome LTMLE study; exact laws, Gateaux checks, and mutations remain in longitudinal evidence
Point-treatment survival	One baseline treatment held over the nodes and time-to-event input; Benkeser, Carone & Gilbert (2018)	longitudinal/data.py, longitudinal/sequential.py	R survtmle 1.1.1 for the arm risks and incidences; pinned lmtp 1.5.4 for a policy at baseline	reporting-policy survival study, cross-fitted survival study, policy study, hazard targeting study, whose survtmle comparator runs are pending, one-step curve study, whose tmle3 comparator runs are pending, and cross-fitted hazard targeting study, which has no comparator; exact laws, Gateaux checks, and mutations remain in longitudinal evidence
Collaborative TMLE	Collaborative selection; van der Laan & Gruber (2010), Ju, Gruber et al. (2019) for the selector paths; Ju, Wyss et al. (2019) for the lasso path and C-TMLE0; Benkeser, Cai & van der Laan (2020), Theorem 1, arm by arm, for the per-arm outcome-adaptive design	estimators/ctmle.py	pinned R ctmle ctmleGlmnet and ctmleGeneral for the lasso and learners paths and for C-TMLE0 (ctmletype = 3); pinned ctmle3 outcome-adaptive source; pinned tmle3 / sl3 fold semantics. Neither ctmle nor ctmle3 supplies weighted parity	C-TMLE estimator-variant evidence, the reporting-policy selector and outcome-adaptive studies, the per-arm outcome-adaptive study and its strata and missing-data study, the candidate-sequence study and its R witness fixture, the C-TMLE0 derivative study and its R witness fixture, selector mutations, support path tests, and the fixed-weight refit controls in the simulated common-cause surface
DR-TMLE	Doubly-robust inference; van der Laan (2014), Benkeser et al. (2016/2017), Benkeser & Hejazi (2023)	estimators/drtmle.py, estimators/reduced.py	pinned R drtmle 1.1.2 source and bounded numerical comparators for complete outcomes, the randomized MAR both-correct limit, and the observational composite indicator	reporting-policy complete-outcome, three-arm missing-outcome and composite missing-data studies, a gated randomized missing-outcome study, a reporting one-inconsistent-nuisance witness study, DR-TMLE contract, theorem identity, Gateaux, remainder, score, cross-fit, and missing-data mutations
Cross-fitting and CV-TMLE	CV-TMLE and cross-fitting; Zheng & van der Laan (2011), Levy (2018), Chernozhukov et al. (2018)	learners/crossfit.py	pinned tmle3 stacked validation likelihood; pinned lmtp and pinned ife for cluster-robust inference on the package’s own grouped fold draw; R tmle 2.1.1 population mean with supplied stitched nuisances; no parity implementation for the complete repeat rule	gated stacked CV-TMLE, fold-evaluated CV-TMLE, fold-targeted CV-TMLE, and binary-outcome clustered CV-TMLE rows; reporting-policy unequal-size clustered CV-TMLE and few-cluster t reference rows; the fold and outcome-scale rules with their witnesses; gated missing-outcome natural-course CV-TMLE and arm-indexed missing-outcome CV-TMLE; reporting-policy repeated CV-TMLE; leakage and repeated-fold mutations
Targeting and fitting options	Stopping rules, step bound and best iterate, per-arm outcome regression, treatment-mechanism covariates and patience. No option adds an estimand	fluctuation/_score.py, fluctuation/iterative.py, estimators/_nuisance.py, estimators/ctmle.py, data/treatment_covariates.py	pinned tmle3 updater, R drtmle stratify, and R ctmle patience and Wg, as comparator fixtures: tmle3_updater.json, drtmle_stratify.json, ctmle_patience.json and ctmle_wg.json	Python exact tests, with witnesses and mutation controls, in the options table; exact tests and R parity, and for ctmle the parity of the option
Inference, diagnostics, and sensitivity	How every method reports uncertainty and sensitivity and validation methods; sensitivity references	inference/, sensitivity/, validation/, assessment.py	no external implementation used as acceptance evidence; three pinned DoWhy sources at revision 2116d5c supply secondary control-flow evidence: the dummy outcome refuter, bootstrap refuter, and simulated common-cause refuter	inference, assessment, sensitivity, and serialization mutations

“Existing implementation” has two meanings in this matrix. One is the local module that executes the method. The other is a pinned external implementation, where the source was audited. External parity is acceptance evidence for the claims it covers. It is never the derivation, and it is never a required gate. Each comparator-backed study declares one comparison kind: a matching implementation, a different estimator, or a related target.

The natural-extension verdicts record which refused compositions qualify for implementation as natural extensions, and the roadmap row that owns each one. The hard-stop review records the verdict on each inherited hard stop of the roadmap, and the row that now owns each part.

How to read the evidence column

The evidence manifest distinguishes independent scientific checks from bounded implementation witnesses. An exact finite-support law can verify a complete functional but miss a term that vanishes at the truth. A Gateaux comparison can verify the influence curve but share a mistake with the law. Remainder-rate and deliberate-mutation tests make those blind spots visible. An external comparison does not substitute for these checks on a changed term.

• Implementation validation studies
  • Implementation validation grid
  • How to read these studies
  • The red-cell ledger
  • Canonical point-treatment TMLE
  • Weighted point-treatment TMLE
  • Learned weighted point-treatment TMLE
  • Ordinary multi-arm TMLE
  • Stratified point-treatment TMLE
  • Stratified incremental MSM
  • Stratified identity-link MSM
  • Stratified DR-TMLE
  • Stacked point-treatment CV-TMLE
  • Clustered point-treatment CV-TMLE
  • Clustered point-treatment CV-TMLE at unequal cluster sizes
  • Clustered TMLE intervals on a t reference at few clusters
  • Clustered cross-fitted end-of-study longitudinal TMLE
  • Cross-fitted clustered longitudinal TMLE on a t reference at few clusters
  • Fold-evaluated point-treatment CV-TMLE
  • Fold-targeted point-treatment CV-TMLE
  • Repeated point-treatment cross-fitted TMLE
  • Selector-based point-treatment C-TMLE
  • Selector-based multi-arm C-TMLE
  • C-TMLE candidate sequences
  • C-TMLE0 derivative covariate
  • Outcome-adaptive point-treatment C-TMLE
  • Outcome-adaptive multi-arm C-TMLE
  • Outcome-adaptive per-arm C-TMLE
  • Outcome-adaptive per-arm C-TMLE with strata and missing data
  • Outcome-adaptive per-arm C-TMLE (parametric)
  • Outcome-adaptive per-arm C-TMLE (parametric crossfit)
  • Outcome-adaptive per-arm C-TMLE (parametric extensions)
  • Canonical DR-TMLE
  • Multi-arm DR-TMLE
  • Ordinary missing-outcome TMLE
  • Ordinary missing-outcome natural-course TMLE
  • Stacked missing-outcome natural-course CV-TMLE
  • Stacked arm-indexed missing-outcome CV-TMLE
  • Missing-outcome attributable-effect TMLE
  • Ordinary controlled direct-effect TMLE
  • Randomized missing-outcome DR-TMLE
  • Observational missing-data DR-TMLE
  • Randomized multi-arm missing-outcome DR-TMLE
  • Point-treatment MSM projection
  • Deterministic point-treatment regimes
  • CV-TMLE of the fold-local learned-rule value
  • CV-TMLE of the fold-local learned-rule value at exceptional and weak-blip laws
  • Stochastic point-treatment regimes
  • Continuous modified treatment policies
  • Point modified treatment policies
  • Incremental propensity interventions
  • Ordinary end-of-study longitudinal TMLE
  • Ordinary weighted end-of-study longitudinal TMLE
  • Ordinary categorical longitudinal TMLE
  • Stochastic categorical longitudinal TMLE
  • Longitudinal modified treatment policies
  • Longitudinal incremental interventions
  • Longitudinal incremental propensity-score interventions under dropout
  • Categorical incremental policies
  • Vector modified treatment policies
  • MTP boundaries
  • MTP pricing
  • MTP tuning
  • Sequentially doubly robust estimation
  • Ordinary longitudinal MSM projection
  • Cross-fitted longitudinal MSM projection
  • Cross-fitted end-of-study longitudinal TMLE
  • Cross-fitted weighted end-of-study longitudinal TMLE
  • Cross-fitted categorical longitudinal TMLE
  • Ordinary survival-curve longitudinal TMLE
  • Cross-fitted survival-curve longitudinal TMLE
  • Ordinary competing-risk longitudinal TMLE
  • Cross-fitted competing-risk longitudinal TMLE
  • Point-treatment survival
  • Cross-fitted point-treatment survival
  • Point survival hazard
  • Point survival curve
  • Point survival curve crossfit
  • Point-treatment survival policies
  • TMLE on a declared known treatment mechanism
  • DR-TMLE on a declared known treatment mechanism
  • DR-TMLE with one inconsistent nuisance
  • LTMLE on declared known node mechanisms
  • Sequential g-computation and IPW
  • Structural nested mean model
  • Parametric g-formula
  • Repeated-outcome longitudinal TMLE
  • Repeated-outcome longitudinal IPSI
  • Omitted-variable bound standard error
  • Calibration-slope warning
  • Default simultaneous bands
  • Full-refit bootstrap and derived contrasts
• What each estimand’s correctness rests on
  • The instruments
  • The table
  • Where the repeated-sampling evidence lives
  • Estimator variants over registered targets
  • Targeting and fitting options
  • Longitudinal estimands outside the target registry
  • Post-fit functionals of reported estimates
  • A simulated law is an instrument too, and it can be wrong the same way
  • What this table says is missing
• Sensitivity and validation methods
  • Diagnostics on the fit you have
  • Sensitivity to untestable assumptions
  • Refutation and simulation you run
  • How the library certifies itself
• How every method reports uncertainty
  • Influence-curve variance
  • Covariance rules
  • Inference status
  • Clusters
  • Transformed parameters
  • Simultaneous bands
  • Reporting a subset of a family
  • What is not on this page
• Point-treatment TMLE
  • What this solves
  • The algorithm as implemented
  • Variations
  • Validation issues special to this method
• CV-TMLE and cross-fitting
  • What this solves
  • The algorithm as implemented
  • Variations
  • Fold and outcome-scale rules
  • Reusable outer split plans
  • Validation issues special to this method
• Collaborative TMLE
  • What this solves
  • The algorithm as implemented
  • Variations
  • Candidate sequences
  • C-TMLE0: the derivative covariate
  • Validation issues special to this method
• DR-TMLE
  • The release claim, in brief
  • What this solves
  • The algorithm as implemented
  • Variations
  • Validation issues special to this method
• Longitudinal TMLE
  • What this solves
  • The algorithm as implemented
  • Functionals of a fitted result
  • Variations
  • The sequentially doubly robust estimator
  • Incremental interventions over time
  • Validation issues special to this method
• Sequential g-computation and IPW
  • What this solves
  • The estimators
  • Inference
  • Settings
  • Diagnostics
  • Refusals
  • Validation
• Structural dynamic effects
  • What this solves
  • The model
  • Identification of the static level and contrast
  • The estimating equation
  • Influence curves
  • Status
  • The model check
  • Refusals
  • Comparator
  • Validation
  • Sources
• Parametric g-formula
  • What this solves
  • Status
  • The estimator
  • Inference
  • Declared treatment mechanisms
  • Fixed doses, rule-set doses and model reliance
  • Positivity
  • The order within a node
  • Settings
  • Refusals
  • How it differs from gfoRmula and pygformula
  • Validation
  • References
• Marginal structural model projections
  • What this solves
  • The algorithm as implemented
  • Variations
  • Validation issues special to this method
• Scope and refusals
  • How to read a refusal
  • Where a multi-valued treatment is supported
  • Vector treatments
• Natural-extension verdicts
  • The terms
  • The verdicts
  • The conditions each part inherits
• Hard-stop review
  • The terms
  • The verdicts
  • The conditions each moved part inherits
  • Abstract-only sources
• References
  • Targeted learning, in general
  • Point treatment and stochastic interventions
  • Known treatment mechanisms
  • Grouped folds and clustered cross-fitting
  • Collaborative TMLE
  • Longitudinal, survival and marginal structural models
  • Incremental interventions
  • Riesz representation and nested targeted learning
  • Sensitivity analysis
  • Negative controls
  • Refutation
  • Simulation studies
  • Multiple testing
  • Calibration of prediction models
  • Doubly-robust inference (DRTMLE)
  • The TWINS example
  • Proposed methods on the roadmap