Stratified DR-TMLE#
This study validates DRTMLE(guard=("Q", "g")) with baseline strata, strata=["V"]. The fit
solves every equation of the doubly robust alternation with one block per stratum. It fits every
reduced regression inside each stratum. Each stratum’s estimate is then the DR-TMLE of the law
given \(V = s\). DR-TMLE estimands states the
construction.
What was tested#
setting |
declaration |
|---|---|
law |
the complete-data binary law of Benkeser et al. (2017), Section 5.1, with a stratum \(V \in \{0, 1, 2\}\) (probabilities 0.5, 0.3 and 0.2) that shifts both intercepts and the treatment effect. |
nuisances |
a logistic outcome GLM on \((A, W_1, W_2, W_1 W_2, V, A V)\) and a logistic treatment GLM on \((W_1, W_2, W_1 W_2, V)\), both correct. A wrong GLM omits \(W_1 W_2\) |
fit |
cross-fitted on ten folds, |
primary scenario |
|
Monte Carlo inference |
exact 99% intervals around every rate, and 99% intervals around every primary endpoint |
The comparator is R drtmle 1.1.2 at commit 538a3a2. drtmle has no stratified form, so the
runner calls it once per stratum subset. Both sides read the same rows, the same ten-fold
assignment, and the subject’s initial nuisance arrays. Inside a stratum, drtmle fits its
reduced regressions on the stratum’s rows, which is the construction the subject uses.
difference |
effect |
|---|---|
the subject’s outer loop stops when every stratum’s equations meet the stop rule. R |
a stratum can take more rounds in the subject than in R |
the marginal estimate of a stratified fit reduces on \((g_n, V)\). A marginal |
the three marginal names are not paired. The property cells measure the marginal ATE against its exact truth |
Accuracy against known truth#
law |
estimand |
what was tested |
implementation |
bias (99% interval) |
coverage |
SE ratio |
result |
|---|---|---|---|---|---|---|---|
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=0 |
|
-0.0012 to 0.0046 |
0.9587 |
1.0690 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=0 |
R |
-0.0015 to 0.0043 |
0.9525 |
1.0630 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=1 |
|
-0.000489 to 0.0072 |
0.9450 |
1.0096 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=1 |
R |
-0.0012 to 0.0066 |
0.9437 |
1.0057 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=2 |
|
0.0022 to 0.0123 |
0.9287 |
0.9848 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=2 |
R |
0.0021 to 0.0122 |
0.9387 |
0.9727 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=0 |
|
-0.0038 to 0.0012 |
0.9537 |
1.0364 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=0 |
R |
-0.0035 to 0.0014 |
0.9450 |
1.0302 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=1 |
|
-0.0054 to 0.000269 |
0.9500 |
0.9912 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=1 |
R |
-0.0053 to 0.000332 |
0.9513 |
0.9952 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=2 |
|
-0.0033 to 0.0022 |
0.9587 |
1.0685 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=2 |
R |
-0.0032 to 0.0023 |
0.9575 |
1.0714 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=0 |
|
-0.0015 to 0.0023 |
0.9575 |
1.0317 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=0 |
R |
-0.0016 to 0.0022 |
0.9587 |
1.0330 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=1 |
|
-0.0021 to 0.0037 |
0.9587 |
1.0100 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=1 |
R |
-0.0027 to 0.0032 |
0.9537 |
1.0052 |
pass |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=2 |
|
0.0023 to 0.0112 |
0.9000 |
0.9388 |
fail |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=2 |
R |
0.0022 to 0.0111 |
0.8925 |
0.9205 |
fail |
Agreement with the canonical implementation#
Comparator-backed: a paired comparison with drtmle-r-stratified on the same samples. Each row reports estimate equivalence within the declared margin, the RMSE ratio and the coverage difference, and names its comparison kind.
law |
estimand |
what was compared |
paired difference |
share of margin used |
RMSE ratio bound |
coverage difference |
comparison kind |
calibration resolution |
result |
|---|---|---|---|---|---|---|---|---|---|
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=0 |
0.000339 |
0.0715 |
1.0052 |
0.0062 |
related target |
0.0121 vs 0.0500 |
superior |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=1 |
0.000652 |
0.1025 |
1.0093 |
0.0012 |
related target |
0.0118 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
average treatment effect, in stratum V=2 |
0.000169 |
0.0203 |
1.0251 |
-0.0100 |
related target |
0.0404 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=0 |
-0.000248 |
0.0608 |
1.0073 |
0.0088 |
related target |
0.0135 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=1 |
-0.000066 |
0.0141 |
1.0146 |
-0.0013 |
related target |
0.0112 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘0’, in stratum V=2 |
-0.000126 |
0.0276 |
1.0098 |
0.0012 |
related target |
0.0105 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=0 |
0.000091 |
0.0292 |
1.0050 |
-0.0012 |
related target |
0.0047 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=1 |
0.000587 |
0.1212 |
1.0089 |
0.0050 |
related target |
0.0123 vs 0.0500 |
equivalent |
Benkeser et al. (2017) binary law with three baseline strata, correct GLM nuisances |
|
counterfactual mean under treatment arm ‘1’, in stratum V=2 |
0.000043 |
0.0058 |
1.0238 |
0.0075 |
related target |
0.0277 vs 0.0500 |
equivalent |
Repeated-sampling properties#
property |
cell |
role |
what was tested |
what must hold |
measured |
result |
|---|---|---|---|---|---|---|
|
|
positive |
average treatment effect over every stratum, from the stratified fit: both the outcome regression and the treatment mechanism are correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.0013 to 0.0042, margin 0.0062, SE ratio 0.9629 |
pass |
|
|
control |
average treatment effect over every stratum, from the stratified fit: both nuisances are misspecified |
bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread |
bias 0.2197 to 0.2237, margin 0.0055, SE ratio 2.4268 |
pass |
|
|
positive |
average treatment effect over every stratum, from the stratified fit: only the outcome regression is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.000440 to 0.0048, margin 0.0059, SE ratio 1.1031 |
pass |
|
|
positive |
average treatment effect over every stratum, from the stratified fit: only the treatment mechanism is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.0136 to 0.0185, margin 0.0067, SE ratio 0.9902 |
fail |
|
|
positive |
average treatment effect in stratum V = 0: both the outcome regression and the treatment mechanism are correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias -0.0011 to 0.0029, margin 0.0087, SE ratio 0.9659 |
pass |
|
|
control |
average treatment effect in stratum V = 0: both nuisances are misspecified |
bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread |
bias 0.2267 to 0.2323, margin 0.0077, SE ratio 2.0577 |
pass |
|
|
positive |
average treatment effect in stratum V = 0: only the outcome regression is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias -0.000165 to 0.0060, margin 0.0084, SE ratio 1.0270 |
pass |
|
|
positive |
average treatment effect in stratum V = 0: only the treatment mechanism is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.0094 to 0.0162, margin 0.0093, SE ratio 0.9505 |
fail |
|
|
positive |
average treatment effect in stratum V = 1: both the outcome regression and the treatment mechanism are correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.000831 to 0.0059, margin 0.0109, SE ratio 0.9792 |
pass |
|
|
control |
average treatment effect in stratum V = 1: both nuisances are misspecified |
bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread |
bias 0.2218 to 0.2289, margin 0.0097, SE ratio 1.8668 |
pass |
|
|
positive |
average treatment effect in stratum V = 1: only the outcome regression is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias -0.0015 to 0.0061, margin 0.0104, SE ratio 1.0494 |
pass |
|
|
positive |
average treatment effect in stratum V = 1: only the treatment mechanism is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.0123 to 0.0214, margin 0.0124, SE ratio 0.9364 |
fail |
|
|
positive |
average treatment effect in stratum V = 2: both the outcome regression and the treatment mechanism are correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.0031 to 0.0097, margin 0.0143, SE ratio 0.9544 |
pass |
|
|
control |
average treatment effect in stratum V = 2: both nuisances are misspecified |
bias interval must fall entirely outside the margin, with the reported standard error still on the scale of the empirical spread |
bias 0.1924 to 0.2010, margin 0.0118, SE ratio 1.7752 |
pass |
|
|
positive |
average treatment effect in stratum V = 2: only the outcome regression is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias -0.0027 to 0.0071, margin 0.0134, SE ratio 1.2492 |
pass |
|
|
positive |
average treatment effect in stratum V = 2: only the treatment mechanism is correctly specified |
bias interval inside the equivalence margin, with the reported standard error on the scale of the empirical spread |
bias 0.0179 to 0.0286, margin 0.0147, SE ratio 1.0627 |
fail |
|
|
positive |
average treatment effect over every stratum, from the stratified fit: both nuisances are correctly specified with an independently computed efficiency bound |
SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands |
coverage 0.9355 to 0.9613, SE ratio 0.9735 to 1.0565, empirical efficiency ratio 0.9755 to 1.0587, reported efficiency ratio 1.0259 to 1.0383 |
pass |
|
|
control |
average treatment effect over every stratum, from the stratified fit: one efficiency-bound unit of independent noise is added to each estimate |
the empirical efficiency ratio must rise above the band |
coverage 0.8157 to 0.8586, SE ratio 0.6843 to 0.7457, empirical efficiency ratio 1.3832 to 1.5074, reported efficiency ratio 1.0258 to 1.0387 |
pass |
|
|
control |
average treatment effect over every stratum, from the stratified fit: the reported standard errors are multiplied by a declared factor below one |
the SE-ratio interval must fall below the calibration band |
coverage 0.8141 to 0.8572, SE ratio 0.6818 to 0.7407, empirical efficiency ratio 0.9751 to 1.0585, reported efficiency ratio 0.7181 to 0.7269 |
pass |
|
|
positive |
average treatment effect in stratum V = 0: both nuisances are correctly specified with an independently computed efficiency bound |
SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands |
coverage 0.9305 to 0.9573, SE ratio 0.9682 to 1.0534, empirical efficiency ratio 0.9672 to 1.0520, reported efficiency ratio 1.0144 to 1.0226 |
pass |
|
|
control |
average treatment effect in stratum V = 0: one efficiency-bound unit of independent noise is added to each estimate |
the empirical efficiency ratio must rise above the band |
coverage 0.8147 to 0.8577, SE ratio 0.6875 to 0.7483, empirical efficiency ratio 1.3617 to 1.4803, reported efficiency ratio 1.0145 to 1.0227 |
pass |
|
|
control |
average treatment effect in stratum V = 0: the reported standard errors are multiplied by a declared factor below one |
the SE-ratio interval must fall below the calibration band |
coverage 0.8115 to 0.8549, SE ratio 0.6766 to 0.7378, empirical efficiency ratio 0.9663 to 1.0532, reported efficiency ratio 0.7103 to 0.7160 |
pass |
|
|
positive |
average treatment effect in stratum V = 1: both nuisances are correctly specified with an independently computed efficiency bound |
SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands |
coverage 0.9266 to 0.9542, SE ratio 0.9471 to 1.0298, empirical efficiency ratio 0.9951 to 1.0817, reported efficiency ratio 1.0201 to 1.0291 |
pass |
|
|
control |
average treatment effect in stratum V = 1: one efficiency-bound unit of independent noise is added to each estimate |
the empirical efficiency ratio must rise above the band |
coverage 0.8131 to 0.8563, SE ratio 0.6884 to 0.7467, empirical efficiency ratio 1.3734 to 1.4887, reported efficiency ratio 1.0202 to 1.0291 |
pass |
|
|
control |
average treatment effect in stratum V = 1: the reported standard errors are multiplied by a declared factor below one |
the SE-ratio interval must fall below the calibration band |
coverage 0.8005 to 0.8449, SE ratio 0.6633 to 0.7202, empirical efficiency ratio 0.9966 to 1.0809, reported efficiency ratio 0.7141 to 0.7204 |
pass |
|
|
positive |
average treatment effect in stratum V = 2: both nuisances are correctly specified with an independently computed efficiency bound |
SE ratio and coverage intervals both inside their calibration bands, with both efficiency-ratio intervals inside their bands |
coverage 0.9069 to 0.9380, SE ratio 0.9294 to 1.0112, empirical efficiency ratio 1.0348 to 1.1237, reported efficiency ratio 1.0296 to 1.0653 |
fail |
|
|
control |
average treatment effect in stratum V = 2: one efficiency-bound unit of independent noise is added to each estimate |
the empirical efficiency ratio must rise above the band |
coverage 0.7984 to 0.8430, SE ratio 0.6702 to 0.7304, empirical efficiency ratio 1.4368 to 1.5556, reported efficiency ratio 1.0294 to 1.0659 |
pass |
|
|
control |
average treatment effect in stratum V = 2: the reported standard errors are multiplied by a declared factor below one |
the SE-ratio interval must fall below the calibration band |
coverage 0.7843 to 0.8302, SE ratio 0.6501 to 0.7072, empirical efficiency ratio 1.0361 to 1.1246, reported efficiency ratio 0.7207 to 0.7458 |
pass |
The property grid calibrates the marginal ATE and each stratum ATE of the both-correct fit, each with a shrunken-SE control and a noise control. The robustness family fits the same targets under four nuisance configurations. The marginal cells read the same fits as the stratum cells.
Seven results are red. The four treatment_correct robustness cells carry a positive bias of
0.34 to 0.60 of their spread. The owner X8-drtmle-one-sided-bias in the
roadmap records a declared diagnostic.
Inside each stratum, the shipped unstratified DRTMLE and R drtmle, handed the same initial
arrays, carry a positive bias too. The direction is shared in every stratum, and the size mostly is. In stratum 2 the
package’s bias, 0.0237, exceeds R’s, 0.0157, on the same arrays: the paired difference is 0.0080
with a standard error of 0.0018. The shipped unstratified fit on the stratum’s rows has the
package’s bias there to 2e-5, so the excess belongs to DRTMLE at about 400 rows and not to the
strata. This is the one-sided bias that RM18-one-sided-bias records for the unstratified study
on the same law.
The marginal estimate is the mixture of the stratum estimates, so it inherits their bias. Its bias is 0.56 of its spread at n = 2,000 (99% interval 0.43 to 0.69), and 0.43 at n = 8,000 (0.25 to 0.61). The two intervals overlap, so the diagnostic does not establish that the bias contracts faster than the spread.
The other three reds sit in stratum 2, which holds about 400 of the 2,000 rows. The two truth rows
of ey[1][V=2] cover 0.900 and 0.8925 in the two implementations on the same draws. The
v2_ate__correctly_specified cell has an SE-ratio interval that ends at 0.9294 against a floor of
0.93. The owner X8-drtmle-small-stratum records the reading. The study was declared under
reporting.
Measured values and declared margins#
quantity |
value |
source |
|---|---|---|
|
800 |
paired replications |
|
2000 |
observations per primary replication |
|
16 |
truth tests passing |
|
18 |
truth tests reported |
|
9 |
paired comparisons passing |
|
9 |
paired comparisons reported |
|
23 |
property cells passing |
|
28 |
property cells reported |
|
0.1375 |
largest primary standardized bias |
|
0.8925 |
lowest primary coverage |
|
0.1212 |
largest paired similarity-margin share |
|
0.9900 |
Monte Carlo confidence level |
|
0.0500 |
nominal test size |
|
0.9500 |
nominal interval coverage |
|
10000 |
bootstrap replications |
|
0.2500 |
standardized-bias margin |
|
0.1000 |
union-model SE-ratio screen, lower limit |
|
10 |
union-model SE-ratio screen, upper limit |
|
0.9000 |
primary coverage floor |
|
0.9900 |
descriptive overcoverage threshold |
|
0.8000 |
primary SE-ratio lower screen |
|
1.2000 |
primary SE-ratio upper screen |
|
0.9300 |
calibration SE-ratio lower bound |
|
1.0700 |
calibration SE-ratio upper bound |
|
0.9200 |
calibration coverage lower bound |
|
0.9800 |
calibration coverage upper bound |
|
0.1000 |
type-I upper bound |
|
0.1500 |
paired similarity margin in pooled SDs |
|
1.1000 |
RMSE-ratio noninferiority bound |
|
-0.0250 |
coverage-difference noninferiority bound |
|
0.0500 |
calibration-excess noninferiority bound |
|
0.8000 |
minimum power lower bound |
|
-0.5000 |
expected root-n slope |
|
-0.6250 |
root-n slope lower bound |
|
-0.3750 |
root-n slope upper bound |
|
-0.2500 |
slower rate the interval must exclude |
|
0.9000 |
efficiency-ratio lower bound |
|
1.1000 |
efficiency-ratio upper bound |
|
0.7000 |
negative-control SE multiplier |
|
0.0233 |
efficiency-bound standard error by quadrature at primary n |
|
0.0330 |
efficiency-bound standard error by quadrature at primary n |
|
0.0417 |
efficiency-bound standard error by quadrature at primary n |
|
0.0526 |
efficiency-bound standard error by quadrature at primary n |
Limits#
The study covers one law with one three-level stratum and a complete binary outcome.
The study fits only the univariate reduction at
guard=("Q", "g"). The exact-law tests intests/unit/test_stratified_drtmle_exact.pycover the other guards, the bivariate reduction, three arms, and the missing-outcome and composite routes.The marginal estimate has no paired comparator.
The law keeps the symmetry of the paper law in each stratum, so it is H1 in both branches. The one-correct cells therefore measure second-order terms only.
tests/unit/test_drtmle_h1_symmetry.pypins it for the unstratified law, and the argument is the same per stratum, and the theorem states why it matters.
Reproduction#
The fixture README, manifest, replications, paired decisions, and property results carry the protocol, provenance, and every published row.