Default simultaneous bands#
This study measures the default simultaneous band of 25 shipped fit shapes. TMLE and LTMLE
default to simultaneous=True. A fit that reports two or more inferential estimates then
publishes a max-t multiplier band over all of them. The
simultaneous bands section of the inference reference
defines the band and lists the other registered studies that measure it.
Each shape is the subject fit of one registered source study. This study refits it with
simultaneous=True and changes nothing else. The law, the sampler, the truth oracle, the sample
size and the fit function are the source study’s own. The samples are this study’s own: it draws
each replication through the source study’s sampler at a seed this study owns.
No canonical implementation is compared. The pinned comparators publish no multiplier band over these families. The zero-row equivalence artifact records the absence of a comparator.
What was tested#
setting |
declaration |
|---|---|
band |
the shipped default: 1000 Rademacher draws on the point path and 2000 on the sequential path, seeded by the subject fit’s |
band cell |
|
control cell |
|
budget |
2,400 replications for each shape, and 4,000 for |
primary scenario |
|
Monte Carlo inference |
exact 99% intervals around every coverage rate, and 99% intervals around every primary endpoint |
The table gives each shape, its source study and its size.
shape |
source study |
what the shape adds |
n |
|---|---|---|---|
|
this study’s primary scenario |
the shipped |
1,000 |
|
ten estimands, with log-scale ratio bands and the PAF |
1,000 |
|
|
a cross-validated variance beside centered multiplier draws |
1,000 |
|
|
fixed observation weights and ratios |
2,000 |
|
|
weighted nuisances and a linear fluctuation |
2,000 |
|
|
a missing outcome |
2,000 |
|
|
DR-TMLE with a missing outcome |
2,000 |
|
|
cross-fitted DR-TMLE corrected curves |
3,000 |
|
|
nine multi-arm parameters with ratios |
2,000 |
|
|
one band for each intermediate level |
2,000 |
|
|
three MSM projection terms |
2,000 |
|
|
grouped folds and multipliers drawn by cluster |
2,000 |
|
|
three shifts and two contrasts |
2,000 |
|
|
three odds multipliers and two contrasts |
2,000 |
|
|
a static and a stochastic regime |
2,000 |
|
|
a static regime and a dynamic rule |
2,000 |
|
|
five cross-fitted regimen parameters |
2,000 |
|
|
ten parameters with two exact duplicate pairs |
2,000 |
|
|
twenty parameters with four exact duplicates |
4,000 |
|
|
observation weights |
2,000 |
|
|
observation weights, cross-fitted |
2,000 |
|
|
five categorical regimens |
2,000 |
|
|
the same, cross-fitted |
2,000 |
|
|
two MSM projection terms |
2,500 |
A duplicate is a parameter that equals another one exactly, because the two plans agree at node one. The band covers it twice. The truth of a duplicate is the truth of its twin.
Accuracy against known truth#
law |
estimand |
what was tested |
implementation |
bias (99% interval) |
coverage |
SE ratio |
result |
|---|---|---|---|---|---|---|---|
binary-outcome law, fitted with the shipped TMLE() defaults |
|
average effect on the untreated |
|
-0.0032 to 0.0020 |
0.9610 |
0.9813 |
pass |
binary-outcome law, fitted with the shipped TMLE() defaults |
|
average treatment effect |
|
-0.0031 to 0.0019 |
0.9600 |
0.9812 |
pass |
binary-outcome law, fitted with the shipped TMLE() defaults |
|
average effect on the treated |
|
-0.0032 to 0.0020 |
0.9590 |
0.9809 |
pass |
binary-outcome law, fitted with the shipped TMLE() defaults |
|
counterfactual mean under no treatment |
|
-0.0020 to 0.0018 |
0.9400 |
0.9564 |
pass |
binary-outcome law, fitted with the shipped TMLE() defaults |
|
counterfactual mean under treatment |
|
-0.0026 to 0.0011 |
0.9380 |
0.9916 |
pass |
binary-outcome law, fitted with the shipped TMLE() defaults |
|
marginal odds ratio, reported on the log scale |
|
-0.0111 to 0.0098 |
0.9580 |
0.9807 |
pass |
binary-outcome law, fitted with the shipped TMLE() defaults |
|
marginal risk ratio, reported on the log scale |
|
-0.0056 to 0.0059 |
0.9520 |
0.9761 |
pass |
Theory properties#
property |
cell |
role |
what was tested |
what must hold |
measured |
result |
|---|---|---|---|---|---|---|
|
|
control |
five categorical regimen means and four contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.6579 to 0.7072 |
pass |
|
|
positive |
five categorical regimen means and four contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9101 to 0.9382 |
fail |
|
|
control |
five cross-fitted categorical regimen means and four contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.6864 to 0.7344 |
pass |
|
|
positive |
five cross-fitted categorical regimen means and four contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9173 to 0.9443 |
fail |
|
|
control |
the controlled direct-effect fit at intermediate level zero: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8260 to 0.8643 |
pass |
|
|
positive |
the controlled direct-effect fit at intermediate level zero: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9015 to 0.9309 |
fail |
|
|
control |
the controlled direct-effect fit at intermediate level one: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8452 to 0.8817 |
pass |
|
|
positive |
the controlled direct-effect fit at intermediate level one: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9228 to 0.9488 |
pass |
|
|
control |
grouped cross-fitted TMLE with cluster multipliers: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8611 to 0.8957 |
pass |
|
|
positive |
grouped cross-fitted TMLE with cluster multipliers: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9241 to 0.9499 |
pass |
|
|
control |
cross-fitted cumulative incidence of two causes, three plans, two horizons: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.6063 to 0.6574 |
pass |
|
|
positive |
cross-fitted cumulative incidence of two causes, three plans, two horizons: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9232 to 0.9492 |
pass |
|
|
control |
the shipped TMLE() defaults on the binary law: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8408 to 0.8777 |
pass |
|
|
positive |
the shipped TMLE() defaults on the binary law: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9365 to 0.9600 |
pass |
|
|
control |
a static regime and a dynamic rule and their contrast: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8523 to 0.8879 |
pass |
|
|
positive |
a static regime and a dynamic rule and their contrast: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9301 to 0.9548 |
pass |
|
|
control |
cross-fitted DR-TMLE over the arm means and the ATE: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8743 to 0.9074 |
pass |
|
|
positive |
cross-fitted DR-TMLE over the arm means and the ATE: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9467 to 0.9681 |
pass |
|
|
control |
fold-evaluated CV-TMLE over five estimands: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8461 to 0.8824 |
pass |
|
|
positive |
fold-evaluated CV-TMLE over five estimands: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9411 to 0.9637 |
pass |
|
|
control |
three incremental odds multipliers and their two contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8518 to 0.8875 |
pass |
|
|
positive |
three incremental odds multipliers and their two contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9351 to 0.9589 |
pass |
|
|
control |
weighted point-treatment TMLE with learned nuisances: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8730 to 0.9063 |
pass |
|
|
positive |
weighted point-treatment TMLE with learned nuisances: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9443 to 0.9663 |
pass |
|
|
control |
the two terms of the longitudinal MSM projection: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.9038 to 0.9267 |
pass |
|
|
positive |
the two terms of the longitudinal MSM projection: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9337 to 0.9527 |
pass |
|
|
control |
cross-fitted end-of-study regimen means and contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8007 to 0.8413 |
pass |
|
|
positive |
cross-fitted end-of-study regimen means and contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9260 to 0.9514 |
pass |
|
|
control |
missing-outcome TMLE over the arm means and the ATE: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8553 to 0.8907 |
pass |
|
|
positive |
missing-outcome TMLE over the arm means and the ATE: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9232 to 0.9492 |
pass |
|
|
control |
missing-outcome DR-TMLE over the arm means and the ATE: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8672 to 0.9012 |
pass |
|
|
positive |
missing-outcome DR-TMLE over the arm means and the ATE: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9328 to 0.9570 |
pass |
|
|
control |
multi-arm DR-TMLE over nine parameters: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.7860 to 0.8278 |
pass |
|
|
positive |
multi-arm DR-TMLE over nine parameters: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9273 to 0.9526 |
pass |
|
|
control |
ordinary in-sample TMLE over ten binary-outcome estimands: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8181 to 0.8572 |
pass |
|
|
positive |
ordinary in-sample TMLE over ten binary-outcome estimands: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9383 to 0.9615 |
pass |
|
|
control |
the three terms of the point-treatment MSM projection: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8646 to 0.8989 |
pass |
|
|
positive |
the three terms of the point-treatment MSM projection: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9278 to 0.9529 |
pass |
|
|
control |
three shift policies and their two contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8316 to 0.8695 |
pass |
|
|
positive |
three shift policies and their two contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9346 to 0.9585 |
pass |
|
|
control |
a static and a known stochastic regime and their contrast: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8642 to 0.8985 |
pass |
|
|
positive |
a static and a known stochastic regime and their contrast: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9388 to 0.9619 |
pass |
|
|
control |
the cross-fitted survival curve of three plans at two horizons: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.7407 to 0.7857 |
pass |
|
|
positive |
the cross-fitted survival curve of three plans at two horizons: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9223 to 0.9484 |
pass |
|
|
control |
weighted point-treatment TMLE over five estimands: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8545 to 0.8899 |
pass |
|
|
positive |
weighted point-treatment TMLE over five estimands: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9328 to 0.9570 |
pass |
|
|
control |
weighted end-of-study regimen means and contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8037 to 0.8441 |
pass |
|
|
positive |
weighted end-of-study regimen means and contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9278 to 0.9529 |
pass |
|
|
control |
cross-fitted weighted end-of-study regimen means and contrasts: the pointwise 95% intervals of the same fits, read jointly |
joint coverage upper endpoint must fall below the nominal rate |
joint coverage 0.8150 to 0.8544 |
pass |
|
|
positive |
cross-fitted weighted end-of-study regimen means and contrasts: the max-t multiplier band over every estimand the law reports, from the same-row centered influence curves |
joint coverage interval inside the calibration coverage band |
joint coverage 0.9269 to 0.9522 |
pass |
Result#
Every primary test passed, and every pointwise control fell below 0.95, so each band’s critical
value does work. The bands of 22 of the 25 shapes kept their joint-coverage interval inside
\([0.92, 0.98]\). The study publishes under the reporting policy, and three bands are red.
band-finite-sample owns them in the
red-cell ledger.
shape |
band coverage |
package critical value |
oracle band coverage |
source pointwise calibration |
|---|---|---|---|---|
|
0.9250 |
2.680 |
0.9267 at 2.689 |
static contrast 0.939 |
|
0.9317 |
2.680 |
0.9325 at 2.689 |
static contrast 0.946 |
|
0.9171 |
2.312 |
0.9200 at 2.331 |
0.942 at Z = 0 |
The declared diagnostic read each red cell’s rows again at the design critical value, which
tests/unit/test_simultaneous_cell_design.py computes. The oracle band also covers below the band
edge or at it, so the multiplier is not the main cause. The package critical value averages a
little below the design one, and that gap accounts for at most 0.0029 of each coverage. Each
source study’s pointwise calibration of the same parameters sits near the lower edge of its own
band. The reading is finite-sample, so the default band stays on.
tests/unit/test_band_shortfall_reading.py rebuilds
every number in the table from the committed rows.
The design critical value of a shape comes from the package’s own influence curves, averaged over the design fits. It is independent of the multiplier draws but not of the curves, so it cannot detect a defect in a curve. The source studies’ pointwise calibration cells are the evidence for the curves.
The first declared run stopped before any joint verdict. One sample of the multi_arm_drtmle
shape activated the propensity bound, and the source study’s fit raises on that, because that
study publishes bound_active as false for every replication. The band does not depend on that
claim, so this study turns the check off and keeps the replication as drawn. A one-off refit
of all 2,400 replications found one with an active bound, replication 1862, where one
propensity entry reached the bound. A second run ran out
of memory at 16 workers, and the published run used 8.
Measured values#
Names beginning margin: are thresholds declared before the run. Everything else is measured
from the committed results and checked at the precision printed.
quantity |
value |
source |
|---|---|---|
|
1000 |
primary replications |
|
1000 |
observations per primary replication |
|
7 |
primary tests against truth |
|
7 |
of those, passing |
|
0 |
external comparisons declared |
|
0 |
external comparisons passing |
|
50 |
repeated-sampling property cells |
|
47 |
cells whose own and family verdicts pass |
|
0.0320 |
largest absolute primary bias in empirical standard deviations |
|
0.9380 |
lowest measured primary-study coverage |
|
0.9081 |
lowest bootstrap primary SE-ratio endpoint |
|
1.0569 |
highest bootstrap primary SE-ratio endpoint |
|
0.9492 |
joint coverage of the band of the shipped TMLE() default |
|
0.8600 |
joint coverage of its pointwise intervals |
|
0.9250 |
joint coverage of the categorical LTMLE band |
|
0.9101 |
its 99% lower endpoint |
|
0.9317 |
joint coverage of the cross-fitted categorical LTMLE band |
|
0.9173 |
its 99% lower endpoint |
|
0.9171 |
joint coverage of the controlled direct-effect band at Z = 0 |
|
0.9015 |
its 99% lower endpoint |
|
0.9583 |
the highest band coverage, cross-fitted DR-TMLE |
|
0.9157 |
the highest control coverage, two longitudinal MSM projection terms |
|
0.9267 |
its 99% upper endpoint |
|
0.9900 |
confidence level of every Monte Carlo interval |
|
0.0500 |
nominal size of the reported intervals |
|
0.9500 |
nominal coverage those intervals claim |
|
10000 |
resamples behind every bootstrap interval |
|
0.2500 |
bias equivalence margin, in empirical standard deviations |
|
0.9000 |
validity floor the coverage lower endpoint must clear |
|
0.9900 |
above this, coverage is conservative rather than invalid |
|
0.8000 |
SE-ratio screen, lower limit |
|
1.2000 |
SE-ratio screen, upper limit |
|
0.9300 |
calibration-cell SE-ratio band, lower limit |
|
1.0700 |
calibration-cell SE-ratio band, upper limit |
|
0.9200 |
calibration and joint-coverage band, lower limit |
|
0.9800 |
calibration and joint-coverage band, upper limit |
|
0.1000 |
largest supported type-I rate. No cell of this study reads it |
|
0.1500 |
paired similarity margin, in pooled empirical standard deviations |
|
1.1000 |
largest external-comparison RMSE ratio bound |
|
-0.0250 |
smallest external-comparison coverage difference bound |
|
0.0500 |
largest external-comparison calibration excess bound |
|
0.8000 |
rejection lower bound a power cell must clear. No cell of this study reads it |
|
-0.5000 |
contraction rate root-n asymptotics predict. No cell of this study reads it |
|
-0.6250 |
accepted root-n slope band, lower limit |
|
-0.3750 |
accepted root-n slope band, upper limit |
|
-0.2500 |
slower rate a root-n interval must exclude |
Limitations#
limitation |
what it means for use |
|---|---|
Each shape is measured at one law and one size |
The band of another law, size or learner is not measured. Each source study page states the law |
The band uses one multiplier seed for every replication |
The shipped band seeds its draws with the fit’s |
No comparator |
The pinned comparators publish no multiplier band over these families |
The design correlations are estimates |
The control’s power comes from the curve covariance averaged over ten fits at each shape’s size, or three for the four slowest shapes. The design test holds the power at the joint coverage plus 0.005 |
Reproduction#
The fixture README gives the regeneration command. The manifest records the seeds, margins, shapes, source hashes, and result hashes.