(a) coverage of nominal 95% intervals for pass@1 (true value 0.5), N prompts, n samples each; 1000 benchmarks
    N   n  naive width  naive coverage  bootstrap width  bootstrap coverage
   50   1        0.274           0.936            0.270               0.958
   50   8        0.098           0.736            0.167               0.934
   50  64        0.035           0.376            0.150               0.952
  200   1        0.138           0.950            0.136               0.957
  200   8        0.049           0.720            0.084               0.940
  200  64        0.017           0.331            0.075               0.943

(b) coverage of the prompt-level bootstrap for pass@k with the unbiased estimator (N = 100 prompts, n = 32 samples, 1000 benchmarks)
   k  true pass@k  mean estimate  plug-in mean  coverage
   1       0.5000         0.5002        0.5002     0.946
   4       0.8170         0.8172        0.8073     0.938
  16       0.9558         0.9555        0.9424     0.897

(c) two models with correlated difficulty: B = A shifted by 0.15 on the logit scale (accuracy gap about 0.026), one sample per prompt
  true gap 0.0264; SD of one paired difference 0.594; SD for unpaired (independent prompts) 0.707
     N  power paired (sim)  (formula)  power unpaired z (sim)  (formula)
   500               0.175      0.169                   0.131      0.133
  1000               0.290      0.291                   0.235      0.220
  2000               0.497      0.513                   0.376      0.388
  4000               0.797      0.805                   0.651      0.658
  N required for 80% power: paired 3954, unpaired 5603

(d) false-positive rate when the two methods are equal: seed SD 0.02, N = 500 prompts, 1 sample per prompt, 4000 experiments
  single run each, paired test over prompts: 0.180     five seeds each, Welch t-test over seeds: 0.044

(e) 20 comparisons, 4 with a real effect (z-statistic mean 3.0), 16 null; 5000 repetitions; alpha = 0.05
procedure             P(any false positive)  false discovery rate  power (true effects found)
uncorrected                           0.558                 0.162                       0.855
Holm                                  0.043                 0.016                       0.505
Benjamini-Hochberg                    0.134                 0.040                       0.629
