19.1 the unbiased pass@k estimator is unbiased, the plug-in is biased downward (n = 10 samples, p = 0.3, k = 5)
  true 0.83193; exact expectation of the unbiased estimator 0.83193; simulated 0.83205; plug-in simulated 0.75986

19.2 design effect of prompt clustering: Var(mean of N*n samples) = Var(p)/N + E[p(1-p)]/(N n)
  N=200 n=1: true SD 0.0354; naive SD (iid samples) 0.0354; ratio 1.00
  N=200 n=8: true SD 0.0219; naive SD (iid samples) 0.0125; ratio 1.75
  N=200 n=64: true SD 0.0195; naive SD (iid samples) 0.0044; ratio 4.42
  simulated SD at N=200, n=8: 0.0220

19.3 sample size for a paired comparison: N = ((z_{1-alpha/2} + z_power) sd / delta)^2
  delta=0.0264 sd=0.594: N = 3974, power at N = 0.800
  delta=0.0264 sd=0.707: N = 5630, power at N = 0.800
  delta=0.05 sd=0.5: N = 785, power at N = 0.800
  delta=0.01 sd=0.5: N = 19623, power at N = 0.800

19.4 family-wise error: 1 - (1 - alpha)^m for independent null tests; Holm controls it
  m=  1: uncorrected FWER 0.050
  m=  5: uncorrected FWER 0.226
  m= 20: uncorrected FWER 0.642
  m=100: uncorrected FWER 0.994
  simulated Holm FWER for m = 20 null tests: 0.0488

19.5 probability that a random n-gram of a clean item occurs in a corpus of T n-grams drawn from a Zipf(1.1) vocabulary
  n=1: chance of matching a fixed corpus n-gram 4.013e-02; expected matches in a corpus of 3,000,000 n-grams 1.2e+05
  n=2: chance of matching a fixed corpus n-gram 1.611e-03; expected matches in a corpus of 3,000,000 n-grams 4.83e+03
  n=3: chance of matching a fixed corpus n-gram 6.464e-05; expected matches in a corpus of 3,000,000 n-grams 194
  n=5: chance of matching a fixed corpus n-gram 1.041e-07; expected matches in a corpus of 3,000,000 n-grams 0.312

19.6 position bias cancels in the average over both orders only in the sign, not in the level
  D= -1.0: fixed-order 0.310  both orders 0.239  unbiased judge 0.231
  D= -0.2: fixed-order 0.540  both orders 0.443  unbiased judge 0.440
  D=  0.0: fixed-order 0.599  both orders 0.500  unbiased judge 0.500
  D=  0.2: fixed-order 0.655  both orders 0.557  unbiased judge 0.560
  D=  1.0: fixed-order 0.832  both orders 0.761  unbiased judge 0.769

19.7 false-positive rate of a single-run comparison when seed variance is ignored
  sampling SD of the difference 0.0266; seed SD of the difference 0.0283; inflation r = 1.461; predicted false-positive rate 0.180
