base policy (uniform): mean gold 2.000;  maximum gold 8.000

best-of-N: expected gold reward (mean over reward-model seeds)
     N     KL      n=50     n=500    n=5000    oracle
     1   0.00     2.000     2.000     2.000     2.000
     2   0.19     2.382     2.596     2.756     2.769
     4   0.64     2.754     3.167     3.445     3.471
     8   1.20     3.091     3.694     4.050     4.093
    16   1.84     3.383     4.175     4.581     4.644
    32   2.50     3.648     4.604     5.048     5.133
    64   3.17     3.899     4.968     5.475     5.582
   128   3.86     4.135     5.262     5.853     5.962
   256   4.55     4.357     5.503     6.202     6.342
   512   5.24     4.543     5.685     6.545     6.700
  1024   5.93     4.658     5.787     6.863     6.950
  2048   6.61     4.689     5.760     7.171     7.191
  4096   7.28     4.651     5.505     7.508     7.510

KL-regularized tilting: expected gold reward at fixed KL (mean over seeds); peak over the whole curve
    KL      n=50     n=500    n=5000    oracle
   0.5     2.723     3.161     3.424     3.503
   1.0     3.045     3.684     4.045     4.159
   2.0     3.520     4.429     4.927     5.099
   3.0     3.911     4.963     5.598     5.813
   4.0     4.238     5.351     6.159     6.401
   5.0     4.481     5.603     6.658     6.894
   6.0     4.637     5.684     7.117     7.314
  peak     4.845     5.712     8.000     8.000
KL@peak      6.21      5.98      8.32      8.32
   end     4.493     4.889     8.000     8.000
