mean over the 3 references: reference gold 1.997;  oracle at beta = 2.0: gold 3.144, KL 0.298, value 2.548

offline (mean of 3 seeds)
  step  E[gold]      KL  dlogpi(winners)  dlogpi(losers)  gold - beta KL
     0    1.997   0.000            0.000           0.000           1.997
    40    3.091   0.283            0.031          -0.553           2.525
    80    3.148   0.308            0.027          -0.592           2.531
   120    3.210   0.347            0.014          -0.643           2.515
   160    3.288   0.408           -0.006          -0.727           2.472
   200    3.418   0.521           -0.045          -0.884           2.376
   240    3.663   0.769           -0.146          -1.198           2.125
   280    4.037   1.227           -0.387          -1.755           1.583
   320    4.467   1.863           -0.790          -2.524           0.740
   360    4.956   2.674           -1.376          -3.515          -0.393
   400    5.367   3.473           -2.082          -4.619          -1.579

online (mean of 3 seeds)
  step  E[gold]      KL  dlogpi(winners)  dlogpi(losers)  gold - beta KL
     0    1.997   0.000            0.000           0.000           1.997
    40    2.829   0.241            0.520          -0.053           2.346
    80    3.045   0.318            0.704           0.063           2.409
   120    3.115   0.334            0.595          -0.048           2.447
   160    3.022   0.299            0.641           0.028           2.424
   200    3.171   0.355            0.637          -0.029           2.462
   240    3.157   0.368            0.768           0.015           2.421
   280    3.126   0.343            0.707           0.012           2.439
   320    3.171   0.365            0.696          -0.024           2.441
   360    3.109   0.327            0.707          -0.023           2.456
   400    3.121   0.351            0.683          -0.037           2.420

