reference: mean gold reward 1.724;  maximum gold reward 8.0
 beta  |pi_DPO - pi*|_inf  E_pi*[R]  E_DPO[R]  KL(pi*||ref)  KL(DPO||ref)  implicit reward error
  0.5            9.67e-10     7.653     7.653         4.902         4.902               2.43e-08
  2.0            1.17e-08     3.587     3.587         0.485         0.485               4.65e-07
  8.0            2.61e-10     2.139     2.139         0.026         0.026               1.26e-07
