reference policy: mean reward 0.167
  beta | trained: reward      KL   value | optimum: reward      KL   value
  0.01 |           0.979  14.311   0.836 |           1.000  14.392   0.856
  0.03 |           0.932  11.742   0.580 |           0.927  11.337   0.587
   0.1 |           0.398   1.287   0.270 |           0.410   1.326   0.278
   0.3 |           0.229   0.124   0.191 |           0.233   0.114   0.199
