reference: gold 1.724;  oracle tilted policy (beta = 2.0): gold 3.587, KL 0.485

regularized value E[gold] - beta KL: oracle 2.616
      n |  DPO gold      KL   value | two-stage gold      KL   value
    100 |     3.946   3.317  -2.688 |          2.054   0.037   1.979
   1000 |     5.091   2.075   0.941 |          2.748   0.169   2.409
  10000 |     3.946   0.739   2.468 |          3.388   0.395   2.599
 100000 |     3.572   0.478   2.615 |          3.536   0.461   2.615
