1281 pairs; 64 distinct responses appear

   method   step  KL(pi||ref)   mean D  mean dlog pi(winner)  E_pi[gold]
      DPO    100        0.703     2.37                 0.673       3.927
      DPO    500        1.533     3.71                 0.429       5.065
      DPO   2000        2.636     5.12                -0.301       6.147
      DPO   6000        4.050     6.75                -1.668       7.174
  squared    100        0.009     0.23                 0.137       1.954
  squared    500        0.009     0.23                 0.137       1.954
  squared   2000        0.009     0.23                 0.137       1.954
  squared   6000        0.009     0.23                 0.137       1.954
 ref-free    100        1.496     4.36                 0.636       4.824
 ref-free    500        3.492     8.35                -1.654       6.767
 ref-free   2000        5.565    12.62                -6.581       7.946
 ref-free   6000        5.727    17.50               -12.631       7.999
