Ex 8.1: the DPO loss depends on the policy only through log pi - log ref, so a constant added to all logits changes nothing
  loss(theta) - loss(theta + 5.3) = -3.3e-16

Ex 8.2: gradient of the DPO loss = -beta E[ sigma(-beta D) (grad log pi(y_w) - grad log pi(y_l)) ]
  max |autograd - formula| = 1.4e-17

Ex 8.4: on one pair, the DPO loss keeps decreasing as pi(y_l) -> 0; the squared loss has a finite minimizer D = 1/(2 beta)
   log pi(y_l)        D   DPO loss  squared loss
         -1.39     0.00    0.69315         0.062
         -1.58     0.25    0.47408         0.000
         -3.14     2.00    0.01815         3.062
         -5.10     4.00    0.00034        14.062
         -9.10     8.00    0.00000        60.062
        -17.10    16.00    0.00000       248.062

Ex 8.5: likelihood displacement with shared parameters (logits = theta . phi(y))
   step   log pi(w) - log ref(w)   log pi(l) - log ref(l)       D
      0                   0.0000                   0.0000   0.000
     10                  -0.1494                  -0.1692   0.020
     50                  -0.7529                  -0.8482   0.095
    100                  -1.4866                  -1.6682   0.182
    200                  -2.8067                  -3.1379   0.331
  mass of the 10 other responses after training: 0.9915 (reference 10/12 = 0.8333)
