Ex 6.2: at ratio 1 the gradient of the clipped objective equals the policy-gradient surrogate gradient
  |clipped - score-function surrogate| = 5.3e-14; sampled gradient vs exact: 5.6e-05

Ex 6.3: k3 = rho - 1 - log rho is nonnegative; its minimum over a grid of rho > 0
  min k3 = 0.00e+00 at rho = 1.0000;  k1 = -log rho takes negative values: min k1 = -9.21

Ex 6.6: the maximizer of E_pi[R] - beta KL(pi||ref) is ref * exp(R/beta) / Z
  max |LBFGS - closed form| = 3.6e-09;  value 0.495312 = beta log Z = 0.495312

Ex 6.7: E_old[ratio] = 1, and Var_old[ratio] = chi-squared divergence
  perturbation 0.1: exact mean 1.000000, sample mean 1.0000; sd 0.0636 vs sqrt(chi2) 0.0637; max ratio 1.06
  perturbation 0.5: exact mean 1.000000, sample mean 0.9999; sd 0.2797 vs sqrt(chi2) 0.2797; max ratio 1.28
  perturbation 1.0: exact mean 1.000000, sample mean 1.0000; sd 0.4872 vs sqrt(chi2) 0.4872; max ratio 1.53

Ex 6.8: gradient of the entropy of a softmax policy, -pi_k (log pi_k + H)
  max |autograd - formula| = 5.6e-17
