Ex 5.2: the gradient of the surrogate at theta_old equals the policy gradient
  max |grad L - grad J| = 2.22e-16   (|grad J| = 0.1149)
  value at the old policy: L = 3.71e-16

Ex 5.3: sum_{t=1}^T (t-1)(T-t+1) = T(T^2-1)/6
  T=  2:        1  formula        1
  T=  6:       35  formula       35
  T= 10:      165  formula      165
  T=100:   166650  formula   166650

Ex 5.5: natural-gradient steps are invariant to a linear reparametrization of a 3-armed softmax policy
  natural steps, policy after the step: max difference 1.11e-16
  plain gradient steps (same quadratic KL): max difference 3.81e-03

Ex 5.6: predicted improvement of the natural step, sqrt(2 delta g^T F^+ g)
  delta=   0.1: sqrt(2 delta g^T F^+ g) = 0.11150
  delta=  0.01: sqrt(2 delta g^T F^+ g) = 0.03526
  delta= 0.001: sqrt(2 delta g^T F^+ g) = 0.01115
