Ex 4.5: a constant critic offset c changes A_t by -c * lambda^(T-t) (terminal reward, gamma = 1)
 lambda    max |A(V+c) - A(V) + c lam^(T-t)|
    0.0                             4.44e-16
    0.5                             5.00e-16
    0.9                             9.44e-16
    1.0                             8.88e-16

Ex 4.7: batch-standardized advantages on a 4-armed bandit with Bernoulli rewards, B = 8
exact gradient        : [-0.0977  0.1403 -0.0442  0.0015]
 leave-one-out: [-0.0976  0.1402 -0.0442  0.0016]  cosine 1.0000  norm ratio 0.999
 mean-centered: [-0.0854  0.1227 -0.0387  0.0014]  cosine 1.0000  norm ratio 0.874
  standardized: [-0.1823  0.2619 -0.0826  0.003 ]  cosine 1.0000  norm ratio 1.867
fraction of batches with all rewards equal (zero standardized advantage): 0.013
