Ex 3.6: exact variances by enumerating all 2^T action sequences
  T       full  closed form  reward-to-go  with baseline
  1     0.1029       0.1029        0.1029         0.0336
  2     0.5964       0.5964        0.3570         0.1113
  4     3.2088       3.2088        1.4700         0.3990
  8    18.1104      18.1104        7.1736         1.5036
 12    51.9624      51.9624       19.5300         3.3138

Ex 3.8: exact gradient            : [-0.0463  0.005   0.0217  0.0196]
        state baseline b = J      : [-0.0462  0.005   0.0216  0.0196]
        action baseline b(a) = mu_a: [ 0.0001 -0.     -0.0001 -0.    ]  (should be ~0)
        predicted bias -sum_a pi_a b(a) score_a: [ 0.0463 -0.005  -0.0217 -0.0196]  = estimate minus exact: [ 0.0464 -0.005  -0.0217 -0.0196]

Ex 3.10 (B = 4): exact gradient          : [-0.0463  0.005   0.0217  0.0196]
        leave-one-out baseline        : [-0.0462  0.005   0.0216  0.0197]
        batch mean incl. own sample   : [-0.0347  0.0037  0.0162  0.0148]
        ratio inclusive / exact       : [0.749 0.739 0.747 0.753]  (predicted 0.75)
