number of trajectories: 648;  probabilities sum to 1.000000000000
J(theta) = -1.582396

form                    max |E[est] - grad J|   total variance
(a) trajectory                       2.78e-16           5.6285
(b) causal                           3.89e-16           3.1179
(c) Q-function                       2.22e-16           1.4280
(d) value baseline                   3.89e-16           3.5517

exact gradient (flattened theta[s, a]): [-0.03472  0.03472 -0.34043  0.34043  0.38173 -0.38173]
