15.1 exact policy gradient vs score-function expectation (2 turns, 2 actions, 3 observations)
  max |finite-difference - score expectation| = 2.638648327613069e-11

15.2 outcome-estimator variance of the sum of two Bernoulli-score terms, enumerated
  mean 0.2500 (= q = 0.2500), variance 0.0625

15.3 GAE identities
  lambda=1  max |A - (G - V)| = 8.881784197001252e-16
  lambda=0  max |A - delta|   = 0.0

15.4 hindsight relabeling is unbiased with importance weights (exact enumeration)
  max |grad J(g') - E[w * score]| = 9.742952972180419e-12   (grad norm 0.07455692369139455 )

15.5 turn-level score is the sum of token scores
  max |grad log pi(turn) - sum of token scores| = 4.556059973737092e-10

15.6 E_{pi_g}[w^2] for per-step policies that differ by a fixed KL, versus horizon
  per-step E[ratio^2] = 1.0408;  KL(pa||pb)=0.0207
  H=  1  Var(w) =     0.0408
  H=  5  Var(w) =     0.2215
  H= 10  Var(w) =     0.4921
  H= 20  Var(w) =     1.2265
  H= 40  Var(w) =     3.9573
  H= 80  Var(w) =    23.5751
