optimal policy under the KL-regularized objective: mean length / expected gold utility u(l) (content term excluded)
  gamma            beta=1.0            beta=0.5           beta=0.25
    0.0       20.00 / -0.36         20.00 / -0.21         20.00 / -0.11  
   0.05       20.90 / -0.38         21.05 / -0.23         21.14 / -0.14  
    0.1       21.80 / -0.42         22.09 / -0.30         22.28 / -0.22  
    0.2       23.60 / -0.62         24.18 / -0.56         24.56 / -0.53  
    0.4       27.19 / -1.39         28.37 / -1.61         29.11 / -1.77  
(base policy: mean length 20.16, utility -1.22)

estimating gamma from n audited responses (true gamma = 0.2, 400 repetitions), then correcting the proxy; beta = 0.5
     n  gamma_hat mean  gamma_hat sd  mean length (corrected)     sd   [uncorrected 24.18; target 20.00]
    20          0.1993        0.0164                    20.02   0.34
    50          0.1995        0.0092                    20.01   0.19
   200          0.2000        0.0047                    20.00   0.10
  1000          0.1999        0.0021                    20.00   0.04

the same with noisy gold labels (noise sd 1.0 added to r), n = 200
gamma_hat 0.2005 +- 0.0100; mean length 19.99 +- 0.21
