=== structured task: the correct answer is the argmax of a hidden linear score of the features ===
mean policy entropy (nats): before 4.719, after 0.655; log 400 = 5.991
train prompts, by initial success probability (averages over 10 instances)
    bin of p_x  prompts/instance  pass@1 before   after  pass@64 before   after   # with pass@64 drop > 0.05
    [0, 0.002)  (no prompts)
  [0.002, 0.02)              10.8          0.016   0.560           0.630   1.000                         0.00
   [0.02, 0.2)             130.1          0.072   0.788           0.951   1.000                         0.00
      [0.2, 1)               9.1          0.285   0.973           1.000   1.000                         0.00
test prompts, by initial success probability (averages over 10 instances)
    bin of p_x  prompts/instance  pass@1 before   after  pass@64 before   after   # with pass@64 drop > 0.05
    [0, 0.002)  (no prompts)
  [0.002, 0.02)               9.1          0.015   0.529           0.611   1.000                         0.00
   [0.02, 0.2)             131.5          0.071   0.742           0.952   1.000                         0.00
      [0.2, 1)              10.4          0.292   0.957           1.000   1.000                         0.00

=== unstructured control: the correct answer of every prompt is a random action, unrelated to the features ===
mean policy entropy (nats): before 4.730, after 1.352; log 400 = 5.991
train prompts, by initial success probability (averages over 10 instances)
    bin of p_x  prompts/instance  pass@1 before   after  pass@64 before   after   # with pass@64 drop > 0.05
    [0, 0.002)             114.2          0.001   0.008           0.036   0.017                        30.80
  [0.002, 0.02)              31.9          0.005   0.067           0.261   0.095                        28.60
   [0.02, 0.2)               3.7          0.037   0.603           0.849   0.656                         1.30
      [0.2, 1)               1.0          0.298   0.933           1.000   1.000                         0.00
test prompts, by initial success probability (averages over 10 instances)
    bin of p_x  prompts/instance  pass@1 before   after  pass@64 before   after   # with pass@64 drop > 0.05
    [0, 0.002)             111.0          0.001   0.000           0.037   0.007                        29.90
  [0.002, 0.02)              36.0          0.005   0.006           0.268   0.054                        33.40
   [0.02, 0.2)               3.0          0.038   0.079           0.851   0.173                         2.80
      [0.2, 1)  (no prompts)
