decode step time for context c = 2048 and its tokens per second per device
 batch b  memory (ms)  compute (ms)  step (ms)   tokens/s     bound
       1         6.03          0.08       6.03        166    memory
       8         6.97          0.63       6.97       1147    memory
      32        10.19          2.52      10.19       3139    memory
      64        14.49          5.03      14.49       4417    memory
     128        23.08         10.07      23.08       5546    memory
     256        40.26         20.14      40.26       6359    memory
batch at which compute equals weight-reading time (c = 0): b = MFU_d * peak / BW = 75
largest batch that fits at c = 2048 on one device: 252; tokens/s there: 6344

one RL iteration: 512 prompts x 8 samples, prompt 512 tokens, mean response 400 tokens (continuous batching, mean context ~ 512 + 200)
  generated tokens 1,638,400; trained tokens 3,735,552 (prompt + response)
  generation 12,712 tokens/s per device (batch 252); training 2,542 tokens/s per device (MFU 0.4, 8 P FLOPs per token)
  single-device times: generation 128.9 s, training 1469.3 s; balanced split n_g : n_t = 0.09 : 1
 devices   n_g   n_t  gen (s)  train (s)  iteration, async (s)  iteration, colocated sync (s)
       8     1     7    128.9      209.9                 209.9                          199.8
      16     1    15    128.9       98.0                 128.9                           99.9
      32     3    29     43.0       50.7                  50.7                           49.9
      64     5    59     25.8       24.9                  25.8                           25.0
(the colocated figure assumes all devices alternate between generating and training and ignores weight-reload and resharding time)
