First, is there a typo in the 3rd set, the re-scaled version? Starting from `BF16 values, 0.0625, 0.375, 0.078125, and 0.25` seems like our poor 0.25 got missed :-(
> 4-bit floating point value by the scaling factor, resulting in: 0.0625, 0.375, 0.09375, and 0.0625.
Shouldn't that last one be 0.25 as it ought to fit in a 16-bit space, just like 0.375.
Also, two questions about the scaling factor: First, how are these 32 (or 16) common-scaling blocks determined? Memory-adjacent? Some higher order data structure?
Second, in the training phase, does that factor evolve in time? i.e. Do the scaling factors appear in the very outset of a training run, as a factor of the initial data? Or does the model have to train for a while before block commonalities are apparent?
p.s.
> This is going to get really nerdy, really quickly here, so we won't judge if you want to jump straight to the why it matters section.
Nah, we love this stuff. Fascinating and from the looks of it, quite significant. Esp. from the PoV of running LLMs locally on some mid/upper range GPUs with constrained memory.