The conclusions of this paper are questionable

[zitian-gao]

Some of the experiments were trained on only 0.5B tokens, or even just a few dozen optimization steps, how can such conclusions be justified? Pretraining should involve at least 100B tokens.

[Hprairie]

Hey, I'm not really sure what experiments you are talking about. In section 5.1, we train a 1.3B model on over 100B tokens.

If you are talking about the micro experiments done in section 3, we stop early, as divergence means that we crash from NaNs or increasing loss. Recovery from a NaN is impossible, and it happens almost instantly...

I'm happy to help clarify any part of the experimental setup that is confusing, but we tried to be as robust as possible with the compute budget we had.