datablations

datablations copied to clipboard

Figure issue about your paper (Figure 4 and Figure 15)

1

Hi, I am reading your paper and I have noticed that figure 4 and figure 15 are exactly the same. Are they meant to be the same? I believe that...

MatthewYZhang

A question about the conclusion of this paper

1

"Scaling Data-Constrained Language Models" is a very nice paper, and I learn a lot from this paper. However, I have a question about this paper: In the abstract and Figure...

OleNet

Add python script to launch jobs

2

This should make it easier for us to investigate scaling laws @TevenLeScao

NouamaneTazi

wonder if LR=1e-3 for mup is optimal value from small-scale proxy model and dropout is crucial for multi-epoch

4

hi authors, thanks for the great work! i just wonder if LR=1e-3 for mup is optimal value from small-scale proxy model and how dropout is critical for multi-epoch training. for...

SeunghyunSEO

Validation loss vs model size per step

3

Hi @Muennighoff Great paper, very impressive work and very detailed - thanks for releasing the data! I wonder about a small discrepancy that I see between your work and scaling...

orena1