Teven comments

Results 11 comments of


                                            Teven

Preprocessing from arrow file to load an HF dataset

Update: this version is ~4 times slower with the change from column format to row format, compared to a hacky version that just gets the column for a certain feature....

Add valid data

Hey Maruf, sorry, not yet, I'm a bit swamped at the moment and the priority switched to cleaning OSCAR-ml additionally ourselves before launching anything on it, maybe @ibeltagy can review...

Compute model param count once

Hey ! The number without embeddings is actually just that: the number of non-embedding parameters, not the number of unique parameters. This is the relevant number to estimate the loss...

Double counts in parameter count

Ah yes, saw it on the other issue then forgot about it - I can take a look at the end of this week.

mC4 sampling & pre-processing

We did use mc4 for early multilingual experiments before switching to OSCAR - let's keep the code for future reference. Thanks for catching this!

faiss::gpu::runMatrixMult ... cublas failed (13): (1024, 12) x (256, 12)' = (1024, 256) gemm params m 256 n 1024 k 12 trA T trB N lda 12 ldb 12 ldc 256

Seeing this with A100 / CUDA 11.5 / faiss-gpu=1.7.2

Teven