Ian Magnusson

references used to solve things here: exposing host gpu to container - (https://towardsdatascience.com/how-to-properly-use-the-gpu-within-a-docker-container-4c699c78c6d1) fixing opencv fail with skbuild - (https://stackoverflow.com/questions/63448467/installing-opencv-fails-because-it-cannot-find-skbuild)

On further inspection, I believe the source of the difference is the `position_ids`. When the batched and padded `past_key_values` are used, the default `position_ids` are computed by [this code](https://github.com/huggingface/transformers/blob/d0acc9537829e7d067edbb791473bbceb2ecf056/src/transformers/models/gpt2/modeling_gpt2.py#L791): ```...

@ArthurZucker thanks for looking into this! Yes using `padding_side="left"` seems like a great solution to this issue! I'm curious what is the intended path for users to figure out this...

Aha good catch @epwalsh! I'll try to familiarize myself with `scripts/prepare_memmap_dataset.py`, since I'll need to use it anyway to get our perplexity eval data into the right format. In terms...

> So I think we're all good? I mean, the seed is the one thing you need to reproduce the exact data order for a given training run. So being...

One serious consideration with respect to how the documents are concatenated thought: When a document is being concatenated to one training instance but it's too large to fit in the...

Sorry to resurrect this issue after so long, but I'm not aware of another issue or PR that covers this: I heard from @drschwenk in a meeting that there was...

Thanks @drschwenk! This is super helpful. I think we’ve met the key goal here of making it so people can reproduce the order if they want to. And we also...

> Keep in mind that code changes over time, and needs maintenance to keep working. If we run this on all affected runs now, once, then we can throw away...

All that said eventually it would be great to work towards some tooling for making sense of the new mapping of tokenized NPY files to specific documents and spans of...