Stas Bekman
Stas Bekman
@ibeltagy?
Yeah, there are multiple issues with circular imports in megatron-lm. We fixed some but more are popping up. And I don't think it was designed for being installed. Remember it...
> Turns out pip install -e . --no-use-pep517 works. I'm still unclear why that is. can this somehow be enabled automatically in setup.py? That's too hard to remember
let's talk what goes where: - `examples` aren't the best place - we now fully own this repo - so we want logical placements - probably should remove this folder...
Thank you for the feedback, @sbmaruf! OK, let's leave the `tools` as it is for now and then we can move the whole thing at once if it makes more...
> sentencepiece tokenizer are not as straightforward as to resizing and remapping. Here is a hack that helps with shrinking an spm vocab: https://discuss.huggingface.co/t/tokenizer-shrinking-recipes/8564 It may or may not help,...
oh, sorry I missed your reply. it's great, but it'd help a lot to have a link from README.md to that part of github - as typically repos don't include...
Awesome work, @tjruwase! At one point it'd be great to have an instructional document of how a user can go from one ZeRO topology to another using the existing checkpoint....
I have tried t5-large, tested your script to work fine with t5-small - need to find a box with a few large gpus to test t5-large. Meanwhile, we should revisit...
oh, great, then I don't need to look for a set of large GPUs :) Thank you for this update, @harshil-shah! Indeed please do let us know when you get...