nvda-bnorick comments

Repositories
Issues
Comments

Results 1 comments of


                                            nvda-bnorick

add more tokenizer arguments

As it stands, this PR deletes `--tiktoken-special-tokens` but doesn't fix all places which refer to the argument, such as [this one](https://github.com/NVIDIA/Megatron-LM/blob/8b42c45bc1df8ddfeccd5c36705880fcf2f4c119/megatron/training/tokenizer/tokenizer.py#L66). I didn't look for others, this was just the...