llama-recipes
llama-recipes copied to clipboard
use AutoTokenizer instead of LlamaTokenizer in checkpoint_converter_fsdp_hf.py
What does this PR do?
Feature/Issue validation/testing
Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced. Please also list any relevant details for your test configuration.
Since LlamaTokenizer
is not compatible with Llama3 tokenizers, running checkpoint_converter_fsdp_hf.py
with llama3 finetuned weights result in TypeError: not a string
error (cf. https://github.com/huggingface/transformers/issues/30607). This PR is suggesting to use AutoTokenizer
instead to make the script compatible with both Llama2/3.
Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline, Pull Request section?
- [ ] Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?
Thanks for contributing 🎉!