OLMo
OLMo copied to clipboard
Fix a bug w.r.t. how local tokenizers are handled
hf_olmo/convert_olmo_to_hf.py currently crashes if the YAML file in the input checkpoint refers to a local tokenizer (it tries to load the local path from HF). I added a check to make sure that the file does not exist first, as in other functions in the repo.
Can you add a note to the Changelog? Then we're good to go.