OLMo icon indicating copy to clipboard operation
OLMo copied to clipboard

Fix a bug w.r.t. how local tokenizers are handled

Open gahdritz opened this issue 2 years ago • 1 comments

hf_olmo/convert_olmo_to_hf.py currently crashes if the YAML file in the input checkpoint refers to a local tokenizer (it tries to load the local path from HF). I added a check to make sure that the file does not exist first, as in other functions in the repo.

gahdritz avatar Feb 16 '24 06:02 gahdritz

Can you add a note to the Changelog? Then we're good to go.

dirkgr avatar Feb 19 '24 19:02 dirkgr