keyError : 'attn_impl' when converting a composer checkpoint to a standard HF checkpoint

[jdongca2003]

I uncomment the following lines in https://github.com/mosaicml/llm-foundry/blob/main/scripts/train/finetune_example/gpt2-arc-easy.yaml

save_num_checkpoints_to_keep: 1 # Important, this cleans up checkpoints saved to DISK save_folder: ./{run_name}/checkpoints

Then run "composer train.py finetune_example/gpt2-arc-easy.yaml". I can see the saved composer checkpoint.

total 1474508 drwxrwxr-x 2 ubuntu ubuntu 4096 May 28 04:55 . drwxrwxr-x 3 ubuntu ubuntu 4096 May 28 04:54 .. -rw-rw-r-- 1 ubuntu ubuntu 1509881277 May 28 04:55 ep1-ba294-rank0.pt lrwxrwxrwx 1 ubuntu ubuntu 18 May 28 04:55 latest-rank0.pt -> ep1-ba294-rank0.pt

But when I tried to convert this composer checkpoint to HF checkpoint , I got error:

python convert_composer_to_hf.py --composer_path /home/ubuntu/llm-foundry/scripts/train/llm/checkpoints/latest-rank0.pt --hf_output_path my_hf_model/ --output_precision bf16

ownloading checkpoint from /home/ubuntu/llm-foundry/scripts/train/llm/checkpoints/latest-rank0.pt -> /tmp/tmpbd4xucf3/local-composer-checkpoint.pt Loading checkpoint into CPU RAM... ############################## Saving HF Model Config... Traceback (most recent call last): File "/home/ubuntu/llm-foundry/scripts/inference/convert_composer_to_hf.py", line 476, in main(parse_args()) File "/home/ubuntu/llm-foundry/scripts/inference/convert_composer_to_hf.py", line 371, in main write_huggingface_pretrained_from_composer_checkpoint( File "/home/ubuntu/llm-foundry/scripts/inference/convert_composer_to_hf.py", line 212, in write_huggingface_pretrained_from_composer_checkpoint hf_config = get_hf_config_from_composer_state_dict(composer_state_dict) File "/home/ubuntu/llm-foundry/scripts/inference/convert_composer_to_hf.py", line 44, in get_hf_config_from_composer_state_dict attn_config['attn_impl'] = hf_config_dict['attn_impl'] KeyError: 'attn_impl'

checkout: master branch.

[dakinggg]

Please see https://github.com/mosaicml/llm-foundry/issues/139#issuecomment-1555426919

[jdongca2003]

thanks. After the above wrapper function is applied, only two files are generated in HF output folder: config.json and pytorch_model.bin

In order to run inference, I still need to copy tokenizer config/vocab from the original model (in the above example, I copied them from original GPT2 HF model folder)

tokenizer.json vocab.json

Then running hf_generate.py works.

Is it possible for the above wrapper function to save HF model with all info ?

[dakinggg]

Yes, adding the tokenizer to the output of this function is something I intend to do soon, but for now you can add these lines to the function in order to dump the tokenizer to the same folder.

[abhi-mosaic]

I think this has been solved now on the main branch. Closing but feel free to reopen if the issue persists.