Unable to convert fine-tune results to 🤗 format

[markbastian]

I am unable to convert fine tune results to the 🤗 format for inference.

Here's an example where I am able to do a simple fine tune using the t5-small_dolly_sft.yaml example, but I can't see how to actually get past the conversion stage so that inferencing can be done.

composer train/train.py \
  train/yamls/finetune/t5-small_dolly_sft.yaml \
  max_duration=10ba \
  save_folder=t5-small_dolly_sft
  
python inference/convert_composer_to_hf.py \
  --composer_path t5-small_dolly_sft/ep0-ba10-rank0.pt \
  --hf_output_path t5-small_dolly_sft-hf \
  --output_precision bf16

When I run the second command, I get errors regarding missing keys in the hf_config_dict in get_hf_config_from_composer_state_dict in the scripts/inference/convert_composer_to_hf.py file. I think the attn_config is not carried forward in fine tunes or perhaps this function should have safer defaults if that data isn't present. At the moment, it will throw exceptions with messages like KeyError: 'attn_pdrop'.

Documentation or a working example of how to do this would be great.

[dakinggg]

Hi, please see https://github.com/mosaicml/llm-foundry/issues/139#issuecomment-1555426919