text-generation-inference

text-generation-inference copied to clipboard

System Info

Following the guide given on https://huggingface.co/blog/sagemaker-huggingface-llm, trying to deploy a fine tuned Falcon 7B model yields the following errors:

Error: DownloadError
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 84, in convert_files
    convert_file(pt_file, sf_file)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 53, in convert_file
    if "state_dict" in pt_state:
TypeError: argument of type 'TrainingArguments' is not iterable

Information

• [X] Docker
• [ ] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

Following steps given on the huggingface website

Expected behavior

Normal SageMaker endpoint inference

In addition it also produced the error

#033[2m2023-06-18T20:18:24.086329Z#033[0m #033[31mERROR#033[0m #033[2mtext_generation_launcher#033[0m#033[2m:#033[0m Download encountered an error: Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 151, in download_weights
    utils.convert_files(local_pt_files, local_st_files)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 84, in convert_files
    convert_file(pt_file, sf_file)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 53, in convert_file

Which model is it ? The tool is trying to convert the training parameters which are not convertible.

We will just need to skip it.

Which model is it ? The tool is trying to convert the training parameters which are not convertible.

We will just need to skip it.

Hi, I am having the exact issue with deploying the mrm8488/t5-base-finetuned-wikiSQL model using the code given on hugging face. I am a beginner, any insights on how to "skip" this?

Indeed there's a training file here: https://github.com/huggingface/text-generation-inference/pull/485

Hi thanks! I tried to installing the file using

!pip install "git+https://github.com/huggingface/text-generation-inference.git@ignore_training_args#egg=text-generation-inference&subdirectory=clients/python"

However, the same error is still showing up. Do I need to uninstall the original package and how would I do so?

Additionally, will skipping training argument conversions affect the model? Sorry I am really new to this field so I am trying to figure things out.

Thanks again!

Any solution to this? Facing the same issue with deploying Fine-Tune L2-70b on g5.48xlarge gptq quantized. Error: DownloadError File "/opt/conda/bin/text-generation-server", line 8, in <module> sys.exit(app()) File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 151, in download_weights utils.convert_files(local_pt_files, local_st_files) File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 84, in convert_files convert_file(pt_file, sf_file) File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 52, in convert_file pt_state = torch.load(pt_file, map_location="cpu") File "/opt/conda/lib/python3.9/site-packages/torch/serialization.py", line 815, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/opt/conda/lib/python3.9/site-packages/torch/serialization.py", line 1033, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args)

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.