"Host a Pretrained Model on SageMaker" tutorial fails with "Received server error (500) from primary and could not load the entire response body."

[thalesfc]

Describe the bug "Host a Pretrained Model on SageMaker" tutorial fails with "An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from primary and could not load the entire response body."

To reproduce

follow https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-script-mode/pytorch_bert/deploy_bert_outputs.html#Deploy-Model

my example notebook: deploy-v4-any-pytorch-tutorial.md

Expected behavior

Screenshots or logs

System information

equirement already satisfied: transformers in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (4.37.1)
Requirement already satisfied: sagemaker in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (2.203.1)
Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (3.12.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (0.20.3)
Requirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (1.22.4)
Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (21.3)
Requirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (2023.10.3)
Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (2.31.0)
Requirement already satisfied: tokenizers<0.19,>=0.14 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (0.15.1)
Requirement already satisfied: safetensors>=0.3.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (0.4.2)
Requirement already satisfied: tqdm>=4.27 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from transformers) (4.66.1)
Requirement already satisfied: attrs<24,>=23.1.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (23.1.0)
Requirement already satisfied: boto3<2.0,>=1.33.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (1.34.19)
Requirement already satisfied: cloudpickle==2.2.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (2.2.1)
Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (0.2.0)
Requirement already satisfied: protobuf<5.0,>=3.12 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (4.25.2)
Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (1.0.1)
Requirement already satisfied: importlib-metadata<7.0,>=1.4.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (6.8.0)
Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (2.1.1)
Requirement already satisfied: pathos in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (0.3.1)
Requirement already satisfied: schema in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (0.7.5)
Requirement already satisfied: jsonschema in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (4.19.1)
Requirement already satisfied: platformdirs in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (3.11.0)
Requirement already satisfied: tblib<3,>=1.7.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (2.0.0)
Requirement already satisfied: urllib3<1.27 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (1.26.18)
Requirement already satisfied: uvicorn==0.22.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (0.22.0)
Requirement already satisfied: fastapi==0.95.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (0.95.2)
Requirement already satisfied: docker in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (6.1.3)
Requirement already satisfied: psutil in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from sagemaker) (5.9.5)
Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from fastapi==0.95.2->sagemaker) (1.10.13)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from fastapi==0.95.2->sagemaker) (0.27.0)
Requirement already satisfied: click>=7.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from uvicorn==0.22.0->sagemaker) (8.1.7)
Requirement already satisfied: h11>=0.8 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from uvicorn==0.22.0->sagemaker) (0.14.0)
Requirement already satisfied: botocore<1.35.0,>=1.34.19 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (1.34.19)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (1.0.1)
Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (0.10.0)
Requirement already satisfied: fsspec>=2023.5.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (2023.10.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.8.0)
Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (3.17.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from packaging>=20.0->transformers) (3.1.1)
Requirement already satisfied: websocket-client>=0.32.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from docker->sagemaker) (1.6.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from requests->transformers) (3.3.1)
Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from requests->transformers) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from requests->transformers) (2023.7.22)
Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from google-pasta->sagemaker) (1.16.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from jsonschema->sagemaker) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from jsonschema->sagemaker) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from jsonschema->sagemaker) (0.10.6)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pandas->sagemaker) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pandas->sagemaker) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pandas->sagemaker) (2023.3)
Requirement already satisfied: ppft>=1.7.6.7 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pathos->sagemaker) (1.7.6.7)
Requirement already satisfied: dill>=0.3.7 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pathos->sagemaker) (0.3.7)
Requirement already satisfied: pox>=0.3.3 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pathos->sagemaker) (0.3.3)
Requirement already satisfied: multiprocess>=0.70.15 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from pathos->sagemaker) (0.70.15)
Requirement already satisfied: contextlib2>=0.5.5 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from schema->sagemaker) (21.6.0)
Requirement already satisfied: anyio<5,>=3.4.0 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (4.0.0)
Requirement already satisfied: sniffio>=1.1 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.3.0)
Requirement already satisfied: exceptiongroup>=1.0.2 in /home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.1.3)

Additional context N/A

[thalesfc]

I was able to isolate the issue to the model_fn function:

[thalesfc]

Not sure if this is related on not, but the tokenizer.save_pretrained() call is skipping one file locally:

4 files added -->

only 3 files show up locally -->

[thalesfc]

I wrapped model_fn in a try catch and here is the error:

[zhaoqizqwang]

Hi, do you have the requirements.txt in code folder? https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker-script-mode/pytorch_bert/code In my test without this file it would have this ModelError

[zhaoqizqwang]

Closing this issue since it's not for python sdk. Track the same issue in example-notebooks repo: https://github.com/aws/amazon-sagemaker-examples/issues/4551