djl
djl copied to clipboard
Failed to deploy Mistral 8X7b with the latest LMI djl-tensorrtllm container on Sagemaker G5.48xlarge
Description
image_uri = image_uris.retrieve(
framework="djl-tensorrtllm",
region=sess.boto_session.region_name,
version="0.29.0"
)
model = sagemaker.Model(
image_uri=image_uri,
role=role,
# specify all environment variable configs in this map
env={
"HF_MODEL_ID": "mistralai/Mixtral-8x7B-Instruct-v0.1",
"TENSOR_PARALLEL_DEGREE": "max",
"OPTION_MAX_NUM_TOKENS": "8192",
"OPTION_QUANTIZE": "awq",
"HF_TOKEN": "hf_xNBRqleBjkvQPnDqFxxxxxxxxxxxxxxxxx",
}
)
Expected Behavior
(what's the expected behavior?)
Error Message
logs attached