djl icon indicating copy to clipboard operation
djl copied to clipboard

Failed to deploy Mistral 8X7b with the latest LMI djl-tensorrtllm container on Sagemaker G5.48xlarge

Open gsjoy8888 opened this issue 1 year ago • 0 comments

Description

image_uri = image_uris.retrieve(
        framework="djl-tensorrtllm",
        region=sess.boto_session.region_name,
        version="0.29.0"
    )
model = sagemaker.Model(
    image_uri=image_uri, 
    role=role,
    # specify all environment variable configs in this map
    env={
        "HF_MODEL_ID": "mistralai/Mixtral-8x7B-Instruct-v0.1",
        "TENSOR_PARALLEL_DEGREE": "max",
        "OPTION_MAX_NUM_TOKENS": "8192",
        "OPTION_QUANTIZE": "awq",
        "HF_TOKEN": "hf_xNBRqleBjkvQPnDqFxxxxxxxxxxxxxxxxx",
    }
)

Expected Behavior

(what's the expected behavior?)

Error Message

error_log.txt

logs attached

gsjoy8888 avatar Aug 29 '24 11:08 gsjoy8888