RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 when running mpt-7b

[paulovasconcellos-hotmart]

Hi Everyone. I'm trying to use the fresh new MPT-7b included in vllm. I'm running on SageMaker Studio, in a g4dn.2xlarge instance, however, I'm getting the following error:

RuntimeError: probability tensor contains either inf, nan or element < 0

My code

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="mosaicml/mpt-7b", dtype='float16')

outputs = llm.generate(prompts, sampling_params) ### error happens here

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

This is my environment:

accelerate @ file:///home/conda/feedstock_root/build_artifacts/accelerate_1683553934867/work
aiofiles==23.1.0
aiohttp==3.8.4
aiosignal==1.3.1
altair==5.0.1
anyio==3.7.0
apex @ file:///apex
appdirs==1.4.4
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1670263926556/work
async-timeout==4.0.2
attrs==22.2.0
awscli @ file:///home/conda/feedstock_root/build_artifacts/awscli_1683792289807/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
bcrypt==4.0.1
blis @ file:///home/conda/feedstock_root/build_artifacts/cython-blis_1668499088869/work
bokeh @ file:///home/conda/feedstock_root/build_artifacts/bokeh_1683730530224/work
boto3 @ file:///home/conda/feedstock_root/build_artifacts/boto3_1683763173043/work
botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1683758921974/work
brotlipy @ file:///home/conda/feedstock_root/build_artifacts/brotlipy_1666764671472/work
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
catalogue @ file:///home/conda/feedstock_root/build_artifacts/catalogue_1666891892909/work
certifi==2023.5.7
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1671179353105/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1678108872112/work
click @ file:///home/conda/feedstock_root/build_artifacts/click_1666798198223/work
cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1674202310934/work
cmake==3.26.3
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/work
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1679481329611/work
commonmark==0.9.1
conda==23.1.0
conda-content-trust @ file:///home/conda/feedstock_root/build_artifacts/conda-content-trust_1621370699668/work
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1669907009957/work
conda_package_streaming @ file:///home/conda/feedstock_root/build_artifacts/conda-package-streaming_1669733752472/work
confection @ file:///home/conda/feedstock_root/build_artifacts/confection_1673621475775/work
contextlib2==21.6.0
contourpy @ file:///home/conda/feedstock_root/build_artifacts/contourpy_1673633665736/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography-split_1679811212387/work
cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work
cymem @ file:///home/conda/feedstock_root/build_artifacts/cymem_1666909672496/work
Cython @ file:///home/conda/feedstock_root/build_artifacts/cython_1680712295460/work
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1680755465990/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
deepspeed @ https://aws-deepspeed-zero-2d-binaries.s3.us-west-2.amazonaws.com/r2.0.0/20230407-184728/1ea3d4b6aa41fe66277daacbb78b3743a310d85a/deepspeed-0.6.1%2B1ea3d4b-py3-none-any.whl#sha256=f59834b5a39738f4f180757dbe8550dfd8fbcc97cd863bc0ee362d1ab81e3873
dgl==1.1.0+cu118
dill==0.3.6
docker-pycreds==0.4.0
docutils @ file:///home/conda/feedstock_root/build_artifacts/docutils_1667993608396/work
einops==0.6.1
exceptiongroup==1.1.2
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work
fastai @ file:///home/jhoward/mambaforge/conda-bld/fastai_1680035345463/work
fastapi==0.99.1
fastcore @ file:///home/jhoward/mambaforge/conda-bld/fastcore_1680034914245/work
fastdownload @ file:///home/jhoward/mambaforge/conda-bld/fastdownload_1657219113869/work
fastprogress @ file:///home/jhoward/mambaforge/conda-bld/fastprogress_1658473398631/work
ffmpy==0.3.0
filelock @ file:///home/conda/feedstock_root/build_artifacts/filelock_1681839547898/work
flash-attn==0.2.8
fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1683740454859/work
frozenlist==1.3.3
fschat==0.2.17
fsspec==2023.5.0
future @ file:///home/conda/feedstock_root/build_artifacts/future_1673596611778/work
gevent==22.10.2
gitdb==4.0.10
GitPython==3.1.31
gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1666808654411/work
google-pasta==0.2.0
gradio==3.35.2
gradio_client==0.2.7
greenlet==2.0.2
grpcio==1.51.3
h11==0.14.0
h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1675704794369/work
hjson==3.1.0
horovod==0.26.1
httpcore==0.17.2
httpx==0.24.1
huggingface-hub==0.15.1
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1683031833737/work
importlib-metadata==4.13.0
inotify-simple==1.2.1
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1683553336538/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1683225895562/work
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1654302431367/work
jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1655568249366/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1663332044897/work
jsonpatch==1.32
jsonpointer==2.3
jsonschema==4.17.3
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1681432441054/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1678994169527/work
kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1666805701884/work
langcodes @ file:///home/conda/feedstock_root/build_artifacts/langcodes_1636741340529/work
libmambapy @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1680002410624/work/libmambapy
linkify-it-py==2.0.2
lit==16.0.3
llvmlite==0.39.1
mamba @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1680002410624/work/mamba
markdown-it-py==2.2.0
markdown2==2.4.9
MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1674135787083/work
matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1678135565516/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mdit-py-plugins==0.3.3
mdurl==0.1.2
mpi4py @ file:///home/conda/feedstock_root/build_artifacts/mpi4py_1667459939419/work
mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1678228039184/work
msgpack==1.0.5
multidict==6.0.4
multiprocess==0.70.14
munkres==1.1.4
murmurhash @ file:///home/conda/feedstock_root/build_artifacts/murmurhash_1666946151787/work
mypy-extensions==1.0.0
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1680692919326/work
nh3==0.2.13
ninja==1.11.1
numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1680825379968/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1668919096861/work
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
opencv-python==4.7.0
orjson==3.9.1
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1681337016113/work
pandas @ file:///home/conda/feedstock_root/build_artifacts/pandas_1683493925851/work
paramiko==3.1.0
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
pathos==0.3.0
pathtools==0.1.2
pathy @ file:///home/conda/feedstock_root/build_artifacts/pathy_1670689864140/work
patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1665356157073/work
peft==0.3.0
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1675487172403/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1682644429438/work
plotly @ file:///home/conda/feedstock_root/build_artifacts/plotly_1680731398751/work
pluggy @ file:///home/conda/feedstock_root/build_artifacts/pluggy_1667232663820/work
ply==3.11
pooch @ file:///home/conda/feedstock_root/build_artifacts/pooch_1679580333621/work
pox==0.3.2
ppft==1.7.6.6
preshed @ file:///home/conda/feedstock_root/build_artifacts/preshed_1666991224827/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1677600924538/work
protobuf==3.20.3
protobuf3-to-dict==0.1.5
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1681775027942/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
py-cpuinfo==9.0.0
pyarrow==12.0.0
pyasn1==0.4.8
pybind11 @ file:///home/conda/feedstock_root/build_artifacts/pybind11-split_1679012409253/work
pybind11-global @ file:///home/conda/feedstock_root/build_artifacts/pybind11-split_1679012409253/work
pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1666836542287/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1679565261911/work
pydub==0.25.1
pyfunctional==1.4.3
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1681904169130/work
pyinstrument==3.4.2
pyinstrument-cext==0.2.4
PyNaCl==1.5.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1680037383858/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work
PyQt5==5.15.7
PyQt5-sip==12.11.0
pyre-extensions==0.0.29
pyrsistent==0.19.3
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-multipart==0.0.6
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1680088766131/work
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1668001474078/work
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1679316826707/work
ray==2.5.1
regex==2023.6.3
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1680286922386/work
retrying==1.3.4
rich @ file:///home/conda/feedstock_root/build_artifacts/rich_1664752510089/work
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1614171254180/work
ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1678272977710/work
ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1670412719074/work
s3fs==0.4.2
s3transfer @ file:///home/conda/feedstock_root/build_artifacts/s3transfer_1683241957497/work
sagemaker==2.154.0
sagemaker-experiments==0.1.43
sagemaker-pytorch-training==2.8.0
sagemaker-training==4.5.0
schema==0.7.5
scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1679675836718/work
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1683719288579/work/dist/scipy-1.10.1-cp310-cp310-linux_x86_64.whl#sha256=eeee39d8a01a8072da1efa959a1490fe1e94114fa147125c39cf3c438f69ca54
seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1672497695270/work
semantic-version==2.10.0
sentencepiece==0.1.99
sentry-sdk==1.27.0
setproctitle==1.3.2
shap @ file:///home/conda/feedstock_root/build_artifacts/shap_1655716950751/work
shellingham @ file:///home/conda/feedstock_root/build_artifacts/shellingham_1676292972954/work
shortuuid==1.0.11
sip @ file:///home/conda/feedstock_root/build_artifacts/sip_1681995008230/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
slicer @ file:///home/conda/feedstock_root/build_artifacts/slicer_1608146800664/work
smart-open @ file:///home/conda/feedstock_root/build_artifacts/smart_open_1630238320325/work
smclarify==0.5
smdebug @ file:///tmp/sagemaker-debugger
smdebug-rulesconfig==1.0.1
smdistributed-dataparallel @ https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.0.0/cu118/2023-03-20/smdistributed_dataparallel-1.8.0-cp310-cp310-linux_x86_64.whl#sha256=4952b8de26aaa2ed51b8e668f68be4abd0bf1b35378e979561d872acba31ecd3
smdistributed-modelparallel @ https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-2.0.0/build-artifacts/2023-04-14-20-14/smdistributed_modelparallel-1.15.0-cp310-cp310-linux_x86_64.whl#sha256=5a772776a6a280581e452208c62d8ca20b0f6d4d2c59ec294f5402dc5b89b1f1
smmap==5.0.0
sniffio==1.3.0
spacy @ file:///home/conda/feedstock_root/build_artifacts/spacy_1681807679135/work
spacy-legacy @ file:///home/conda/feedstock_root/build_artifacts/spacy-legacy_1674550301837/work
spacy-loggers @ file:///home/conda/feedstock_root/build_artifacts/spacy-loggers_1672303484730/work
srsly @ file:///home/conda/feedstock_root/build_artifacts/srsly_1677657434449/work
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
starlette==0.27.0
statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1683305553485/work
svgwrite==1.4.3
sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1679342590084/work
tabulate==0.9.0
tblib==1.7.0
tenacity @ file:///home/conda/feedstock_root/build_artifacts/tenacity_1677600641219/work
thinc @ file:///home/conda/feedstock_root/build_artifacts/thinc_1683130983739/work
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work
tiktoken==0.4.0
tokenizers==0.13.3
toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work
torch==2.0.1
torchaudio==2.0.1
torchdata @ file:///opt/conda/conda-bld/torchdata_1679615656247/work
torchnet==0.0.4
torchtext==0.15.1
torchvision==0.15.1
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1681817446549/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1677948868469/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
transformers==4.28.1
triton==2.0.0
typer @ file:///home/conda/feedstock_root/build_artifacts/typer_1667832226065/work
typing-inspect==0.9.0
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1678559861143/work
tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1680081134351/work
uc-micro-py==1.0.2
unicodedata2 @ file:///home/conda/feedstock_root/build_artifacts/unicodedata2_1667239886688/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1678635778344/work
uvicorn==0.22.0
visdom==0.2.4
vllm @ git+https://github.com/vllm-project/vllm.git@98fe8cb5420c28fa8dcc3110b6c898848dd57e45
wandb==0.15.4
wasabi @ file:///home/conda/feedstock_root/build_artifacts/wasabi_1673945962927/work
wavedrom==2.0.3.post3
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1673864653149/work
websocket-client==1.5.1
websockets==11.0.3
Werkzeug==2.3.4
xformers @ git+https://github.com/facebookresearch/xformers@1f449ef81680707d38e1739c627b5bffee7732c6
xyzservices @ file:///home/conda/feedstock_root/build_artifacts/xyzservices_1676835466992/work
yarl==1.9.2
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1677313463193/work
zope.event==4.6
zope.interface==6.0
zstandard==0.19.0