OpenLLM

OpenLLM copied to clipboard

Describe the bug

new install of openllm on python 3.12.2

running cmd: TRUST_REMOTE_CODE=True openllm start mistralai/Mistral-7B-Instruct-v0.1

To reproduce

1. install pyenv: brew install pyenv
2. add path to zsh:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

3. install pyenv-virtualenv: git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
4. echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.zshrc
5. install python 3.12.2: pyenv install 3.12.2
6. set global: pyenv global 3.12.2
7. install openllm: pip install openllm
8. quickstart: TRUST_REMOTE_CODE=True openllm start mistralai/Mistral-7B-Instruct-v0.1

Logs

2024-04-14T16:54:13+0800 [ERROR] [runner:llm-mistral-runner:1] An exception occurred while instantiating runner 'llm-mistral-runner', see details below:
2024-04-14T16:54:13+0800 [ERROR] [runner:llm-mistral-runner:1] Traceback (most recent call last):
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
self._set_handle(LocalRunnerRef)
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
runner_handle = handle_class(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_runners.py", line 165, in __init__
self.llm, self.config, self.model, self.tokenizer = llm, llm.config, llm.model, llm.tokenizer
^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_llm.py", line 459, in model
model = openllm.serialisation.load_model(self, *self._model_decls, **self._model_attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/__init__.py", line 63, in caller
return getattr(importlib.import_module(f'.{serde}', 'openllm.serialisation'), fn)(llm, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/__init__.py", line 97, in load_model
auto_class = infer_autoclass_from_llm(llm, config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/_helpers.py", line 33, in infer_autoclass_from_llm
raise ValueError(
ValueError: Invalid configuration for mistralai/Mistral-7B-Instruct-v0.1. ``trust_remote_code=True`` requires `transformers.PretrainedConfig` to contain a `auto_map` mapping

2024-04-14T16:54:13+0800 [ERROR] [runner:llm-mistral-runner:1] Traceback (most recent call last):
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/server/base_app.py", line 75, in lifespan
on_startup()
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 317, in init_local
raise e
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
self._set_handle(LocalRunnerRef)
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
runner_handle = handle_class(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_runners.py", line 165, in __init__
self.llm, self.config, self.model, self.tokenizer = llm, llm.config, llm.model, llm.tokenizer
^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_llm.py", line 459, in model
model = openllm.serialisation.load_model(self, *self._model_decls, **self._model_attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/__init__.py", line 63, in caller
return getattr(importlib.import_module(f'.{serde}', 'openllm.serialisation'), fn)(llm, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/__init__.py", line 97, in load_model
auto_class = infer_autoclass_from_llm(llm, config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/_helpers.py", line 33, in infer_autoclass_from_llm
raise ValueError(
ValueError: Invalid configuration for mistralai/Mistral-7B-Instruct-v0.1. ``trust_remote_code=True`` requires `transformers.PretrainedConfig` to contain a `auto_map` mapping

Environment

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.1.11 python: 3.12.2 platform: macOS-14.4.1-arm64-arm-64bit uid_gid: 501:20

accelerate==0.29.2
aiohttp==3.9.4
aiosignal==1.3.1
anyio==4.3.0
appdirs==1.4.4
asgiref==3.8.1
attrs==23.2.0
bentoml==1.1.11
bitsandbytes==0.41.3.post2
build==0.10.0
cattrs==23.1.2
certifi==2024.2.2
charset-normalizer==3.3.2
circus==0.18.0
click==8.1.7
click-option-group==0.5.6
cloudpickle==3.0.0
coloredlogs==15.0.1
contextlib2==21.6.0
datasets==2.18.0
deepmerge==1.1.1
Deprecated==1.2.14
dill==0.3.8
distlib==0.3.8
distro==1.9.0
einops==0.7.0
fastcore==1.5.29
filelock==3.13.4
filetype==1.2.0
frozenlist==1.4.1
fs==2.4.16
fsspec==2024.2.0
ghapi==1.0.5
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
huggingface-hub==0.22.2
humanfriendly==10.0
idna==3.7
importlib-metadata==6.11.0
inflection==0.5.1
Jinja2==3.1.3
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
mypy-extensions==1.0.0
networkx==3.3
numpy==1.26.4
nvidia-ml-py==11.525.150
openllm==0.4.44
openllm-client==0.4.44
openllm-core==0.4.44
opentelemetry-api==1.20.0
opentelemetry-instrumentation==0.41b0
opentelemetry-instrumentation-aiohttp-client==0.41b0
opentelemetry-instrumentation-asgi==0.41b0
opentelemetry-sdk==1.20.0
opentelemetry-semantic-conventions==0.41b0
opentelemetry-util-http==0.41b0
optimum==1.18.1
orjson==3.10.0
packaging==24.0
pandas==2.2.2
pathspec==0.12.1
pillow==10.3.0
pip-requirements-parser==32.0.1
pip-tools==7.3.0
platformdirs==4.2.0
prometheus_client==0.20.0
protobuf==5.26.1
psutil==5.9.8
pyarrow==15.0.2
pyarrow-hotfix==0.6
pydantic==1.10.15
Pygments==2.17.2
pyparsing==3.1.2
pyproject_hooks==1.0.0
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
python-multipart==0.0.9
pytz==2024.1
PyYAML==6.0.1
pyzmq==25.1.2
regex==2023.12.25
requests==2.31.0
rich==13.7.1
safetensors==0.4.2
schema==0.7.5
scipy==1.13.0
sentencepiece==0.2.0
setuptools==69.5.1
simple-di==0.1.5
six==1.16.0
sniffio==1.3.1
starlette==0.37.2
sympy==1.12
tokenizers==0.15.2
torch==2.2.2
tornado==6.4
tqdm==4.66.2
transformers==4.39.3
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
uvicorn==0.29.0
virtualenv==20.25.1
watchfiles==0.21.0
wheel==0.43.0
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4
zipp==3.18.1

System information (Optional)

No response

Apr 14 '24 09:04 billy-le

you can run openllm without "TRUST_REMOTE_CODE=True ".

Apr 19 '24 09:04 yufeng1684

This seems like a users issue, mistral is already supported by transformers, so no need to use TRUST_REMOTE_CODE=True.

Jun 01 '24 13:06 aarnphm