BentoML
BentoML copied to clipboard
bentoml
bug: aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
Describe the bug
This keeps happening with mainly transformer based models. However, this error keeps prevailing only after containerizing like using
bentoml containerize bento_svc:latest and running the container like docker run --rm -p 3000:3000 bento_svc:t5mqzwdlzoxbjdx2
I have worked on multiple models like the one in documentation https://docs.bentoml.com/en/latest/quickstarts/deploy-a-transformer-model-with-bentoml.html even distil bert models and also sentence embedding.
I cloned the source code of this sentence embedding and tried using containerize and run. it still fails but works like charm if used docker container out of the box.
https://github.com/bentoml/sentence-embedding-bento/blob/main/requirements.txt
This happened with bentoml 1.1.6 and updated version 1.1.7 as well.
Also, I am using Mac OS with Apple Silicon chip if that matters.
I am able to run with scikit learn models and even torch based models that I trained.
Wonder what am I doing wrong here.
strack-trace for further introspection:
2023-10-16T02:32:33+0000 [ERROR] [api_server:4] Exception on /encode [POST] (trace=f755596b7d185db8d620262d333fa9ae,span=2c59e1cd9a90f9cb,sampled=0,service.name=sentence-embedding-svc) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/http_app.py", line 341, in api_func output = await api.func(*args) File "/home/bentoml/bento/src/service.py", line 38, in encode return await embed_runner.encode.async_run(docs.dict()) File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner.py", line 55, in async_run return await self.runner._runner_handle.async_run_method(self, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner_handle/remote.py", line 216, in async_run_method async with self._client.post( File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1167, in __aenter__ self._resp = await self._coro File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 586, in _request await resp.start(conn) File "/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 905, in start message, payload = await protocol.read() # type: ignore[union-attr] File "/usr/local/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read await self._waiter aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
To reproduce
No response
Expected behavior
No response
Environment
Environment variable
BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''
System information
bentoml: 1.1.7
python: 3.9.18
platform: macOS-13.6-arm64-i386-64bit
uid_gid: 502:20
conda: 23.1.0
in_conda_env: True
conda_packages
name: bento_39
channels:
- defaults
dependencies:
- ca-certificates=2023.08.22=hca03da5_0
- libcxx=14.0.6=h848a8c0_0
- libffi=3.4.4=hca03da5_0
- ncurses=6.4=h313beb8_0
- openssl=3.0.11=h1a28f6b_2
- pip=23.2.1=py39hca03da5_0
- python=3.9.18=hb885b13_0
- readline=8.2=h1a28f6b_0
- setuptools=68.0.0=py39hca03da5_0
- sqlite=3.41.2=h80987f9_0
- tk=8.6.12=hb8d0fd4_0
- tzdata=2023c=h04d1e81_0
- wheel=0.41.2=py39hca03da5_0
- xz=5.4.2=h80987f9_0
- zlib=1.2.13=h5a0b063_0
- pip:
- absl-py==2.0.0
- aiohttp==3.8.6
- aiosignal==1.3.1
- annotated-types==0.6.0
- anyio==4.0.0
- appdirs==1.4.4
- asgiref==3.7.2
- astunparse==1.6.3
- async-timeout==4.0.3
- attrs==23.1.0
- bentoml==1.1.7
- build==1.0.3
- cachetools==5.3.1
- cattrs==23.1.2
- certifi==2023.7.22
- charset-normalizer==3.3.0
- circus==0.18.0
- click==8.1.7
- click-option-group==0.5.6
- cloudpickle==2.2.1
- contextlib2==21.6.0
- deepmerge==1.1.0
- deprecated==1.2.14
- exceptiongroup==1.1.3
- filelock==3.12.4
- flatbuffers==23.5.26
- frozenlist==1.4.0
- fs==2.4.16
- fsspec==2023.9.2
- gast==0.5.4
- google-auth==2.23.3
- google-auth-oauthlib==1.0.0
- google-pasta==0.2.0
- grpcio==1.59.0
- h11==0.14.0
- h5py==3.10.0
- httpcore==0.18.0
- httpx==0.25.0
- huggingface-hub==0.17.3
- idna==3.4
- importlib-metadata==6.0.1
- inflection==0.5.1
- jinja2==3.1.2
- joblib==1.3.2
- keras==2.14.0
- libclang==16.0.6
- markdown==3.5
- markdown-it-py==3.0.0
- markupsafe==2.1.3
- mdurl==0.1.2
- ml-dtypes==0.2.0
- mpmath==1.3.0
- multidict==6.0.4
- networkx==3.1
- numpy==1.26.0
- oauthlib==3.2.2
- opentelemetry-api==1.20.0
- opentelemetry-instrumentation==0.41b0
- opentelemetry-instrumentation-aiohttp-client==0.41b0
- opentelemetry-instrumentation-asgi==0.41b0
- opentelemetry-sdk==1.20.0
- opentelemetry-semantic-conventions==0.41b0
- opentelemetry-util-http==0.41b0
- opt-einsum==3.3.0
- packaging==23.2
- pathspec==0.11.2
- pillow==10.0.1
- pip-requirements-parser==32.0.1
- pip-tools==7.3.0
- prometheus-client==0.17.1
- protobuf==4.24.4
- psutil==5.9.5
- pyasn1==0.5.0
- pyasn1-modules==0.3.0
- pydantic==2.4.2
- pydantic-core==2.10.1
- pygments==2.16.1
- pynvml==11.5.0
- pyparsing==3.1.1
- pyproject-hooks==1.0.0
- python-dateutil==2.8.2
- python-json-logger==2.0.7
- python-multipart==0.0.6
- pyyaml==6.0.1
- pyzmq==25.1.1
- regex==2023.10.3
- requests==2.31.0
- requests-oauthlib==1.3.1
- rich==13.6.0
- rsa==4.9
- safetensors==0.4.0
- schema==0.7.5
- scikit-learn==1.3.1
- scipy==1.11.3
- simple-di==0.1.5
- six==1.16.0
- sniffio==1.3.0
- starlette==0.31.1
- sympy==1.12
- tensorboard==2.14.1
- tensorboard-data-server==0.7.1
- tensorflow==2.14.0
- tensorflow-estimator==2.14.0
- tensorflow-io-gcs-filesystem==0.34.0
- tensorflow-macos==2.14.0
- termcolor==2.3.0
- threadpoolctl==3.2.0
- tokenizers==0.14.1
- tomli==2.0.1
- torch==2.1.0
- torchvision==0.16.0
- tornado==6.3.3
- tqdm==4.66.1
- transformers==4.34.0
- typing-extensions==4.8.0
- urllib3==2.0.6
- uvicorn==0.23.2
- watchfiles==0.20.0
- werkzeug==3.0.0
- wrapt==1.14.1
- xgboost==2.0.0
- yarl==1.9.2
- zipp==3.17.0
prefix: /Users/user/miniconda3/envs/bento_39
pip_packages
absl-py==2.0.0
aiohttp==3.8.6
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.0.0
appdirs==1.4.4
asgiref==3.7.2
astunparse==1.6.3
async-timeout==4.0.3
attrs==23.1.0
bentoml==1.1.7
build==1.0.3
cachetools==5.3.1
cattrs==23.1.2
certifi==2023.7.22
charset-normalizer==3.3.0
circus==0.18.0
click==8.1.7
click-option-group==0.5.6
cloudpickle==2.2.1
contextlib2==21.6.0
deepmerge==1.1.0
Deprecated==1.2.14
exceptiongroup==1.1.3
filelock==3.12.4
flatbuffers==23.5.26
frozenlist==1.4.0
fs==2.4.16
fsspec==2023.9.2
gast==0.5.4
google-auth==2.23.3
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
grpcio==1.59.0
h11==0.14.0
h5py==3.10.0
httpcore==0.18.0
httpx==0.25.0
huggingface-hub==0.17.3
idna==3.4
importlib-metadata==6.0.1
inflection==0.5.1
Jinja2==3.1.2
joblib==1.3.2
keras==2.14.0
libclang==16.0.6
Markdown==3.5
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mdurl==0.1.2
ml-dtypes==0.2.0
mpmath==1.3.0
multidict==6.0.4
networkx==3.1
numpy==1.26.0
oauthlib==3.2.2
opentelemetry-api==1.20.0
opentelemetry-instrumentation==0.41b0
opentelemetry-instrumentation-aiohttp-client==0.41b0
opentelemetry-instrumentation-asgi==0.41b0
opentelemetry-sdk==1.20.0
opentelemetry-semantic-conventions==0.41b0
opentelemetry-util-http==0.41b0
opt-einsum==3.3.0
packaging==23.2
pathspec==0.11.2
Pillow==10.0.1
pip-requirements-parser==32.0.1
pip-tools==7.3.0
prometheus-client==0.17.1
protobuf==4.24.4
psutil==5.9.5
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydantic==2.4.2
pydantic_core==2.10.1
Pygments==2.16.1
pynvml==11.5.0
pyparsing==3.1.1
pyproject_hooks==1.0.0
python-dateutil==2.8.2
python-json-logger==2.0.7
python-multipart==0.0.6
PyYAML==6.0.1
pyzmq==25.1.1
regex==2023.10.3
requests==2.31.0
requests-oauthlib==1.3.1
rich==13.6.0
rsa==4.9
safetensors==0.4.0
schema==0.7.5
scikit-learn==1.3.1
scipy==1.11.3
simple-di==0.1.5
six==1.16.0
sniffio==1.3.0
starlette==0.31.1
sympy==1.12
tensorboard==2.14.1
tensorboard-data-server==0.7.1
tensorflow==2.14.0
tensorflow-estimator==2.14.0
tensorflow-io-gcs-filesystem==0.34.0
tensorflow-macos==2.14.0
termcolor==2.3.0
threadpoolctl==3.2.0
tokenizers==0.14.1
tomli==2.0.1
torch==2.1.0
torchvision==0.16.0
tornado==6.3.3
tqdm==4.66.1
transformers==4.34.0
typing_extensions==4.8.0
urllib3==2.0.6
uvicorn==0.23.2
watchfiles==0.20.0
Werkzeug==3.0.0
wrapt==1.14.1
xgboost==2.0.0
yarl==1.9.2
zipp==3.17.0
As some additional notes to the above issue, it definitely seems to be memory issue, we were also facing the same issue and could resolve it for now by adding more memory to the container. In our case, we are deploying in ECS and running multiple models (i.e. multiple runners) in each EC2 instance.
As some additional notes to the above issue, it definitely seems to be memory issue, we were also facing the same issue and could resolve it for now by adding more memory to the container. In our case, we are deploying in ECS and running multiple models (i.e. multiple runners) in each EC2 instance.
Thanks for your finding! Do you find the memory slowly building up over time? Can you share a graph of memory consumption with respect to request volume?
Thanks @sudeepg545 ! I will look into this POV. Will update on this @jianshen92 .
The memory issue has been in recent releases, can you please have a try? @nadimintikrish @sudeepg545
