OpenLLM
OpenLLM copied to clipboard
bentoml
bug: Missing Dependency when running
Describe the bug
Hello,
I followed the instructions on github (nothing more) and when I try to run it with the following command:
sudo docker run -it --rm -p 3000:3000 google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce serve
I'm getting the following error.
Error: [bentoml-cli] serve failed: Failed loading Bento from directory /home/bentoml/bento: Failed to import module "generated_flan_t5_service": No module named 'orjson'
To reproduce
No response
Logs
`sudo docker run -it --rm -p 3000:3000 google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce serve
`
`Error: [bentoml-cli] `serve` failed: Failed loading Bento from directory /home/bentoml/bento: Failed to import module "generated_flan_t5_service": No module named 'orjson'`
Environment
python:3.9.17 bentoml:1.0.22 openllm:0.1.8
Here is the Docker Build ouput `sudo openllm build flan-t5 --model-id google/flan-t5-xl Packing 'flan-t5' into a Bento with kwargs={}... Building Bento for LLM 'flan-t5' Building Bento for LLM 'flan-t5'
██████╗ ██████╗ ███████╗███╗ ██╗██╗ ██╗ ███╗ ███╗ ██╔═══██╗██╔══██╗██╔════╝████╗ ██║██║ ██║ ████╗ ████║ ██║ ██║██████╔╝█████╗ ██╔██╗ ██║██║ ██║ ██╔████╔██║ ██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║██║ ██║ ██║╚██╔╝██║ ╚██████╔╝██║ ███████╗██║ ╚████║███████╗███████╗██║ ╚═╝ ██║ ╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝╚══════╝╚══════╝╚═╝ ╚═╝
Successfully built Bento(tag="google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce").
Possible next steps:
- Push to BentoCloud with
bentoml push: $ bentoml push google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce - Containerize your Bento with
bentoml containerize: $ bentoml containerize google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce Tip: To enable additional BentoML feature for 'containerize', use '--enable-features=FEATURE[,FEATURE]' [see 'bentoml containerize -h' for more advanced usage]
sudo bentoml containerize google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce Building OCI-compliant image for google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce with docker
[+] Building 300.4s (19/19) FINISHED => [internal] load build definition from Dockerfile 0.5s => => transferring dockerfile: 3.36kB 0.0s => [internal] load .dockerignore 0.5s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/nvidia/cuda:11.6.2-cudnn8-runtime-ubuntu20.04 0.5s => [base-container 1/14] FROM docker.io/nvidia/cuda:11.6.2-cudnn8-runtime-ubuntu20.04@sha256:3e1c8a9754bb530b0 40.2s => => resolve docker.io/nvidia/cuda:11.6.2-cudnn8-runtime-ubuntu20.04@sha256:3e1c8a9754bb530b01c21be43f65fcc4a74 0.0s => => sha256:3e1c8a9754bb530b01c21be43f65fcc4a74c9bec42f61183112e936dda5cbe43 743B / 743B 0.0s => => sha256:462ba0e2badc619e410d3a10f0471056553b71a9c48187920d5ec4b92031d713 2.42kB / 2.42kB 0.0s => => sha256:2e33cbfb80bf58ced106d63a60762db7d436deb6c2cc793c08cca42aa1f4f017 12.77kB / 12.77kB 0.0s => => sha256:ca1778b6935686ad781c27472c4668fc61ec3aeb85494f72deb1921892b9d39e 27.50MB / 27.50MB 0.4s => => sha256:65966f22ed3d88416d069fddf13902e152ae54b6c9a04e2787938f831014f9af 7.94MB / 7.94MB 0.5s => => sha256:ad6b7dd7c65f1dabb00d95a454994d3664d03efbd0d64ae34fef79a3ec25893a 23.61MB / 23.61MB 0.6s => => extracting sha256:ca1778b6935686ad781c27472c4668fc61ec3aeb85494f72deb1921892b9d39e 0.9s => => sha256:037a5c0ebcaab7c1b8cfc3c82745d94c7d394e874f8629a33609c9b46360d75f 184B / 184B 0.8s => => sha256:7921c0ffa124d1669cc033a479b9ef5a4817f57cb1a089caca43a4ef8b5ab6a1 1.12GB / 1.12GB 11.0s => => sha256:ea504443a0386bfe9f1f778e0318641268d763b0415beace78b81b49cee43262 6.89kB / 6.89kB 1.0s => => sha256:ca95f35017dce66c17b318dd49cf041bfe6b4705e84014031c4aa5058fa14d57 62.68kB / 62.68kB 0.9s => => sha256:651795e98cbbf0ffe858eb5000428947c028af902bb9ef0410a7da9b9488dc69 1.68kB / 1.68kB 1.0s => => sha256:b753b50441b1c0c7c235e859ba57459a755d421b828aaecd09b15f77f1c6701d 1.52kB / 1.52kB 1.3s => => sha256:6703542d8e6b9b8e8b19e4f5079960adab08b0a7026fa541bd72a11b51b8d588 727.16MB / 727.16MB 9.2s => => extracting sha256:65966f22ed3d88416d069fddf13902e152ae54b6c9a04e2787938f831014f9af 0.3s => => extracting sha256:ad6b7dd7c65f1dabb00d95a454994d3664d03efbd0d64ae34fef79a3ec25893a 0.6s => => extracting sha256:037a5c0ebcaab7c1b8cfc3c82745d94c7d394e874f8629a33609c9b46360d75f 0.0s => => extracting sha256:ea504443a0386bfe9f1f778e0318641268d763b0415beace78b81b49cee43262 0.0s => => extracting sha256:7921c0ffa124d1669cc033a479b9ef5a4817f57cb1a089caca43a4ef8b5ab6a1 16.3s => => extracting sha256:ca95f35017dce66c17b318dd49cf041bfe6b4705e84014031c4aa5058fa14d57 0.0s => => extracting sha256:651795e98cbbf0ffe858eb5000428947c028af902bb9ef0410a7da9b9488dc69 0.0s => => extracting sha256:b753b50441b1c0c7c235e859ba57459a755d421b828aaecd09b15f77f1c6701d 0.0s => => extracting sha256:6703542d8e6b9b8e8b19e4f5079960adab08b0a7026fa541bd72a11b51b8d588 10.7s => [internal] load build context 66.7s => => transferring context: 11.40GB 66.7s => [base-container 2/14] RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Pa 3.6s => [base-container 3/14] RUN --mount=type=cache,target=/var/lib/apt --mount=type=cache,target=/var/cache/apt s 26.5s => [base-container 4/14] RUN --mount=type=cache,target=/var/lib/apt --mount=type=cache,target=/var/cache/apt 55.7s => [base-container 5/14] RUN ln -sf /usr/bin/python3.9 /usr/bin/python3 && ln -sf /usr/bin/pip3.9 /usr/bin/ 0.4s => [base-container 6/14] RUN curl -O https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && rm - 8.9s => [base-container 7/14] RUN groupadd -g 1034 -o bentoml && useradd -m -u 1034 -g 1034 -o -r bentoml 0.5s => [base-container 8/14] RUN mkdir /home/bentoml/bento && chown bentoml:bentoml /home/bentoml/bento -R 0.3s => [base-container 9/14] WORKDIR /home/bentoml/bento 0.0s => [base-container 10/14] COPY --chown=bentoml:bentoml ./env/python ./env/python/ 0.0s => [base-container 11/14] RUN --mount=type=cache,target=/root/.cache/pip bash -euxo pipefail /home/bentoml/bent 84.3s => [base-container 12/14] COPY --chown=bentoml:bentoml . ./ 31.4s => [base-container 13/14] RUN rm -rf /var/lib/{apt,cache,log} 0.3s => [base-container 14/14] RUN chmod +x /home/bentoml/bento/env/docker/entrypoint.sh 0.4s => exporting to image 46.7s => => exporting layers 46.7s => => writing image sha256:64188b623d999137c14601d043c4f5fdd5d66416d5799070e9f8c4745b7c4e7a 0.0s => => naming to docker.io/library/google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce 0.0s Successfully built Bento container for "google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce" with tag(s) "google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce" To run your newly built Bento container, run: docker run -it --rm -p 3000:3000 google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce serve`
You shouldn't run openllm with sudo first and foremost, try again without sudo
This seems like the weirdest bug to me. orjson is a core dependency and should already include during install openllm in the container?
Would you mind running bentoml containerize ... --opt progress=plain and dump all of the output to a text file and send it here? Thanks.
I discover a bug on main and I will release a new version patch version promptly
you need to provide bentoml containerize --opt no-cache to remove the previous cache layer. It will still use the previous cache hit afaik
Still the same issue. We installed everything from scratch on another vm as well, doesn’t work
Hey there, I have identified the issue and will have a patch release soon. sorry for the trouble
I was having the same error. Upgraded to latest version (openllm, version 0.1.17). Build step with --overwrite and containerize step with --opt no-cache. It was solved.
