cog
cog copied to clipboard
replicate
Cog build error in ubuntu 22.04 - Python
Hello I'm trying to compile mixtral-ai but failing over over with this issue,
Technical Summary:
Platform: Ubuntu 22.04 Python installed: 3.11.8 Cog: cog version 0.9.4 (built 2024-01-24T22:16:49Z) GItHub LLM models: - https://github.com/nateraw/replicate-examples/tree/main/mixtral-vllm - https://github.com/nateraw/replicate-examples/tree/main/llama-2-70b-chat-awq
Logs:
Building Docker image from environment in cog.yaml as mixtral-ai...
[+] Building 0.9s (24/24) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.38kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.4 0.8s
=> CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s
=> [internal] load metadata for docker.io/nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04 0.0s
=> [internal] load metadata for docker.io/library/python:3.11 0.0s
=> [deps 1/5] FROM docker.io/library/python:3.11 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 90.84kB 0.0s
=> [stage-1 1/10] FROM docker.io/nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04 0.0s
=> CACHED [stage-1 2/10] RUN --mount=type=cache,target=/var/cache/apt set -eux; apt-get update -qq; apt-get install -qqy --no-install-recommends curl; rm -rf /var/lib/apt/lists/*; TINI_VERSION=v0.19.0; TINI_ARCH="$(d 0.0s
=> CACHED [stage-1 3/10] RUN --mount=type=cache,target=/var/cache/apt apt-get update -qq && apt-get install -qqy --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev li 0.0s
=> CACHED [stage-1 4/10] RUN curl -s -S -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash && git clone https://github.com/momo-lab/pyenv-install-latest.git "$(pyenv root)"/ 0.0s
=> CACHED [deps 2/5] COPY .cog/tmp/build3662659224/cog-0.0.1.dev-py3-none-any.whl /tmp/cog-0.0.1.dev-py3-none-any.whl 0.0s
=> CACHED [deps 3/5] RUN --mount=type=cache,target=/root/.cache/pip pip install -t /dep /tmp/cog-0.0.1.dev-py3-none-any.whl 0.0s
=> CACHED [deps 4/5] COPY .cog/tmp/build3662659224/requirements.txt /tmp/requirements.txt 0.0s
=> CACHED [deps 5/5] RUN --mount=type=cache,target=/root/.cache/pip pip install -t /dep -r /tmp/requirements.txt 0.0s
=> CACHED [stage-1 5/10] RUN --mount=type=bind,from=deps,source=/dep,target=/dep cp -rf /dep/* $(pyenv prefix)/lib/python*/site-packages || true 0.0s
=> CACHED [stage-1 6/10] RUN CUDA_HOME=/usr/local/cuda pip install vllm==0.2.5 0.0s
=> CACHED [stage-1 7/10] RUN curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.5.4/pget_linux_x86_64" && chmod +x /usr/local/bin/pget 0.0s
=> CACHED [stage-1 8/10] RUN bash -c 'ln -s /usr/local/lib/python3.11/site-packages/torch/lib/lib{nv,cu}* /usr/lib' 0.0s
=> CACHED [stage-1 9/10] WORKDIR /src 0.0s
=> [stage-1 10/10] COPY . /src 0.0s
=> preparing layers for inline cache 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:ec8d4c70cf02582148c4668e0a660b9ce96049c00610695c4d6da4b9437e6885 0.0s
=> => naming to docker.io/library/mixtral-ai 0.0s
Validating model schema...
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/cog/command/openapi_schema.py", line 21, in <module>
raise CogError(app.state.setup_result.logs)
cog.errors.CogError: Error while loading predictor:
Traceback (most recent call last):
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/cog/server/http.py", line 131, in create_app
predictor = load_predictor_from_ref(predictor_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/cog/predictor.py", line 184, in load_predictor_from_ref
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/src/predict.py", line 5, in <module>
from vllm import AsyncLLMEngine
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/__init__.py", line 3, in <module>
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 6, in <module>
from vllm.config import (CacheConfig, ModelConfig, ParallelConfig,
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/config.py", line 9, in <module>
from vllm.utils import get_cpu_memory, is_hip
File "/root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/utils.py", line 8, in <module>
from vllm._C import cuda_utils
ImportError: /root/.pyenv/versions/3.11.8/lib/python3.11/site-packages/vllm/_C.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE
not yet... just moved to other utils sadly
בתאריך יום ב׳, 1 ביולי 2024 ב-8:47 מאת hongminpark < @.***>:
Did you solved this ?
— Reply to this email directly, view it on GitHub https://github.com/replicate/cog/issues/1567#issuecomment-2199287466, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM452K74JE75LNN72CQ6W53ZKDUOBAVCNFSM6AAAAABEOHEVCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJZGI4DONBWGY . You are receiving this because you authored the thread.Message ID: @.***>
Hi @Mougatsu. Sorry for not responding sooner.
Those example models don't work for me, either. Replicate has an official Cog model for VLLM here: https://github.com/replicate/cog-vllm. Please give that a try instead, and let me know if that doesn't work for you.
