Weired unexpected pydantic error with certain models

[ElyasMoshirpanahi]

I tried to deploy this model https://github.com/zsxkib/InstantID

using cog today and it failed throwing me a pydantic error :

cannot import VaildtorError from pydantic ,

python version 3.11.9 cog version 0.9.5 pydantic vesion 1.10.15

Important note: I used to be able to deploy and build this model without any problem , this new problem occurred 3 days ago (April 19) I pulled the image from cog and I could push it again to replicate however after extracting it to my local env and trying to deploy the same code and structure it failed

Any help or clue is appreciated

[mattt]

Hi @ElyasMoshirpanahi. Thanks for reporting. I can't think of anything that changed in Cog that could've caused that. Can you please share the command you ran and the full output?

[ElyasMoshirpanahi]

Hi @mattt here is my full log:

[+] Building 106.0s (6/21) docker:default => [internal] load build context 102.6s [+] Building 106.2s (6/21) docker:default => [internal] load build context 102.7s [+] Building 106.3s (6/21) docker:default => [internal] load build context 102.9s [+] Building 106.5s (6/21) docker:default => => transferring context: 4.83GB 103.0s [+] Building 744.5s (23/23) FINISHED docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 2.38kB 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.1s => resolve image config for docker.io/docker/dockerfile:1.4 1.2s => docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.5s => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB 0.0s => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB 0.2s => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB 0.0s => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B 0.0s => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 0.1s => [internal] load metadata for docker.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 1.3s => [internal] load metadata for docker.io/library/python:3.11 1.4s => [internal] load build context 275.5s => => transferring context: 10.32GB 275.3s => [stage-1 1/9] FROM docker.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04@sha256:8f9dd0d09d3ad3900357a1cf7f887888b5b74056636cd6ef03c160c3cd4b1d95 174.3s => => resolve docker.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04@sha256:8f9dd0d09d3ad3900357a1cf7f887888b5b74056636cd6ef03c160c3cd4b1d95 0.1s => => sha256:bd746eb3b9953805ebe644847a227e218b5da775f47007c69930569a75c9ad7d 2.84kB / 2.84kB 0.0s => => sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e 29.54MB / 29.54MB 0.4s => => sha256:5e3b7ee7738140e8f4608c3945b6e1ed4f9fb75db53a04e19ba0a6661e7cc4fe 4.62MB / 4.62MB 0.4s => => sha256:5bd037f007fdda13ae5a5f43a199d6677db1f9059c2980c84726e3a43fab169a 56.23MB / 56.23MB 0.7s => => sha256:8f9dd0d09d3ad3900357a1cf7f887888b5b74056636cd6ef03c160c3cd4b1d95 743B / 743B 0.0s => => sha256:d0117ee15b5fd0bbcb42c8fd3e35f9bc0f06fe3a947a4ec240f9b73738c7cf54 17.79kB / 17.79kB 0.0s => => extracting sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e 1.0s => => sha256:4cda774ad2ecef28c9a1cd97594f7199071c83769f91c5d109eb1cb6770ecdff 188B / 188B 0.7s => => sha256:775f22adee620daec0db645bad7027db4c1ecf22520412e1b2466fc73d54d19b 6.88kB / 6.88kB 0.8s => => sha256:16c36d0187d03bd0de84d870ded86c45fabd78f4bfdb2ed90177e5fc4dd33d11 63.70kB / 63.70kB 0.9s => => sha256:263fc748118f7937f811e3e9c9355318db07dd2dd1dccc370dadaa7d0b5ed692 1.38GB / 1.38GB 22.8s => => sha256:e7a56570655c990ecc804c77873efc83f9a6c31064e3e8a5dc02430213f2d74c 1.69kB / 1.69kB 0.9s => => sha256:23b7d8e07c16707ff4ec3ca558a8099c454953c840156c318a60a6b4273846a0 2.46GB / 2.46GB 51.1s => => sha256:507fc9045cbad45c1c4ca554a6453fe0a1c9ae74667db0612fec7475256d5c23 1.52kB / 1.52kB 1.0s => => sha256:922ac8fcb88926d95550e82f83c14a4f3f3eaab635e7acf43ee0c59dea0c14d7 88.23kB / 88.23kB 1.2s => => sha256:68075f2beca1cfd3f243ec110000716dff39d895f4d5e0d3faba7ace430f9633 1.43GB / 1.43GB 41.2s => => extracting sha256:5e3b7ee7738140e8f4608c3945b6e1ed4f9fb75db53a04e19ba0a6661e7cc4fe 0.2s => => extracting sha256:5bd037f007fdda13ae5a5f43a199d6677db1f9059c2980c84726e3a43fab169a 1.4s => => extracting sha256:4cda774ad2ecef28c9a1cd97594f7199071c83769f91c5d109eb1cb6770ecdff 0.0s => => extracting sha256:775f22adee620daec0db645bad7027db4c1ecf22520412e1b2466fc73d54d19b 0.0s => => extracting sha256:263fc748118f7937f811e3e9c9355318db07dd2dd1dccc370dadaa7d0b5ed692 22.1s => => extracting sha256:16c36d0187d03bd0de84d870ded86c45fabd78f4bfdb2ed90177e5fc4dd33d11 0.0s => => extracting sha256:e7a56570655c990ecc804c77873efc83f9a6c31064e3e8a5dc02430213f2d74c 0.0s => => extracting sha256:507fc9045cbad45c1c4ca554a6453fe0a1c9ae74667db0612fec7475256d5c23 0.0s => => extracting sha256:23b7d8e07c16707ff4ec3ca558a8099c454953c840156c318a60a6b4273846a0 51.3s => => extracting sha256:922ac8fcb88926d95550e82f83c14a4f3f3eaab635e7acf43ee0c59dea0c14d7 0.0s => => extracting sha256:68075f2beca1cfd3f243ec110000716dff39d895f4d5e0d3faba7ace430f9633 13.3s => [deps 1/5] FROM docker.io/library/python:3.11@sha256:db07fba48daaf1c68c03676aadc73866414d25b4c278029f9873c784517613bf 142.0s => => resolve docker.io/library/python:3.11@sha256:db07fba48daaf1c68c03676aadc73866414d25b4c278029f9873c784517613bf 0.1s => => sha256:db07fba48daaf1c68c03676aadc73866414d25b4c278029f9873c784517613bf 2.14kB / 2.14kB 0.0s => => sha256:41936fec553de22d1de0c450fcbf8f6b9486508f27329073204455a252d916f7 7.32kB / 7.32kB 0.0s => => sha256:21d1a3d86b1656a745a9ccfdd3a0797a694c9d48450e65734127183506a38e8c 2.01kB / 2.01kB 0.0s => => sha256:609c73876867487da051ad470002217da69bb052e2538710ade0730d893ff51f 49.56MB / 49.56MB 51.5s => => sha256:7247ea8d81e671d079d67f3a9909315ef4641b45db90d62a1b18e3430c1937d4 24.05MB / 24.05MB 51.2s => => sha256:b4580645a8e50b87a19330da289a9b1540022379f2c99d3f0112e3c5c4a8d051 211.14MB / 211.14MB 56.5s => => sha256:aa7e0aca67ddfc342e2afe83df590a0e228e34781e7206639ddd9da72a71a0af 6.39MB / 6.39MB 51.7s => => extracting sha256:609c73876867487da051ad470002217da69bb052e2538710ade0730d893ff51f 7.7s => => sha256:be374d06f38273b62ddd7aa5bc3ce3f9c781fd49a1f5a5dd94a46d2986920d7a 64.14MB / 64.14MB 52.4s => => sha256:928988b537f87dd6ce2dff578fa9dc94138f9497eb6c7d575fad7b65b14bca7a 19.84MB / 19.84MB 52.5s => => sha256:95604f0c57a72f2a5dc8b36c0f35918967c5f347fe0945f7d3bded9e6c639f20 245B / 245B 57.2s => => sha256:af495b5540acf407853862e768a57adb5e020fc219969662e8941f0d610b74de 3.13MB / 3.13MB 57.3s => => extracting sha256:7247ea8d81e671d079d67f3a9909315ef4641b45db90d62a1b18e3430c1937d4 0.6s => => extracting sha256:be374d06f38273b62ddd7aa5bc3ce3f9c781fd49a1f5a5dd94a46d2986920d7a 2.2s => => extracting sha256:b4580645a8e50b87a19330da289a9b1540022379f2c99d3f0112e3c5c4a8d051 6.1s => => extracting sha256:aa7e0aca67ddfc342e2afe83df590a0e228e34781e7206639ddd9da72a71a0af 0.6s => => extracting sha256:928988b537f87dd6ce2dff578fa9dc94138f9497eb6c7d575fad7b65b14bca7a 0.7s => => extracting sha256:95604f0c57a72f2a5dc8b36c0f35918967c5f347fe0945f7d3bded9e6c639f20 0.0s => => extracting sha256:af495b5540acf407853862e768a57adb5e020fc219969662e8941f0d610b74de 0.3s => [stage-1 2/9] RUN --mount=type=cache,target=/var/cache/apt set -eux; apt-get update -qq; apt-get install -qqy --no-install-recommends curl; rm -rf /var/lib/apt/lists/; TINI_VERSION=v0.19.0; TINI_ARCH=" 6.9s => [stage-1 3/9] RUN --mount=type=cache,target=/var/cache/apt apt-get update -qq && apt-get install -qqy --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev 24.7s => [stage-1 4/9] RUN curl -s -S -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash && git clone https://github.com/momo-lab/pyenv-install-latest.git "$(pyenv roo 71.6s => [deps 2/5] COPY .cog/tmp/build1197687401/cog-0.0.1.dev-py3-none-any.whl /tmp/cog-0.0.1.dev-py3-none-any.whl 2.0s => [stage-1 5/9] RUN --mount=type=cache,target=/var/cache/apt apt-get update -qq && apt-get install -qqy libgl1-mesa-glx libglib2.0-0 && rm -rf /var/lib/apt/lists/ 12.5s => [deps 3/5] RUN --mount=type=cache,target=/root/.cache/pip pip install -t /dep /tmp/cog-0.0.1.dev-py3-none-any.whl 6.9s => [deps 4/5] COPY .cog/tmp/build1197687401/requirements.txt /tmp/requirements.txt 0.2s => [deps 5/5] RUN --mount=type=cache,target=/root/.cache/pip pip install -t /dep -r /tmp/requirements.txt 163.7s => [stage-1 6/9] RUN --mount=type=bind,from=deps,source=/dep,target=/dep cp -rf /dep/* $(pyenv prefix)/lib/python*/site-packages || true 43.6s => [stage-1 7/9] RUN curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.6.0/pget_linux_x86_64" && chmod +x /usr/local/bin/pget 1.5s => [stage-1 8/9] WORKDIR /src 0.1s => [stage-1 9/9] COPY . /src 66.9s => preparing layers for inline cache 38.6s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:f7c7316643155567ab6221565293e6d293358c91ed351e57426e5784ff13bd82 0.0s => => naming to r8.im/zaubar/simple-juggernaut-xl-v7 0.0s Validating model schema...

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/cog/command/openapi_schema.py", line 46, in raise CogError(app.state.setup_result.logs) cog.errors.CogError: Error while loading predictor:

Traceback (most recent call last): File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/cog/server/http.py", line 130, in create_app predictor = load_slim_predictor_from_ref(predictor_ref, "predict") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/cog/predictor.py", line 233, in load_slim_predictor_from_ref module = load_full_predictor_from_file(module_path, module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/cog/predictor.py", line 193, in load_full_predictor_from_file spec.loader.exec_module(module) File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/src/cog/predict.py", line 37, in from insightface.app import FaceAnalysis File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/insightface/init.py", line 18, in from . import app File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/insightface/app/init.py", line 2, in from .mask_renderer import * File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/insightface/app/mask_renderer.py", line 4, in import albumentations as A File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/albumentations/init.py", line 3, in from .augmentations import * File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/albumentations/augmentations/init.py", line 1, in from .blur.functional import * File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/albumentations/augmentations/blur/init.py", line 1, in from .functional import * File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/albumentations/augmentations/blur/functional.py", line 9, in from albumentations.augmentations.geometric.functional import scale File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/albumentations/augmentations/geometric/init.py", line 2, in from .resize import * File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/albumentations/augmentations/geometric/resize.py", line 6, in from pydantic import Field, ValidationInfo, field_validator ImportError: cannot import name 'ValidationInfo' from 'pydantic' (/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydantic/init.cpython-311-x86_64-linux-gnu.so)

ⅹ Failed to get type signature: exit status 1

Thanks for helping in advance

[ElyasMoshirpanahi]

@mattt Found that this error came from a dependency update of instant id , and to be specific , this library albumentations version is suppose to be 1.4.3 but since there were not version pinning in instant id version 1.4.4 was installed and that caused the error.

I think you can close this issue and the end result would be to pin this dependency in the cog.yaml like this :

build:
  # set to true if your model requires a GPU
  gpu: true
  # cuda: "12.1"
  
  # a list of ubuntu apt packages to install
  system_packages:
    - "libgl1-mesa-glx"
    - "libglib2.0-0"

  # python version in the form '3.11' or '3.11.4'
  python_version: "3.11"

  # a list of packages in the format <package-name>==<version>
  python_packages:
    - "albumentations==1.4.3"
    - "opencv-python==4.9.0.80"
    - "transformers==4.37.0"
    - "accelerate==0.26.1"
    - "insightface==0.7.3"
    - "diffusers==0.25.1"
    - "onnxruntime==1.16.3"
    - "omegaconf==2.3.0"
    - "pandas==2.0.3"
  # commands run after the environment is setup
  run:
    - curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.6.0/pget_linux_x86_64" && chmod +x /usr/local/bin/pget
    - pip freeze
# predict.py defines how predictions are run on your model
predict: "cog/predict.py:Predictor"