[Issue] Tensor mismatch at Runtime

[RatraSahil]

RuntimeError: The expanded size of the tensor (628) must match the existing size (129) at non-singleton dimension 3. Target sizes: [1, 32, 1, 628]. Tensor sizes: [1, 1, 1, 129]

Issue: Upon running the demo colab notebook here, up until yesterday, I was able to get the expected results, but somehow since today the code throws the below mentioned error :

RuntimeError: The expanded size of the tensor (628) must match the existing size (129) at non-singleton dimension 3. Target sizes: [1, 32, 1, 628]. Tensor sizes: [1, 1, 1, 129]

The error gets triggered upon calling the following piece of code :

generate_ids = model.generate(**inputs, max_new_tokens=200)

Could you please have a look and possibly explain what causes the sudden mismatch? Please note that the example instances used remain unchanged, i.e, the code is trhowing the tensor mismatch error even with the example included in the demo notebook.

[mu-cai]

Hi Ratra,

Thanks for bringing this up. I have not made any changes to the clip or llm part since last December. Can you check whether the package version, etc, matches?

Thanks you Mu

[RatraSahil]

Hi!

Thank you for your prompt response! Could you please share the versions of the dependencies you used to debvelop your code? Because the demo colab notebook keeps throwing the same tensor shape mismatch error, even when replicated locally. I am suspecting that one or more of the dependencies has been updated to a never version, and the current code is not compatible with it.

Thank you in advance, Sahil

[mu-cai]

Here is my package info!

Package Version Editable project location

---

accelerate 0.21.0 aiofiles 23.2.1 aiohttp 3.9.1 aiosignal 1.3.1 altair 5.2.0 annotated-types 0.6.0 anyio 3.7.1 async-timeout 4.0.3 attrs 23.1.0 bitsandbytes 0.41.0 certifi 2023.11.17 charset-normalizer 3.3.2 click 8.1.7 cmake 3.27.9 colorama 0.4.6 contourpy 1.2.0 cycler 0.12.1 deepspeed 0.14.0 einops 0.6.1 einops-exts 0.0.4 exceptiongroup 1.2.0 fastapi 0.104.1 ffmpy 0.3.1 filelock 3.13.1 fonttools 4.46.0 frozenlist 1.4.0 fsspec 2023.12.0 gradio 4.9.1 gradio_client 0.7.3 h11 0.14.0 hjson 3.1.0 httpcore 0.17.3 httpx 0.24.0 huggingface-hub 0.19.4 idna 3.6 importlib-resources 6.1.1 Jinja2 3.1.2 joblib 1.3.2 jsonschema 4.20.0 jsonschema-specifications 2023.11.2 kiwisolver 1.4.5 linkify-it-py 2.0.2 lit 17.0.6 markdown-it-py 2.2.0 markdown2 2.4.11 MarkupSafe 2.1.3 matplotlib 3.8.2 mdit-py-plugins 0.3.3 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.4 networkx 3.2.1 ninja 1.11.1.1 numpy 1.26.2 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusparse-cu11 11.7.4.91 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 openai 0.28.0 orjson 3.9.10 packaging 23.2 pandas 2.1.3 peft 0.4.0 Pillow 10.1.0 pip 23.3.1 psutil 5.9.6 py-cpuinfo 9.0.0 pydantic 2.5.2 pydantic_core 2.14.5 pydub 0.25.1 Pygments 2.17.2 pynvml 11.5.0 pyparsing 3.1.1 python-dateutil 2.8.2 python-multipart 0.0.6 pytz 2023.3.post1 PyYAML 6.0.1 referencing 0.31.1 regex 2023.10.3 requests 2.31.0 rich 13.7.0 rpds-py 0.13.2 safetensors 0.4.1 scikit-learn 1.2.2 scipy 1.11.4 semantic-version 2.10.0 sentencepiece 0.1.99 setuptools 68.0.0 shellingham 1.5.4 shortuuid 1.0.11 six 1.16.0 sniffio 1.3.0 starlette 0.27.0 svgwrite 1.4.3 sympy 1.12 threadpoolctl 3.2.0 timm 0.6.13 tokenizers 0.13.3 tomlkit 0.12.0 toolz 0.12.0 torch 2.0.1 torchvision 0.15.2 tqdm 4.66.1 transformers 4.31.0 triton 2.0.0 typer 0.9.0 typing_extensions 4.8.0 tzdata 2023.3 uc-micro-py 1.0.2 urllib3 2.1.0 uvicorn 0.24.0.post1 vip-llava 1.1.3 code/ViP-LLaVA wavedrom 2.0.3.post3 websockets 11.0.3 wheel 0.41.2 yarl 1.9.3

[RatraSahil]

Hi!

Thank you so much for your reply! Just to update you on this issue, Huggingface updated the Transformers library to v4.39.1, and incrementally above. The updated versions are leading to the tensor mismatch issues in the code. The presented Colab notebook works as expected when used with transformers v4.39.0.

Hope that helps anyone who comes across this same issue. :)

Best, Sahil

[JustinKai0527]

@RatraSahil can you run this github locally, I install the command but get different verision mention in issue, thx!