[ChatQnA] TEI 1.5 causes an error starting the tei-reranking-server

[wsfowler]

I was updating my Terraform/Ansible recipe this morning and I ran into this error while trying to start the example using the provided compose.yaml file.

Error: Could not create backend

Caused by:
    Could not start backend: Failed to create ONNX Runtime session: Deserialize tensor onnx::MatMul_3121 failed.GetFileLength for /data/models--BAAI--bge-reranker-large/snapshots/55611d7bca2a7133960a6d3b71e083071bbfc312/onnx/model.onnx_data failed:Invalid fd was supplied: -1

The reranking container then exits.

CONTAINER ID   IMAGE                                                   COMMAND                  CREATED          STATUS                      PORTS                                                                                  NAMES
ff5e425f0098   opea/chatqna-ui:latest                                  "docker-entrypoint.s…"   11 minutes ago   Up 11 minutes               0.0.0.0:5173->5173/tcp, :::5173->5173/tcp                                              chatqna-xeon-ui-server
db9e4993a00c   opea/chatqna:latest                                     "python chatqna.py"      11 minutes ago   Up 11 minutes               0.0.0.0:8888->8888/tcp, :::8888->8888/tcp                                              chatqna-xeon-backend-server
57845af811cd   opea/reranking-tei:latest                               "python reranking_te…"   11 minutes ago   Up 11 minutes               0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                                              reranking-tei-xeon-server
82d83238b4fe   opea/llm-tgi:latest                                     "bash entrypoint.sh"     11 minutes ago   Up 11 minutes               0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                                              llm-tgi-server
7347b2fb4a3b   opea/embedding-tei:latest                               "python embedding_te…"   11 minutes ago   Up 11 minutes               0.0.0.0:6000->6000/tcp, :::6000->6000/tcp                                              embedding-tei-server
97ef7cfeb315   opea/dataprep-redis:latest                              "python prepare_doc_…"   11 minutes ago   Up 11 minutes               0.0.0.0:6007-6009->6007-6009/tcp, :::6007-6009->6007-6009/tcp                          dataprep-redis-server
12c84f7e9667   opea/retriever-redis:latest                             "/home/user/comps/re…"   11 minutes ago   Up 11 minutes               0.0.0.0:7000->7000/tcp, :::7000->7000/tcp                                              retriever-redis-server
c573346d18c1   redis/redis-stack:7.2.0-v9                              "/entrypoint.sh"         11 minutes ago   Up 11 minutes               0.0.0.0:6379->6379/tcp, :::6379->6379/tcp, 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp   redis-vector-db
c7d1c7f0763d   ghcr.io/huggingface/text-generation-inference:2.1.0     "/tgi-entrypoint.sh …"   11 minutes ago   Up 11 minutes               0.0.0.0:9009->80/tcp, :::9009->80/tcp                                                  tgi-service
3ac952569089   ghcr.io/huggingface/text-embeddings-inference:cpu-1.5   "text-embeddings-rou…"   11 minutes ago   Exited (1) 10 minutes ago                                                                                          tei-reranking-server
47f803c58222   ghcr.io/huggingface/text-embeddings-inference:cpu-1.5   "text-embeddings-rou…"   11 minutes ago   Up 11 minutes               0.0.0.0:6006->80/tcp, :::6006->80/tcp                                                  tei-embedding-server

I found the following issue on the TEI repo: https://github.com/huggingface/text-embeddings-inference/issues/341

If I switch the reranking service to use 1.4, then it starts correctly and the example works.

root@ip-172-31-29-103:/opt/GenAIExamples/ChatQnA/docker/xeon# docker ps -a
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED         STATUS         PORTS                                                                                  NAMES
03154665e268   opea/chatqna-ui:latest                                  "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes   0.0.0.0:5173->5173/tcp, :::5173->5173/tcp                                              chatqna-xeon-ui-server
319d28eeb771   opea/chatqna:latest                                     "python chatqna.py"      2 minutes ago   Up 2 minutes   0.0.0.0:8888->8888/tcp, :::8888->8888/tcp                                              chatqna-xeon-backend-server
3a0ca8da9838   opea/llm-tgi:latest                                     "bash entrypoint.sh"     2 minutes ago   Up 2 minutes   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                                              llm-tgi-server
3a78c4595faa   opea/reranking-tei:latest                               "python reranking_te…"   2 minutes ago   Up 2 minutes   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                                              reranking-tei-xeon-server
54dd50950c0d   opea/dataprep-redis:latest                              "python prepare_doc_…"   2 minutes ago   Up 2 minutes   0.0.0.0:6007-6009->6007-6009/tcp, :::6007-6009->6007-6009/tcp                          dataprep-redis-server
ec9fc6be5ddd   opea/retriever-redis:latest                             "/home/user/comps/re…"   2 minutes ago   Up 2 minutes   0.0.0.0:7000->7000/tcp, :::7000->7000/tcp                                              retriever-redis-server
3432e3e8b6a4   opea/embedding-tei:latest                               "python embedding_te…"   2 minutes ago   Up 2 minutes   0.0.0.0:6000->6000/tcp, :::6000->6000/tcp                                              embedding-tei-server
d154ee44dce7   ghcr.io/huggingface/text-generation-inference:2.1.0     "/tgi-entrypoint.sh …"   2 minutes ago   Up 2 minutes   0.0.0.0:9009->80/tcp, :::9009->80/tcp                                                  tgi-service
632f404a5be9   redis/redis-stack:7.2.0-v9                              "/entrypoint.sh"         2 minutes ago   Up 2 minutes   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp, 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp   redis-vector-db
d11a7d3770e7   ghcr.io/huggingface/text-embeddings-inference:cpu-1.4   "text-embeddings-rou…"   2 minutes ago   Up 2 minutes   0.0.0.0:8808->80/tcp, :::8808->80/tcp                                                  tei-reranking-server
ab513f0a9810   ghcr.io/huggingface/text-embeddings-inference:cpu-1.5   "text-embeddings-rou…"   2 minutes ago   Up 2 minutes   0.0.0.0:6006->80/tcp, :::6006->80/tcp                                                  tei-embedding-server