NVIDIA
Milvus failed to connect to the default network defined in compose-rendered.yaml
Hello,
I have a NVWB local installation on a Ubuntu 22.04 Desktop machine with 1x L40S GPU and the latest CUDA 12.8 and GPU driver 570.124.06
I cloned the latest nim-anywhere on 3/4 and can start all applications successfully.
However, after a server reboot, the Environment failed to start, even after a clean rebuild.
Below is the error from the Output/Compose window.
Container nim-anywhere-nvwb-init-service-1 Created Container nim-anywhere-redis-1 Created Container nim-anywhere-milvus-1 Created Attaching to milvus-1, nvwb-init-service-1, redis-1
nvwb-init-service-1 exited with code 0 Gracefully stopping... (press Ctrl+C again to force) Error response from daemon: network 26e2d4ff4f36db70b6f2ac72991959c28a8b4df190d8e6dd7a345f0100a04c79 not found
From Output/Chain Server window:
INFO: Will watch for changes in these directories: ['/project/code']
INFO: Uvicorn running on http://0.0.0.0:3030 (Press CTRL+C to quit)
INFO: Started reloader process [275] using WatchFiles
2025-03-05 19:37:02,777 [ERROR][_create_connection]: Failed to create new connection using: cc45c2256b0e4b4dad5f6310ea8864e8 (milvus_client.py:918)
Process SpawnProcess-1:
return _bootstrap._gcd_import(name[level:], package, level)
File "
The error can be replicated using the upload-pdfs jupyter notebook.
from langchain_milvus.vectorstores.milvus import Milvus print(config.milvus); #url='http://milvus:19530' collection_name='collection_1'
vector_store = Milvus( embedding_function=embedding_model, connection_args={"uri": config.milvus.url}, collection_name=config.milvus.collection_name, auto_id=True, )
Just to verify that I can launch the Milvus independently ie outside NVWB client.
user@user-as-4125gs-tnrt:~/nvidia-workbench/NVIDIA-nim-anywhere$docker compose up milvus
user@user-as-4125gs-tnrt:~/nvidia-workbench/NVIDIA-nim-anywhere$docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS nvidia-nim-anywhere-milvus-1 milvusdb/milvus:v2.4.6 "/tini -- milvus run…" milvus 9 minutes ago Up 2 minutes (healthy)
user@user-as-4125gs-tnrt:~/nvidia-workbench/NVIDIA-nim-anywhere$ docker compose exec milvus curl http://localhost:9091/healthz OK(base)
Hi there,
I'm happy to help get this issue resolved.
I'm unable to replicate this issue. It seems like the compose is shutting down based on
Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: network 26e2d4ff4f36db70b6f2ac72991959c28a8b4df190d8e6dd7a345f0100a04c79 not found
Are you using one of the docker compose profiles? If so, which one? Can you verify the exact steps you took, starting with cloning the project, to produce this issue?
Hello - I have figured out the root cause.
- To replicate, I navigate the the project folder and use docker compose to bring up the milvus service defined in the compose-rendered.yaml file.
$ docker compose -f compose-rendered.yaml up milvus [+] Running 2/0 ✔ Container nim-anywhere-nvwb-init-service-1 Created 0.0s ✔ Container nim-anywhere-milvus-1 Created 0.0s Attaching to milvus-1 Gracefully stopping... (press Ctrl+C again to force) Error response from daemon: network 9c84bd78ebddea06d96faabc6a47a2070f85b8c80786ff984d29640ce86f7fb0 not found
There is no docker network with the id 9c84bd78eb.
$ docker network ls NETWORK ID NAME DRIVER SCOPE d32aa2191925 bridge bridge local 495d39fe01a3 host host local 0f8e78f89479 nim-anywhere_default bridge local 7402aefc60a2 none null local 284f7f3cba4a nvidia-nim-anywhere_default bridge local 766e591ebad5 workbench bridge local
-
Upon inspecting the compose-rendered.yaml file, the network section under "milvus" service has the following definition:
networks: default: null workbench: {}
I have attached my compose-rendered.yaml file.
-
I can change the networks setting to use workbench explicitly.
networks:
- workbench
Then "docker compose -f compose-rendered.yaml up milvus" works fine.
What is interesting is that the missing network with id 9c84bd78ebddea06d96faabc6a47a2070f85b8c80786ff984d29640ce86f7fb0 is from the previous run of NVWB app. So seems the NVWB is somehow using an outdated "workbench" network. Or, the "workbench" somehow got regenerated when NVWB app is running.
$ docker compose -f compose-rendered.yaml up milvus [+] Running 2/0 ✔ Container nim-anywhere-nvwb-init-service-1 Created 0.0s ✔ Container nim-anywhere-milvus-1 Created 0.0s Attaching to milvus-1 Gracefully stopping... (press Ctrl+C again to force) Error response from daemon: network 26e2d4ff4f36db70b6f2ac72991959c28a8b4df190d8e6dd7a345f0100a04c79 not found (nvwb:local/nim-anywhere) (base) user@user-as-4125gs-tnrt:~/.nvwb/project-runtime-info/NVIDIA-nim-anywhere-f65c90dd741f9068b1fcf6ef1e89eed95a31af39$ less compose-rendered.yaml (nvwb:local/nim-anywhere) (base) user@user-as-4125gs-tnrt:~/.nvwb/project-runtime-info/NVIDIA-nim-anywhere-f65c90dd741f9068b1fcf6ef1e89eed95a31af39$ docker network ls NETWORK ID NAME DRIVER SCOPE ce873194bdeb bridge bridge local 495d39fe01a3 host host local 0f8e78f89479 nim-anywhere_default bridge local 7402aefc60a2 none null local 284f7f3cba4a nvidia-nim-anywhere_default bridge local 9c84bd78ebdd workbench bridge local
My fix now is to use the "Edit compose file" feature of NVWB app to change such network setting for "milvus".