dify docker-weaviate-1 keeps restarting

Self Checks

[x] This is only for bug report, if you would like to ask a question, please head to Discussions.
[x] I have searched for existing issues search for existing issues, including closed ones.
[x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[x] Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I intended to start the Dify service by running docker-compose up -d as normal under 'dify/docker', but the container docker-weaviate-1 keeps restarting and I cannot process uploaded files from the frontend. The logs of docker-weaviate-1 are as follows:

{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2025-02-25T07:33:40Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2025-02-25T07:33:40Z"}
{"action":"memberlist_init","bind_port":7946,"error":"Failed to get final advertise address: No private IP address found, and explicit IP not provided","hostname":"node1","level":"error","msg":"memberlist not created","time":"2025-02-25T07:33:40Z"}
{"action":"startup","error":"create member list: Failed to get final advertise address: No private IP address found, and explicit IP not provided","level":"error","msg":"could not init cluster state","time":"2025-02-25T07:33:40Z"}

I've done a plenty of research across the web but found little helps. I appreciate any advice!

PS: I am running Dify on a Ubuntu server.

✔️ Expected Behavior

docker-weaviate-1 should start and run properly

❌ Actual Behavior

docker-weaviate-1 keeps restarting

Feb 25 '25 07:02 Houlson

@Houlson Hello there! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's work together to solve your issue.

The issue you're experiencing with the docker-weaviate-1 container restarting is likely related to the error message: "Failed to get final advertise address: No private IP address found, and explicit IP not provided." This suggests a network configuration problem where the container is unable to determine its IP address for cluster communication.

Here are some steps you can take to resolve this issue:

Check Network Configuration: Ensure that your Docker network settings are correctly configured. You might need to specify an explicit IP address for the Weaviate container. This can be done by setting the CLUSTER_HOSTNAME environment variable to a valid IP address in your docker-compose.middleware.yaml file.
Docker Network Mode: Consider using a different Docker network mode, such as host mode, which might help in resolving IP address issues. You can set this in your docker-compose.yaml file under the Weaviate service configuration.
Environment Variables: Verify that all necessary environment variables are correctly set in your middleware.env file. Ensure that the WEAVIATE_CLUSTER_HOSTNAME is set to a valid hostname or IP address.
Docker Compose Version: Make sure you are using a compatible version of Docker Compose that supports the features and configurations you are using.
Logs and Debugging: Continue to monitor the logs for any additional error messages that might provide more context. You can increase the verbosity of the logs if needed to get more detailed information.

If these steps do not resolve the issue, you may need to further investigate the network settings on your host machine or consult Docker's documentation for more advanced network troubleshooting techniques.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

Feb 25 '25 08:02 dosubot[bot]

May I ask, have you solved this problem? How did you do it, I had this problem too!

Mar 07 '25 16:03 miaoxiannv

May I ask, have you solved this problem? How did you do it, I had this problem too!

Yes. My team restricted the range of ports available for Docker on our remote Linux server to prevent it from accidentally allocating too many ports. I cannot provide you with more details, but I can return to you next Monday when I get back to office. You can check if there is any similar restriction on Docker network config, or try consult with your team.

Mar 08 '25 01:03 Houlson

May I ask, have you solved this problem? How did you do it, I had this problem too!

Yes. My team restricted the range of ports available for Docker on our remote Linux server to prevent it from accidentally allocating too many ports. I cannot provide you with more details, but I can return to you next Monday when I get back to office. You can check if there is any similar restriction on Docker network config, or try consult with your team.

Thank you very much！

Mar 08 '25 09:03 miaoxiannv

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Mar 10 '25 15:03 miaoxiannv

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Glad that you have solved the most difficult problem! Could you please tell me in which way the database cannot be reached? For example, you can reach it from your local environment, but not from Dify. It would be helpful if you provide the log of the corresponding container.

Mar 10 '25 16:03 Houlson

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Glad that you have solved the most difficult problem! Could you please tell me in which way the database cannot be reached? For example, you can reach it from your local environment, but not from Dify. It would be helpful if you provide the log of the corresponding container.

I was testing the connection to weaivate using curl and found that it was connecting successfully, but on Dify it says that the vector database connection fails!

Mar 10 '25 16:03 miaoxiannv

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Glad that you have solved the most difficult problem! Could you please tell me in which way the database cannot be reached? For example, you can reach it from your local environment, but not from Dify. It would be helpful if you provide the log of the corresponding container.

I was testing the connection to weaivate using curl and found that it was connecting successfully, but on Dify it says that the vector database connection fails!

This error can be caused by many issues. Could you please provide me with a screenshot of the container log?

Mar 10 '25 16:03 Houlson

Mar 10 '25 16:03 miaoxiannv

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Glad that you have solved the most difficult problem! Could you please tell me in which way the database cannot be reached? For example, you can reach it from your local environment, but not from Dify. It would be helpful if you provide the log of the corresponding container.

I was testing the connection to weaivate using curl and found that it was connecting successfully, but on Dify it says that the vector database connection fails!

This error can be caused by many issues. Could you please provide me with a screenshot of the container log?

May I ask if this has anything to do with environment variables. I am using env.example directly in the setup copying to env. appreciate your reply not only

Mar 10 '25 16:03 miaoxiannv

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Glad that you have solved the most difficult problem! Could you please tell me in which way the database cannot be reached? For example, you can reach it from your local environment, but not from Dify. It would be helpful if you provide the log of the corresponding container.

I was testing the connection to weaivate using curl and found that it was connecting successfully, but on Dify it says that the vector database connection fails!

This error can be caused by many issues. Could you please provide me with a screenshot of the container log?

May I ask if this has anything to do with environment variables. I am using env.example directly in the setup copying to env. appreciate your reply not only

The use of .env file should be fine, Weaviate is the default vector DB. Which container does this log come from? If you are using docker compose please inspect the Weaviate and Worker container, the error should be displayed there.

Mar 10 '25 16:03 Houlson

After I modified the network connection method of Weaivate, although Weaivate no longer keeps restarting, the vector database still cannot be connected. Has your team encountered this problem?

Glad that you have solved the most difficult problem! Could you please tell me in which way the database cannot be reached? For example, you can reach it from your local environment, but not from Dify. It would be helpful if you provide the log of the corresponding container.

I was testing the connection to weaivate using curl and found that it was connecting successfully, but on Dify it says that the vector database connection fails!

This error can be caused by many issues. Could you please provide me with a screenshot of the container log?

May I ask if this has anything to do with environment variables. I am using env.example directly in the setup copying to env. appreciate your reply not only

The use of .env file should be fine, Weaviate is the default vector DB. Which container does this log come from? If you are using docker compose please inspect the Weaviate and Worker container, the error should be displayed there.

2025-03-10 18:28:18.849 INFO [Dummy-2] [document_indexing_task.py:59] - Start process document: 928e6281-2ac2-4857-83e8-dca64a68c645 2025-03-10 18:28:25.540 WARNING [Thread-3 (process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/init.py:44: SyntaxWarning: invalid escape sequence '.' re_han_default = re.compile("([\u4E00-\u9FD5a-zA-Z0-9+#&.%-]+)", re.U)

2025-03-10 18:28:25.540 WARNING [Thread-3 (_process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/init.py:46: SyntaxWarning: invalid escape sequence '\s' re_skip_default = re.compile("(\r\n|\s)", re.U)

2025-03-10 18:28:25.548 WARNING [Thread-3 (_process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/finalseg/init.py:78: SyntaxWarning: invalid escape sequence '.' re_skip = re.compile("([a-zA-Z0-9]+(?:.\d+)?%?)")

2025-03-10 18:28:26.043 WARNING [Thread-3 (_process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/posseg/init.py:16: SyntaxWarning: invalid escape sequence '.' re_skip_detail = re.compile("([.0-9]+|[a-zA-Z0-9]+)")

2025-03-10 18:28:26.043 WARNING [Thread-3 (process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/posseg/init.py:17: SyntaxWarning: invalid escape sequence '.' re_han_internal = re.compile("([\u4E00-\u9FD5a-zA-Z0-9+#&.]+)")

2025-03-10 18:28:26.044 WARNING [Thread-3 (_process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/posseg/init.py:18: SyntaxWarning: invalid escape sequence '\s' re_skip_internal = re.compile("(\r\n|\s)")

2025-03-10 18:28:26.044 WARNING [Thread-3 (_process_keyword_index)] [warnings.py:112] - /app/api/.venv/lib/python3.12/site-packages/jieba/posseg/init.py:21: SyntaxWarning: invalid escape sequence '.' re_num = re.compile("[.0-9]+")

2025-03-10 18:28:27.945 WARNING [Thread-3 (_process_keyword_index)] [log.py:232] - Building prefix dict from the default dictionary ... 2025-03-10 18:28:27.944 DEBUG [Thread-3 (_process_keyword_index)] [init.py:113] - Building prefix dict from the default dictionary ... 2025-03-10 18:28:28.868 WARNING [Thread-3 (_process_keyword_index)] [log.py:232] - Dumping model to file cache /tmp/jieba.cache 2025-03-10 18:28:28.868 DEBUG [Thread-3 (_process_keyword_index)] [init.py:146] - Dumping model to file cache /tmp/jieba.cache 2025-03-10 18:28:28.944 WARNING [Thread-3 (_process_keyword_index)] [log.py:232] - Loading model cost 0.998 seconds. 2025-03-10 18:28:28.944 DEBUG [Thread-3 (_process_keyword_index)] [init.py:164] - Loading model cost 0.998 seconds. 2025-03-10 18:28:28.944 WARNING [Thread-3 (_process_keyword_index)] [log.py:232] - Prefix dict has been built successfully. 2025-03-10 18:28:28.944 DEBUG [Thread-3 (_process_keyword_index)] [init.py:166] - Prefix dict has been built successfully. 2025-03-10 18:28:29.560 ERROR [Dummy-2] [indexing_runner.py:96] - consume document failed Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 221, in getaddrinfo addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/gevent/resolver/thread.py", line 63, in getaddrinfo return self.pool.apply(_socket.getaddrinfo, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/gevent/pool.py", line 161, in apply return self.spawn(func, *args, **kwds).get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "src/gevent/event.py", line 330, in gevent._gevent_cevent.AsyncResult.get File "src/gevent/event.py", line 360, in gevent._gevent_cevent.AsyncResult.get File "src/gevent/event.py", line 348, in gevent._gevent_cevent.AsyncResult.get File "src/gevent/event.py", line 328, in gevent._gevent_cevent.AsyncResult._raise_exception File "/app/api/.venv/lib/python3.12/site-packages/gevent/_compat.py", line 50, in reraise raise value.with_traceback(tb) File "/app/api/.venv/lib/python3.12/site-packages/gevent/threadpool.py", line 173, in __run_task thread_result.set(func(*args, **kwargs)) ^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 445, in request self.endheaders() File "/usr/local/lib/python3.12/http/client.py", line 1333, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.12/http/client.py", line 1093, in _send_output self.send(msg) File "/usr/local/lib/python3.12/http/client.py", line 1037, in send self.connect() File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 276, in connect self.sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 205, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x7f4b21d24c50>: Failed to resolve 'weaviate' ([Errno -3] Temporary failure in name resolution)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='weaviate', port=8080): Max retries exceeded with url: /v1/meta (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f4b21d24c50>: Failed to resolve 'weaviate' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/app/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py", line 45, in _init_client client = weaviate.Client( ^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/weaviate/client.py", line 150, in init self._connection = Connection( ^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/weaviate/connect/connection.py", line 654, in init self._server_version = self.get_meta()["version"] ^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/weaviate/connect/connection.py", line 675, in get_meta response = self.get(path="/meta") ^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/weaviate/connect/connection.py", line 516, in get return self._session.get( ^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='weaviate', port=8080): Max retries exceeded with url: /v1/meta (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f4b21d24c50>: Failed to resolve 'weaviate' ([Errno -3] Temporary failure in name resolution)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/app/api/core/indexing_runner.py", line 80, in run self._load( File "/app/api/core/indexing_runner.py", line 570, in _load tokens += future.result() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/indexing_runner.py", line 625, in _process_chunk index_processor.load(dataset, chunk_documents, with_keywords=False) File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 77, in load vector = Vector(dataset) ^^^^^^^^^^^^^^^ File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 35, in init self._vector_processor = self._init_vector() ^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 56, in _init_vector return vector_factory_cls().init_vector(self._dataset, self._attributes, self._embeddings) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py", line 277, in init_vector return WeaviateVector( ^^^^^^^^^^^^^^^ File "/app/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py", line 36, in init self._client = self._init_client(config) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py", line 49, in _init_client raise ConnectionError("Vector database connection error") ConnectionError: Vector database connection error 2025-03-10 18:28:29.576 INFO [Dummy-2] [document_indexing_task.py:76] - Processed dataset: bc3cd266-d04e-499a-a1d0-22ebffbd4721 latency: 10.788108342792839 2025-03-10 18:28:29.597 INFO [Dummy-2] [trace.py:128] - Task tasks.document_indexing_task.document_indexing_task[1f1e7973-40b2-4205-a842-56ca60e1409c] succeeded in 10.809113562107086s: None I got this error once I set weaivate to host, docker-worker-1. I tried without host mode, but found that I couldn't start weaivate properly without host

Mar 10 '25 18:03 miaoxiannv

I have not encountered this problem before, so I am afraid that I probably could not offer you much help. Weaviate being a vector DB is too problematic when setting up so my team switched to Qdrant, and I also heard that Weaviate does not support RAG (unlike Qdrant) from one of the issues under this Github project. I saw you opened a new issue, hope you find a solution there :)

Mar 11 '25 01:03 Houlson

I have not encountered this problem before, so I am afraid that I probably could not offer you much help. Weaviate being a vector DB is too problematic when setting up so my team switched to Qdrant, and I also heard that Weaviate does not support RAG (unlike Qdrant) from one of the issues under this Github project. I saw you opened a new issue, hope you find a solution there :)

Thank you very much for the guidance!

Mar 11 '25 01:03 miaoxiannv

May I ask, have you solved this problem? How did you do it, I had this problem too!

Yes. My team restricted the range of ports available for Docker on our remote Linux server to prevent it from accidentally allocating too many ports. I cannot provide you with more details, but I can return to you next Monday when I get back to office. You can check if there is any similar restriction on Docker network config, or try consult with your team.

Thank you very much！

Hello, can you tell me how fix the problem, I encountered the same problem which is:

dify-weaviate-1  | {"action":"startup","error":"create member list: Failed to get final advertise address: No private IP address found, and explicit IP not provided","level":"error","msg":"could not init cluster state","time":"2025-03-26T05:59:06Z"}

Is the solution to configure certain variables in docker-compose.yml or do we need to modify certain configurations in the Docker software？

Thank you very much

Mar 26 '25 06:03 UndefinedMan

May I ask, have you solved this problem? How did you do it, I had this problem too!

Yes. My team restricted the range of ports available for Docker on our remote Linux server to prevent it from accidentally allocating too many ports. I cannot provide you with more details, but I can return to you next Monday when I get back to office. You can check if there is any similar restriction on Docker network config, or try consult with your team.

Thank you very much！

Hello, can you tell me how fix the problem, I encountered the same problem which is:
dify-weaviate-1  | {"action":"startup","error":"create member list: Failed to get final advertise address: No private IP address found, and explicit IP not provided","level":"error","msg":"could not init cluster state","time":"2025-03-26T05:59:06Z"}
Is the solution to configure certain variables in docker-compose.yml or do we need to modify certain configurations in the Docker software？

Thank you very much

It's probably because of some configs in Docker, not in docker-compose,yaml. Refer to my previous post and try modify the ports allocated for Docker and see if it works.

Mar 26 '25 06:03 Houlson

https://weaviate.io/developers/weaviate/installation/kubernetes

If you see No private IP address found, and explicit IP not provided, set the pod subnet to be in an valid ip address range of the following:

10.0.0.0/8 100.64.0.0/10 172.16.0.0/12 192.168.0.0/16 198.19.0.0/16

check your /etc/docker/daemon.json

Mar 26 '25 06:03 leipeizhao