dify icon indicating copy to clipboard operation
dify copied to clipboard
Published 1 week ago • verified publisher icon langgenius

Opik API check failed: timed out

Open filwu8 opened this issue 9 months ago • 5 comments

Self Checks

  • [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
  • [x] I have searched for existing issues search for existing issues, including closed ones.
  • [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

v1.0.1 version worked well, but after upgrading to v1.1.0 version, an error was reported, and the log is as follows:

Image

api-1 | 2025-03-19 14:18:05.128 INFO [Thread-448 (process_request_thread)] [_internal.py:97] - 172.23.0.11 - - [19/Mar/2025 06:18:05] "GET /console/api/workspaces/current/plugin/tasks?page=1&page_size=100 HTTP/1.1" 200 - api-1 | 2025-03-19 14:18:05.586 DEBUG [Thread-449 (process_request_thread)] [connectionpool.py:241] - Starting new HTTP connection (1): plugin_daemon:5002 api-1 | 2025-03-19 14:18:05.588 DEBUG [Thread-449 (process_request_thread)] [connectionpool.py:544] - http://plugin_daemon:5002 "GET /plugin/4ae5752c-eada-494f-b83d-b7348d75ae0a/management/install/tasks?page=1&page_size=100 HTTP/1.1" 200 None api-1 | 2025-03-19 14:18:05.590 INFO [Thread-449 (process_request_thread)] [_internal.py:97] - 172.23.0.11 - - [19/Mar/2025 06:18:05] "GET /console/api/workspaces/current/plugin/tasks?page=1&page_size=100 HTTP/1.1" 200 - api-1 | 2025-03-19 14:18:05.698 DEBUG [Thread-446 (process_request_thread)] [_trace.py:47] - connect_tcp.failed exception=ConnectTimeout(TimeoutError('timed out')) api-1 | 2025-03-19 14:18:05.698 INFO [Thread-446 (process_request_thread)] [opik_trace.py:461] - Opik API check failed: timed out api-1 | Traceback (most recent call last): api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions api-1 | yield api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 236, in handle_request api-1 | resp = self._pool.handle_request(req) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request api-1 | raise exc from None api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request api-1 | response = connection.handle_request( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 101, in handle_request api-1 | raise exc api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 78, in handle_request api-1 | stream = self._connect(request) api-1 | ^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 124, in _connect api-1 | stream = self._network_backend.connect_tcp(**kwargs) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp api-1 | with map_exceptions(exc_map): api-1 | ^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/usr/local/lib/python3.12/contextlib.py", line 158, in exit api-1 | self.gen.throw(value) api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions api-1 | raise to_exc(exc) from exc api-1 | httpcore.ConnectTimeout: timed out api-1 | api-1 | The above exception was the direct cause of the following exception: api-1 | api-1 | Traceback (most recent call last): api-1 | File "/app/api/core/ops/opik_trace/opik_trace.py", line 458, in api_check api-1 | self.opik_client.auth_check() api-1 | File "/app/api/.venv/lib/python3.12/site-packages/opik/api_objects/opik_client.py", line 134, in auth_check api-1 | self._rest_client.check.access( api-1 | File "/app/api/.venv/lib/python3.12/site-packages/opik/rest_api/check/client.py", line 55, in access api-1 | _response = self._client_wrapper.httpx_client.request( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/opik/rest_api/core/http_client.py", line 213, in request api-1 | response = self.httpx_client.request( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_client.py", line 837, in request api-1 | return self.send(request, auth=auth, follow_redirects=follow_redirects) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_client.py", line 926, in send api-1 | response = self._send_handling_auth( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_client.py", line 954, in _send_handling_auth api-1 | response = self._send_handling_redirects( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_client.py", line 991, in _send_handling_redirects api-1 | response = self._send_single_request(request) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1027, in _send_single_request api-1 | response = transport.handle_request(request) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 235, in handle_request api-1 | with map_httpcore_exceptions(): api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/usr/local/lib/python3.12/contextlib.py", line 158, in exit api-1 | self.gen.throw(value) api-1 | File "/app/api/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions api-1 | raise mapped_exc(message) from exc api-1 | httpx.ConnectTimeout: timed out api-1 | 2025-03-19 14:18:05.701 INFO [Thread-446 (process_request_thread)] [_internal.py:97] - 172.23.0.11 - - [19/Mar/2025 06:18:05] "PATCH /console/api/apps/d7a8b0d5-5b07-45c2-abf6-5ecbbf678a8a/trace-config HTTP/1.1" 400 -

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

filwu8 avatar Mar 19 '25 06:03 filwu8

The "Opik API check failed: timed out" error you're encountering after upgrading to Dify version 1.1.0 suggests a connection timeout issue with the Opik API. Here are a few steps you can take to troubleshoot and potentially resolve this issue:

  1. Check Network Connectivity: Ensure that your server hosting the Dify application has a stable network connection and can reach the Opik API endpoint. You might want to test connectivity using tools like ping or curl.

  2. Review Timeout Settings: The default timeout settings for HTTP connections in the Dify codebase are a 10-second connect timeout and a 60-second read timeout [1]. You can adjust these settings by setting the environment variables API_TOOL_DEFAULT_CONNECT_TIMEOUT and API_TOOL_DEFAULT_READ_TIMEOUT to higher values if needed.

  3. Verify Opik API Configuration: Double-check the configuration settings for the Opik API in your Dify setup. Ensure that the API keys and endpoints are correctly configured and have not changed during the upgrade.

  4. Check for Dependency Changes: The Dify 1.1.0 release uses version 1.3.4 of the Opik library [2]. Ensure that this version is compatible with your setup and that there are no breaking changes affecting the API integration.

  5. Inspect Logs for Additional Clues: Review the logs for any additional error messages or warnings that might provide more context about the timeout issue.

By following these steps, you should be able to identify and address the root cause of the timeout error. If the issue persists, further investigation into network configurations or potential changes in the Opik API might be necessary.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

dosubot[bot] avatar Mar 19 '25 06:03 dosubot[bot]

Does Dify not support self-hosted Docker deployment of opik?？

filwu8 avatar Mar 19 '25 07:03 filwu8

I couldn't find specific information in the Dify documentation or codebase about support for self-hosted Docker deployment of Opik. However, there are configuration settings for integrating Opik with Dify, which involve obtaining an Opik API Key and configuring it within the Dify application for monitoring purposes [1][2]. If you're experiencing issues with self-hosting, you might want to check the Dify troubleshooting guides and FAQs for any related information [3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

dosubot[bot] avatar Mar 19 '25 07:03 dosubot[bot]

Hi @filwu8, I wrote the Opik integration. Dify should support the self-hosted docker deployment of opik. The issue likely lies on the configured Opik URL. The communication will happens between the worker container of Dify (https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml#L461) and the Opik container.

The Dify worker container is running with two specific networks: https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml#L480-L482. You will need to setup an Opik URL that is accessible from the worker container and that might needs to change the network configuration on either Dify or Opik docker-compose, is that something that you comfortable doing?

If you want to test that Opik URL, you can connect to the running worker container and uses curl to test it like this:

docker exec -ti dify-worker-1 bash
curl http://opik/api/v1/private/auth // Where http://opik/api is your configured Opik URL

docker inspect can also helps you check if the two containers can communicate with each other

Lothiraldan avatar Mar 20 '25 13:03 Lothiraldan

Hi @filwu8, I wrote the Opik integration. Dify should support the self-hosted docker deployment of opik. The issue likely lies on the configured Opik URL. The communication will happens between the worker container of Dify (https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml#L461) and the Opik container.

The Dify worker container is running with two specific networks: https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml#L480-L482. You will need to setup an Opik URL that is accessible from the worker container and that might needs to change the network configuration on either Dify or Opik docker-compose, is that something that you comfortable doing?

If you want to test that Opik URL, you can connect to the running worker container and uses curl to test it like this:

docker exec -ti dify-worker-1 bash curl http://opik/api/v1/private/auth // Where http://opik/api is your configured Opik URL docker inspect can also helps you check if the two containers can communicate with each other

thank you！It‘s work！

but error url ：

Image

i think you can fix url local URL or API URL

filwu8 avatar Mar 20 '25 14:03 filwu8

Hi, @filwu8. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • You reported a timeout error after upgrading Dify from version 1.0.1 to 1.1.0 with the Opik API integration in a self-hosted Docker setup.
  • Troubleshooting steps were provided, including checking network connectivity and timeout settings.
  • Lothiraldan suggested checking the Opik URL configuration and network settings, which resolved the issue.
  • You confirmed the solution worked but noted an error with the URL, suggesting a fix for local or API URLs.

Next Steps

  • Please let me know if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If there are no further updates, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

dosubot[bot] avatar Apr 20 '25 16:04 dosubot[bot]