sandbox container cannot use custom pip source

[wuhuanyan]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

cd dify-1.0.1/docker/
vi .env
PIP_MIRROR_URL="http://10.125.150.xx:82/ --trusted-host 10.125.150.xx"

cd dify-1.0.1/docker/
vi docker-compose.yaml

# Set networks
networks:
  ssrf_proxy_network:
    driver:bridge
    # internal: true

# I even added the default network to the sandbox configuration or didn't set it at all
sandbox:
#  networks:
#    - default
#    - ssrf_proxy_network

Then restart the services:

docker compose down
docker compose up

However, strangely, even though I can connect to the pip source inside the container, I cannot use pip install.

docker exec -it docker-sandbox-1 bash
curl http://10.125.150.xx:82/  # Can return a response
pip install xxxxx -i http://10.125.150.xx:82/ --trusted-host 10.125.150.xx  # Throws a network connection error

I noticed the sandbox configuration file:

cd dify-1.0.1/docker/
cat volumes/sandbox/conf/config.yaml
enable_network: True

I really don't know where else it is restricted. For this reason, I left the docker directory and reran a sandbox container for testing.

cd ~
docker run --name my_sandbox langgenius/dify=sandbox:0.2.10
docker exec -it my_sandbox bash
pip install xxxxx -i http://10.125.150.xx:82/ --trusted-host 10.125.150.xx  # Installed normally

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[wuhuanyan]

中文翻译

cd dify-1.0.1/docker/
vi .env
PIP_MIRROR_URL="http://10.125.150.xx:82/ --trusted-host 10.125.150.xx"

cd dify-1.0.1/docker/
vi docker-compose.yaml

# 设置networks
networks:
  ssrf_proxy_network:
    driver:bridge
    # internal: true

# 我甚至把sandbox的网络配置增加default，或不设定
sandbox:
#  networks:
#    - default
#    - ssrf_proxy_network

然后重启服务：

docker compose down
docker compose up

但奇怪的是，即使我在容器里面可以连接pip源，但是无法使用pip install

docker exec -it docker-sandbox-1 bash
curl http://10.125.150.xx:82/  # 可以返回响应
pip install xxxxx -i http://10.125.150.xx:82/ --trusted-host 10.125.150.xx  # 抛出网络连接错误

我留意到sandbox的配置文件：

cd dify-1.0.1/docker/
cat volumes/sandbox/conf/config.yaml
enable_network: True

我实在不在到还有哪里限制了。为此我脱离了docker目录，重新run一个sandbox容器进行测试。

cd ~
docker run --name my_sandbox langgenius/dify=sandbox:0.2.10
docker exec -it my_sandbox bash
pip install xxxxx -i http://10.125.150.xx:82/ --trusted-host 10.125.150.xx  # 正常安装

[dosubot[bot]]

I found a similar unsolved discussion titled "Unable to use external HTTP requests after installing Dify with docker-compose on Ubuntu" ^[1]. This might be related to your issue with network connectivity for pip installations in the sandbox container.

In your case, it seems like the network configuration might be affecting the pip installation. Here are a few things to consider:

1. Network Configuration: Ensure that the ssrf_proxy_network is not set as internal, as this restricts access to external networks. You've already commented out the internal: true line, which is correct.

2. Proxy Settings: Check if the HTTP_PROXY and HTTPS_PROXY environment variables are set correctly in your Docker environment. These should point to a valid proxy if you're using one.

3. PIP_MIRROR_URL: Make sure that the PIP_MIRROR_URL is correctly set in the .env file and that the URL is accessible from within the container.

4. Sandbox Configuration: The enable_network setting in your config.yaml is set to True, which should allow network access. Double-check that this setting is being applied correctly.

5. Testing Outside Docker Directory: You've mentioned that running a sandbox container outside the Docker directory works fine. This suggests that the issue might be related to the specific Docker network or environment settings in your Dify setup.

Review these configurations to ensure they are set up correctly. If the issue persists, consider checking the proxy server's configuration or any firewall settings that might be affecting network access.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[crazywoola]

This PIP_MIRROR_URL only affects the plugin_daemon container. https://github.com/langgenius/dify/blob/85160b0487c269b08650d8d5f18d3226eccdf4ac/docker/docker-compose.yaml#L584

[wuhuanyan]

This PIP_MIRROR_URL only affects the plugin_daemon container.此PIP_MIRROR_URL仅影响plugin_daemon容器。

dify/docker/docker-compose.yaml

Line 584 in 85160b0

PIP_MIRROR_URL: ${PIP_MIRROR_URL:-}

This is no longer relevant because I always manually execute commands in the container's bash. 这个已经没有关系，因为我都是到容器的bash中，去手动执行

pip install xxxxx -i http://10.125.150.xx:82/ --trusted-host 10.125.150.xx

[wuhuanyan]

@dosu I can manually execute commands in the plugin container.

pip install xxxxx -i http://10.125.150.xx:82/ --trusted-host 10.125.150.xx

[dosubot[bot]]

If you can manually execute the pip install command in the plugin container, it suggests that the network settings for that container are correct. However, since the issue persists in the sandbox container, consider the following:

1. Network Isolation: Ensure that the sandbox container is not isolated from external networks. Check the network settings in your docker-compose.yaml to confirm that the sandbox container is connected to the appropriate network.

2. Firewall and Security Groups: Verify that there are no firewall rules or security group settings blocking outbound connections from the sandbox container to your custom pip source.

3. Docker Network Configuration: Try explicitly setting the network for the sandbox container in your docker-compose.yaml to ensure it has the same network access as the plugin container.

4. Environment Variables: Double-check that any necessary environment variables for network access are correctly set in the sandbox container.

If these steps don't resolve the issue, consider running a network diagnostic tool within the sandbox container to further investigate connectivity issues.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[crazywoola]

https://github.com/langgenius/dify/blob/85160b0487c269b08650d8d5f18d3226eccdf4ac/docker/docker-compose.yaml#L556-L558

If the configuration files are proper set. Then it should be a problem of ssrf_proxy

https://github.com/langgenius/dify/blob/85160b0487c269b08650d8d5f18d3226eccdf4ac/docker/docker-compose.yaml#L553-L554

You should allow the 10.125.150.xx in this file https://github.com/langgenius/dify/blob/85160b0487c269b08650d8d5f18d3226eccdf4ac/docker/docker-compose.yaml#L599

[wuhuanyan]

Okay, I'm not very familiar with this configuration. I'll study it further. However, I have already removed the network from the sandbox container. It should no longer be related to the network. This is worth thinking about. 好的，这个配置我不太熟悉，我再研究一下。 不过我已经移除了sandbox容器中的network。理应不再和网络有关联。值得思考。

[ycnjckl]

服务器在加硬盘，我凭印象说一下吧。 一个是sandbox的yaml里面你要加上pip的镜像源。 一个是sandbox的conf文件里面，你要把允许的数组队列加上，要是不在外网的话，就生成一个[0,1,2.......499]，这是允许所有权限的，如果你不放心的话，就自己去找一个允许权限的脚本检测工具。 另外一个是sandbox的requirements.txt里面添加正确的库名，大小写有区分的。 镜像源一定要很快的那种，不然每次重启都要安装一遍，速度慢的话，你可能以为出错了，其实单纯的就是pip没有下载完。

[jhscut]

服务器在加硬盘，我凭印象说一下吧。 一个是sandbox的yaml里面你要加上pip的镜像源。 一个是sandbox的conf文件里面，你要把允许的数组队列加上，要是不在外网的话，就生成一个[0,1,2.......499]，这是允许所有权限的，如果你不放心的话，就自己去找一个允许权限的脚本检测工具。 另外一个是sandbox的requirements.txt里面添加正确的库名，大小写有区分的。 镜像源一定要很快的那种，不然每次重启都要安装一遍，速度慢的话，你可能以为出错了，其实单纯的就是pip没有下载完。

请问你知道requirements.txt安装的目录在哪里吗？我是内网，需要在外网下依赖