dify 403 Error with websearch using searxng engine

Self Checks

[X] This is only for bug report, if you would like to ask a question, please head to Discussions.
[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.15

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Followed this tutorial that integrates searxng into dify and ran into problems. https://www.youtube.com/watch?v=yXAJwDtAbLo&ab_channel=MattWilliams Issue looks the same as https://github.com/open-webui/open-webui/issues/2824 I'm not able to connect to the searxng with dify although I have no problems using a browser. I've tried various urls http://192.168.1.15:8080 says I don't have permissions, it is either read protected or or not readable by the server.

ALSO WHY IS THE URL BEING HIDDEN, and WHY DO THE ERROR MESSAGES ONLY LAST A 1/4 of a second. This is very hard to use when things are going wrong.

✔️ Expected Behavior

It to work

❌ Actual Behavior

It didn't work.

Jul 29 '24 20:07 iplayfast

To resolve the 403 error when integrating SearxNG with Dify, follow these steps:

Check Container Status: Ensure all containers are running correctly.
```
docker compose ps
```
Inspect Logs: Check the logs of the docker-api-1 and docker-nginx-1 containers for any errors.
```
docker logs docker-api-1
docker logs docker-nginx-1
```
Resource Allocation: Ensure your Docker VM has sufficient resources allocated (minimum 2 vCPUs and 8 GB of memory).
Environment Variables: Verify that the environment variables in your .env file are correctly configured.
Restart Services: Restart the services to apply any changes.
```
docker compose down
docker compose up -d
```
Network Configuration: Ensure that the SearxNG service is accessible from the Docker container. Replace localhost with host.docker.internal in the service URL.
```
http://host.docker.internal:8080
```
Nginx Configuration: A 403 error can also be caused by Nginx forwarding the service to the wrong location. First, make sure the container is running, then run the following command with root privileges:
```
docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }}: {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'
```
Find these two lines in the output:
```
/docker-web-1: 172.19.0.5
/docker-api-1: 172.19.0.7
```
Open the location where you stored the Dify source code, open dify/docker/nginx/conf.d, replace http://api:5001 with http://172.19.0.7:5001, and replace http://web:3000 with http://172.19.0.5:3000. Afterward, restart the Nginx container or reload the configuration.

These steps should help resolve the 403 error when integrating SearxNG with Dify ^[1]^[2].

To continue talking to Dosu, mention @dosu.

Jul 29 '24 21:07 dosubot[bot]

@iplayfast Hi, this tool need to be set json format as output, do you modify the settings.yml of searxng and restart the container? can you open this url http://127.0.0.1:8081/search?q=apple&format=json&categories=general in the browser to see whether it is worked?

Jul 30 '24 02:07 hjlarry

官网的文档有点问题，应该是

# remove format to deny access, use lower case.
# formats: [html, csv, json, rss]
search:
    formats:
      - html
      - json    # <-- add this line

Aug 01 '24 19:08 jianhuatu

I did set the json in the format. (before opening this issue). @dosu Following your steps this is what I get.

docker compose ps
no configuration file provided: not found

however

docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS                  PORTS                                                                          NAMES
e5c4b965aac3   searxng/searxng                      "/sbin/tini -- /usr/…"   50 minutes ago   Up 50 minutes           0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                      goofy_poincare
643a7e4133a2   nginx:latest                         "sh -c 'cp /docker-e…"   10 days ago      Up 13 hours             0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8088->80/tcp, :::8088->80/tcp   docker-nginx-1
2f03f44c2b34   langgenius/dify-api:0.6.15           "/bin/bash /entrypoi…"   10 days ago      Up 13 hours             5001/tcp                                                                       docker-worker-1
eb017ab09d70   langgenius/dify-api:0.6.15           "/bin/bash /entrypoi…"   10 days ago      Up 13 hours             5001/tcp                                                                       docker-api-1
d2e51d56de93   postgres:15-alpine                   "docker-entrypoint.s…"   10 days ago      Up 13 hours (healthy)   5432/tcp                                                                       docker-db-1
01dd0ec3862c   langgenius/dify-sandbox:0.2.1        "/main"                  10 days ago      Up 13 hours                                                                                            docker-sandbox-1
c7809d79fca6   redis:6-alpine                       "docker-entrypoint.s…"   10 days ago      Up 13 hours (healthy)   6379/tcp                                                                       docker-redis-1
f965152767be   ubuntu/squid:latest                  "sh -c 'cp /docker-e…"   10 days ago      Up 13 hours             3128/tcp                                                                       docker-ssrf_proxy-1
57b017c687c1   semitechnologies/weaviate:1.19.0     "/bin/weaviate --hos…"   10 days ago      Up 13 hours                                                                                            docker-weaviate-1
d56cc8b4cbff   langgenius/dify-web:0.6.15           "/bin/sh ./entrypoin…"   10 days ago      Up 13 hours             3000/tcp                                                                       docker-web-1
eea4e237bc5e   ghcr.io/open-webui/open-webui:main   "bash start.sh"          2 weeks ago      Up 13 hours (healthy)   0.0.0.0:3000->8080/tcp, :::3000->8080/tcp                                      open-webui

so I think that's equilent.

Running migrations
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Preparing database migration...
Database migration skipped
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
[2024-08-09 04:37:52 +0000] [1] [INFO] Starting gunicorn 22.0.0
[2024-08-09 04:37:52 +0000] [1] [INFO] Listening at: http://0.0.0.0:5001 (1)
[2024-08-09 04:37:52 +0000] [1] [INFO] Using worker: gevent
[2024-08-09 04:37:52 +0000] [96] [INFO] Booting worker with pid: 96

(only showing last batch)

docker logs docker-nginx-1
shows a ton of stuff, but no errors that I can see

docker compose down says no configuration file docker compose up says no configuration file

So I did

sudo systemctl restart docker
chris@FORGE:~/ai/websearch/searxng$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED       STATUS                             PORTS                                                                          NAMES
643a7e4133a2   nginx:latest                         "sh -c 'cp /docker-e…"   10 days ago   Up 14 seconds                      0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8088->80/tcp, :::8088->80/tcp   docker-nginx-1
2f03f44c2b34   langgenius/dify-api:0.6.15           "/bin/bash /entrypoi…"   10 days ago   Up 11 seconds                      5001/tcp                                                                       docker-worker-1
eb017ab09d70   langgenius/dify-api:0.6.15           "/bin/bash /entrypoi…"   10 days ago   Up 11 seconds                      5001/tcp                                                                       docker-api-1
d2e51d56de93   postgres:15-alpine                   "docker-entrypoint.s…"   10 days ago   Up 13 seconds (healthy)            5432/tcp                                                                       docker-db-1
01dd0ec3862c   langgenius/dify-sandbox:0.2.1        "/main"                  10 days ago   Up 13 seconds                                                                                                     docker-sandbox-1
c7809d79fca6   redis:6-alpine                       "docker-entrypoint.s…"   10 days ago   Up 14 seconds (health: starting)   6379/tcp                                                                       docker-redis-1
f965152767be   ubuntu/squid:latest                  "sh -c 'cp /docker-e…"   10 days ago   Up 13 seconds                      3128/tcp                                                                       docker-ssrf_proxy-1
57b017c687c1   semitechnologies/weaviate:1.19.0     "/bin/weaviate --hos…"   10 days ago   Up 17 seconds                                                                                                     docker-weaviate-1
d56cc8b4cbff   langgenius/dify-web:0.6.15           "/bin/sh ./entrypoin…"   10 days ago   Up 11 seconds                      3000/tcp                                                                       docker-web-1
eea4e237bc5e   ghcr.io/open-webui/open-webui:main   "bash start.sh"          2 weeks ago   Up 14 seconds (health: starting)   0.0.0.0:3000->8080/tcp, :::3000->8080/tcp                                      open-webui

which should have restarted the works.

I don't understand what you mean in #6

docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }}: {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'
/docker-nginx-1: 172.22.0.5
/docker-worker-1: 172.22.0.8172.21.0.4
/docker-api-1: 172.22.0.9172.21.0.5
/docker-db-1: 172.22.0.7
/docker-sandbox-1: 172.21.0.2172.24.0.2172.23.0.2
/docker-redis-1: 172.22.0.4
/docker-ssrf_proxy-1: 172.22.0.3172.21.0.3
/docker-weaviate-1: 172.22.0.2
/docker-web-1: 172.22.0.6172.24.0.4
/open-webui: 172.17.0.2172.24.0.3

Your last instruction ' Open the location where you stored the Dify source code, open dify/docker/nginx/conf.d, replace http://api:5001 with http://172.19.0.7:5001, and replace http://web:3000 with http://172.19.0.5:3000. Afterward, restart the Nginx container or reload the configuration.

' can't be followed as dify/docker/ngix/conf.d is a directory. Did you mean default.conf? and are you talking about the proxy_pass?

(This is so frustrating!)

Aug 09 '24 17:08 iplayfast

To resolve the 403 error and the issues you're facing with the Docker setup, follow these steps:

Ensure JSON Format is Enabled: Make sure the settings.yml file of SearxNG includes the JSON format:
```
search:
    formats:
      - html
      - json    # <-- add this line 
```
Restart the SearxNG container after making this change.

Correct Docker Compose Command: It seems like the docker-compose.yml file might not be in the current directory. Ensure you are in the correct directory where the docker-compose.yml file is located. If the file is missing, you need to create it. Here is a basic structure for the docker-compose.yml file:

version: '3.8'

services:
  api:
    image: langgenius/dify-api:0.6.15
    ports:
      - "5001:5001"
    depends_on:
      - db
      - redis
      - weaviate

  web:
    image: langgenius/dify-web:0.6.15
    ports:
      - "3000:3000"
    depends_on:
      - api

  worker:
    image: langgenius/dify-api:0.6.15
    ports:
      - "5001:5001"
    depends_on:
      - db
      - redis
      - weaviate

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: your_user
      POSTGRES_PASSWORD: your_password
      POSTGRES_DB: your_db
    ports:
      - "5432:5432"

  redis:
    image: redis:6-alpine
    ports:
      - "6379:6379"

  weaviate:
    image: semitechnologies/weaviate:1.19.0
    ports:
      - "8080:8080"

  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    depends_on:
      - web

Nginx Configuration: Yes, you should edit the default.conf file in the dify/docker/nginx/conf.d directory. Update the proxy_pass directives with the correct IP addresses obtained from the docker inspect command. For example:

server {
    listen 80;
    server_name your_domain_or_ip;

    location /api/ {
        proxy_pass http://172.22.0.9:5001;  # Replace with the correct IP
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass http://172.22.0.6:3000;  # Replace with the correct IP
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Restart Nginx: After updating the default.conf file, restart the Nginx container:
```
docker restart docker-nginx-1
```
Check URL Accessibility: Open the URL in your browser to ensure it works:
```
http://127.0.0.1:8081/search?q=apple&format=json&categories=general
```

These steps should help resolve the 403 error and ensure proper integration between Dify and SearxNG ^[1]^[2].

To continue talking to Dosu, mention @dosu.

Aug 09 '24 17:08 dosubot[bot]