Mintplex-Labs
[BUG]: An error occurred while streaming response. NetworkError when attempting to fetch resource.
How are you running AnythingLLM?
Docker (local)
What happened?
I have Anything LLM running pointed at my local Ollama. My local Ollama is working fine in Open WebUI and when I curl it:
$ curl https://ollama
Ollama is running%
Also, it would appear Anything LLM is communicating in some form with Ollama as it shows my models:
There are no error in the logs either:
No pending migrations to apply.
[backend] info: [EncryptionManager] Loaded existing key & salt for encrypting arbitrary data.
[backend] info: [TELEMETRY ENABLED] Anonymous Telemetry enabled. Telemetry helps Mintplex Labs Inc improve AnythingLLM.
[backend] info: prisma:info Starting a sqlite pool with 17 connections.
[backend] info: [TELEMETRY SENT] {"event":"server_boot","distinctId":"50247cfa-0b04-4800-89b5-af27967efe6e","properties":{"commit":"--","runtime":"docker"}}
[backend] info: [CommunicationKey] RSA key pair generated for signed payloads within AnythingLLM services.
[backend] info: [EncryptionManager] Loaded existing key & salt for encrypting arbitrary data.
[backend] info: Primary server in HTTP mode listening on port 3001
[backend] info: [BackgroundWorkerService] Feature is not enabled and will not be started.
But when I go to use the frontend I get the following error:
When I look in the console I see the following error:
ns_error_net_interrupt
I'm wondering if this is because I'm serving AnythingLLM behind a Caddy proxy? I looked in the example env file and didn't see anything related to proxy settings.
I can curl Ollama from the AnythingLLM container as well.
Are there known steps to reproduce?
No response
Pretty sure this has to do with Caddy as the proxy. The desktop app works fine with my Ollama.
I would agree, if you are in docker you must use the host gateway binding:
https://host.docker.internal:PORT on Mac/Windows or http://172.17.0.1:11434
What is http://ollama:11434 mapping to? interesting that the /model seems to work but chats are not?
Also, I realize now that this is about streaming. Does your proxy support web sockets? That is how chat works and 99% of the time the proxy is blocking this and drops connection instantly.
In nginx:
# In server block
.....
# Enable websocket connections for agent protocol.
location ~* ^/api/agent-invocation/(.*) {
proxy_pass http://localhost:3001;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
# Regular streaming/HTTP support
location / {
proxy_connect_timeout 605;
proxy_send_timeout 605;
proxy_read_timeout 605;
send_timeout 605;
keepalive_timeout 605;
proxy_buffering off;
proxy_cache off;
proxy_pass http://ip.address.goes.here:3001$request_uri;
}
....
@timothycarambat I'm using Caddy. Specifically, caddy-docker-proxy. I've had no issues using Ollama behind Caddy with other tooling: Open WebUI, Enchanted on iOS, various Paperless-ngx POCs, etc. all work fine. The Anything LLM Mac client works fine with my Ollama behind Caddy as well, so this feels isolated to the Docker version.
http://ollama is just hitting the internal Docker container for Ollama over the Caddy Docker network. I also tried https://ollama.mydomain.com which works with the other Ollama clients I mentioned above, but it also had the same error.
Would there be any issues with AnythingLLM behind Caddy? Wondering if you meant Ollama or AnythingLLM must use the host port bindings.
I should mention my working Open WebUI is also behind Caddy using the same setup.
In general if you have a setup of
- Docker container on host running on port 11434
- AnythingLLM in another container on host 3001
To connect AnythingLLM to the service on 11434 you would need to use this URL in AnythingLLM
http://host.docker.internal:11434 this would use the network bridge to then go from
AnythingLLM<->host network<->container2
I have never used caddy and do not exactly know or understand what that entails. However, the /models endpoint works and the only distinctive difference between calling /models and /stream-chat is that stream chat uses SSR. Which might explain the NS_ERROR_NET_INTERRUPT.
So clearly the https://ollama endpoint is reachable over traditional REST/HTTP methods, but the client disconnects when the request is SSR
https://github.com/Mintplex-Labs/anything-llm/blob/b658f5012de0658272fb0853a1c00a1fda2ccf1f/server/endpoints/chat.js#L45-L49
Does caddy have issues with text/event-stream header flushing instantly? Might be just the right combo of the docker +SRR+caddy all together
https://github.com/caddyserver/caddy/issues/3765
@timothycarambat I tried flush_interval -1 without any success. I don't do host ports like you describe for any of my services. Have you gotten this to work behind any reverse proxy? If so, mind sharing the config? Worth noting the Caddy documentation says the flush_interval option is ignored and headers are flushed immediately when Content-Type: text/event-stream.
I have found another lead though: https://github.com/leafac/caddy-express-sse which suggests this as a workaround:
res.type("text/event-stream").write(":\n\n");
See the Caddy bug filed here: https://github.com/leafac/caddy-express-sse?tab=readme-ov-file
There's also this open issue here about compression: https://github.com/caddyserver/caddy/issues/6293
I also found this issue that looks similarly related: https://github.com/Giveth/giveth-dapps-v2/issues/4620
Here's where Caddy automatically sets flush_interval to -1 for text/event-stream: https://github.com/cmfcmf/caddy/blob/master/modules/caddyhttp/reverseproxy/streaming.go#L88-L107
I have gotten AnythingLLM to run fine behind a reverse proxy, but for Nginx
# Default server configuration
# Example config for regular setup + SSL + Websockets.
server {
listen 80;
server_name domain-here.com;
return 301 https://domain-here.com$request_uri;
}
server {
listen 443 ssl;
ssl on;
server_name domain-here.com;
# Enable websocket connections for agent protocol.
location ~* ^/api/agent-invocation/(.*) {
proxy_pass http://localhost:3001;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
location / {
proxy_connect_timeout 605;
proxy_send_timeout 605;
proxy_read_timeout 605;
send_timeout 605;
keepalive_timeout 605;
proxy_buffering off;
proxy_cache off;
proxy_pass http://public-ip-address-here:3001$request_uri;
}
}
This has NGINX in front of the service running on 3001 on an Ubuntu machine. Then has http://domain-here.com on 80 proxy to 3001 (can enable SSL if you need to also with the ssl_cert directives