Agent not working

[Nunsferatus]

How are you running AnythingLLM?

Docker (local)

What happened?

This is the response that I receive no matter what.

Am I doing something wrong or is this a bug?!

Are there known steps to reproduce?

No response

[SamuelEnzi]

As far as i know agents only work with OpenAI as an LLM provider for now. What are your provider settings?

[SamuelEnzi]

@Nunsferatus here is a screenshot

[Nunsferatus]

Thank you for your quick response.

Here is the agent's configuration that I'm using.

[SamuelEnzi]

@Nunsferatus Yes it seems like you use Ollama which does currently not support agent as far as i know. You should switch to OpenAI Provider or wait until this feature is implemented by the team.

[Nunsferatus]

I don't think that you are correct. As you can see in the following video at minute 9:53, Tim Carambat uses, as far as I understand, the same configuration that I am trying to use.

https://www.youtube.com/watch?v=4UFrVvy7VlA

[SamuelEnzi]

You are right it is shown in the video. Either there is conflicting Information or Tim is using a different version. For me it does not work either when i try other providers.

[Nunsferatus]

Thank you Samuel for your help.

Hopefully, someone who is more knowledgeable than we can figure out what is happening...

[timothycarambat]

@SamuelEnzi Yeah, the latest docker and desktop versions now support OSS model providers like Ollama.

@Nunsferatus, this is worth me writing an article on for our docs page because we get this question at least twice a day and I touched on it in the video at bit.

TLDR: LLMs are not deterministic and its ability to know and call a tool is dependent on many factors, but the main on is the model size + quantization. In that video i am using LLama3-8B Q8, which is okay, but not perfect!

We can tell and LLM to do x,y,z but that does not mean it will listen and its steerability and coherence of a question is always a little random. We currently even set the temperature of every tool call to zero, but that still does not mean it will call the tools.

This is also me assuming that the PDF you want to summarize is in the workspace you are asking. Sometimes the LLM is so hard-headed that you even need to tell it the exact tool. Use doc-summarizer to summarize mypdf.pdf and it still may not work.

This is just a reality of LLMs, they certainly are not coherent and using bigger and better models produces better results but that is still not the golden ticket. The reason some LLMs like OpenAI do tool calls separately is to "force" them to use a tool, whereas with our wrapper we cannot exert that level on control on the LLM. Which also could be a bad idea to "force" this as it may just constantly loop, or never call anything, and a whole bunch of other issues.

Basically, it is not magic and its really model dependent, but neither of those guarantee any OSS LLM will always 100% understand and comprehend what your question was and leverage tools to answer that.

What model and quantization are you running? Closing for now, but we can continue disucssion

[tiangao88]

Sorry to reopen this discussion. I Just installed the latest docker image and have the same issue while using gpt-4o

How can I see which version of AnythingLLM I am actually running? Maybe I did something wrong in the configuration.

[Nunsferatus]

Hi Timothy, Thank you for your answer and I think that either an article or a new video on it would be just fantastic.

As you can see from my previous post I am using llama3:8b-instruct-q8_0 Which if I'm not mistaken is exactly the one that you use in the video.

I have also tried mistral:7b-instruct-q8_0 with the same result.

I have tried many ways of making the question to see if that would solve the problem none of my tries worked.

I am running it in Docker do you think that's the issue? I have it that way because I would prefer to have it in a way that I can use on whatever platform/Device I want as long as I'm in my network without having to install it on every device.

Thanks again

[Anto79-ops]

@tiangao88 im experieing the same as you, it just closes the agent immediately wthiout any actions. I did get a log

Failed to run validation function on agent_search_provider Invalid SERP provider.

Which is strange because I had web-search disabled. Then I got proper Google search crendentaisl, but still exiting almost immeadiately from the agent chat.

I'm not sure what is happened, but what makes this harder is that there are no logs being produced. You may have to restart your instance (if in docker) and try again, and look into your logs. It logs the error above (if that is the error) only once.

Sorry to reopen this discussion. I Just installed the latest docker image and have the same issue while using gpt-4o

How can I see which version of AnythingLLM I am actually running? Maybe I did something wrong in the configuration.

[SamuelEnzi]

@Anto79-ops @tiangao88 It seems like the issue has something todo with the docker host. I host my instance on a server (docker) within my local network and i have the same issue as @tiangao88 where it instantly exits from the agent.

To confirm my theory i installed the windows desktop application and used the OLLAMA instance hosted on the same server than before. Locally the agent works without any issue.

Here are two screenshots. First is the Desktop application and second the docker instance

[max-zirkonzahn]

If you are running Apache as a reverse proxy, the problem seems to be Apache. The frontend tries to establish a WSS connection when calling the agent and fails to do so, as can be seen in the webbrowser console. When you try to access the frontend directly without the Apache proxy, the agent chat works.

I tried to add

        RewriteEngine on
        RewriteCond ${HTTP:Upgrade} websocket [NC]
        RewriteCond ${HTTP:Connection} upgrade [NC]
        RewriteRule .* "ws:/<ip of AnythingLLMserver>/$1" [P,L]

to my VirtualHost, but that didn't fix the problem. (If I made some errors here, please correct me)

I switched from Apache to HAproxy, based on my experience with WSS and Apache. Agent chat is working now. I hope this helps

[tiangao88]

If you are running Apache as a reverse proxy, the problem seems to be Apache. The frontend tries to establish a WSS connection when calling the agent and fails to do so, as can be seen in the webbrowser console. When you try to access the frontend directly without the Apache proxy, the agent chat works.

I tried to add

        RewriteEngine on
        RewriteCond ${HTTP:Upgrade} websocket [NC]
        RewriteCond ${HTTP:Connection} upgrade [NC]
        RewriteRule .* "ws:/<ip of AnythingLLMserver>/$1" [P,L]

to my VirtualHost, but that didn't fix the problem. (If I made some errors here, please correct me)

I switched from Apache to HAproxy, based on my experience with WSS and Apache. Agent chat is working now. I hope this helps

You have indeed found the root cause, thanks! When accessing directly the docker container IP, bypassing all proxy, the Agent functionalities are working. Now I have to find how to customize Nginx configuration to let websocket connections through.

[Anto79-ops]

can I ask, is Apache something that is already running and installed? I don't recall installing/using this. thanks

[max-zirkonzahn]

can I ask, is Apache something that is already running and installed? I don't recall installing/using this. thanks

Sure, Apache is not installed by default, but some people, like myself, add it to add SSL/TLS to the connection to protect the data in transit.

[Anto79-ops]

thank you, I can confirm I'm getting the same problem as shown here with my Chrome logs

I will see if switching to HAproxy fixes it. thank you!

[man2004]

@tiangao88 , please see if https://github.com/Mintplex-Labs/anything-llm/issues/1257#issuecomment-2089399353 helps.

[greigmarshall]

I had the same error shown in the screenshots from @max-zirkonzahn and @Anto79-ops. I'm using nginx for reverse proxy and was able to resolve the issue by adding the following to the appropriate server block:

    location /api {
        proxy_pass http://<server IP:port>/api/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_connect_timeout 7d;
        proxy_send_timeout 7d;
        proxy_read_timeout 7d;
    }

This configuration can be applied to anything that uses websocket.

[tnypxl]

@greigmarshall Was running into this issue and your suggestion fix it! You rock!

[timothycarambat]

FWIW; this is buried in the docs, we should duplicate it to the BARE_METAL docs at least as an addendum for proxies since this issue often gets overlooked and makes agents look broken

https://github.com/Mintplex-Labs/anything-llm/blob/master/cloud-deployments/aws/cloudformation/aws_https_instructions.md#step-7-create-simple-http-proxy-configuration-for-anythingllm