Bitmonkey

@TechScribe-Deaf "there should be a full disablement of ANY unloading of the model as an option." well just to prove @pdevine correct (a fellow Devine here), I came here to...

try pop this at the top of your example for LM Studio... os.environ["OPENAI_API_BASE"]="http://localhost:1234/v1"

I did a workaround on my end which seems to work, although I just copied your code into my codebase, Nnt a great solution but it helps me learn whats...

From digging around in the code my feelings were that it was very OpenAI centric and their models can handle the context much better that local models that certainly need...

If I was you I would probably separate the local llm people using crewai and the Openai peps. I feel we have an opportunity right now with the poor llms...

Your script is not working for me, but did do "better" with the system template. After a bit of digging around I believe the problem may come from Langchain's inability...

Can confirm its working a bit with Hermes2 Pro.... seems llama3 just doesn't get us man.

Ollama needs to be sent the raw flag set to true.... but here is the catch, Langchain will not send it so this is what I do which seems to...

Use at your own risk though. Let me know how it works out.