[Feature Request]: Better local LLM support

[yingfeng]

Is there an existing issue for the same feature request?

• [X] I have checked the existing issues.

Is your feature request related to a problem?

Local LLM, especially for LLAMA families should be easily integrated

Describe the feature you'd like

Support ollama

Describe implementation you've considered

No response

Documentation, adoption, use case

No response

Additional information

No response

[MariasStory]

Please implement the llama.cpp server api directly.

[KevinHuSh]

supported with the latest source code and docker images.