Is it possible, that a model ist also able to use the online search during a chat?

[hswlab]

I assume it is not possible for a locally running chatbot to perform an online search during a conversation to retrieve current information, e.g. weather or news?

[martindevans]

That's probably something that would be built into an app on top on LLamaSharp, rather than directly into LLamaSharp itself.

[SignalRT]

@hswlab, You could see LLamaSharp as the component that manages the LLM. It's easy to make that kind of things if you use Semantic Kernel in top of LLamaSharp. SK is designed to manage the integration of functions, plugins, etc.

Documentation: https://learn.microsoft.com/en-us/semantic-kernel/overview/

Something similar of what you are looking for: https://www.youtube.com/watch?v=q2c5gaLTFag

[ynanech]

I am also very interested in implementing the plugin function of llama2, similar to the ChatGPT plugin, which is very practical

[martindevans]

Do you think there are any features that LLamaSharp would need to support for you to build plugin/function calling?

I think using a Grammar and appropriate prompting should do the job?

[stephentoub]

Do you think there are any features that LLamaSharp would need to support for you to build plugin/function calling?

SK now passes the Kernel into the connector, giving the connector access to all the plugins. This makes it possible for the connector to support automated function calling. The SK OpenAI-based implementation now does this... it'd be awesome if LlamaSharp could as well, if using a model fine-tuned for function calling. Alternatively, the connector could implement it via a separate planning step, still as part of the chat call from the client.

[SanftMonster]

@stephentoub If I didn't misunderstand, you're saying LLamaSharp should provide a wrapper to make it possible to execute multi-step call depending on a rule? I'm not familiar to SK...I'm wondering whether it's expected to be done in LLamaSharp or the integration with SK.

[stephentoub]

@stephentoub If I didn't misunderstand, you're saying LLamaSharp should provide a wrapper to make it possible to execute multi-step call depending on a rule? I'm not familiar to SK...I'm wondering whether it's expected to be done in LLamaSharp or the integration with SK.

I'm saying some models support function calling, where you can pass to the model a list of functions/tools you'd like it to be able to use optionally, and as part of its response it can ask you to invoke one or more of those functions with a particular set of arguments it provides. You can do so manually, with or without SK, but SK also makes it possible for the IChatCompletion implementation to handle that automatically: the caller passes in the Kernel containing the available plugins, and the IChatCompletion implementation can in turn pass that function information along to the model, and if the model asks for a function to be invoked, the IChatCompletion implementation can do the invocation and give the results back to the model, so that the caller doesn't have to. Here's an example with the OpenAI connector: https://github.com/microsoft/semantic-kernel/blob/ffea465c05d0d9d53a2277d16ee6e969f279f4e6/dotnet/samples/KernelSyntaxExamples/Example59_OpenAIFunctionCalling.cs

[SanftMonster]

I'm saying some models support function calling, where you can pass to the model a list of functions/tools you'd like it to be able to use optionally, and as part of its response it can ask you to invoke one or more of those functions with a particular set of arguments it provides. You can do so manually, with or without SK, but SK also makes it possible for the IChatCompletion implementation to handle that automatically: the caller passes in the Kernel containing the available plugins, and the IChatCompletion implementation can in turn pass that function information along to the model, and if the model asks for a function to be invoked, the IChatCompletion implementation can do the invocation and give the results back to the model, so that the caller doesn't have to. Here's an example with the OpenAI connector: https://github.com/microsoft/semantic-kernel/blob/ffea465c05d0d9d53a2277d16ee6e969f279f4e6/dotnet/samples/KernelSyntaxExamples/Example59_OpenAIFunctionCalling.cs

Thank you! It seems that we should support it in the integration with Sk for convenience. The example above makes sense. FYI @xbotter

[MrCSharp22]

Hi @AsakusaRinne has there been any progress on this? Is there a better place to track the progress of integrating LlamaSharp with SK?

[kosimas]

@MrCSharp22 I'm currently working on a solution in my fork. You can find a working example here https://github.com/kosimas/LLamaSharp/blob/master/LLama.Examples/Examples/SemanticKernelFunctionCalling.cs For the example you have to use this model https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v3

The implementation is a bit dirty and not complete. I will create a pull request a soon as I'm happy with my code and no one else has come with a better solution. I'm also open for ideas and improvements.

[MrCSharp22]

Thanks @kosimas that is working great! I was trying a similar approach but was using v2 of that model. Would love to see this brought into upstream LlamaSharp.

[MithrilMan]

any update on this?

[nitrovent]

I see there's now an Ollama connector for Semantic Kernel available (prerelease). Will it later support function calling? In my first test it didn't.