Adding a version of Ollama to dynamically lookup Server, Model, RequestHeaders per request

[AshleyTuring]

Published to NPM here: https://www.npmjs.com/package/genkitx-ollamadynamic

The plugin is called ollamadynamic. This plugin is a dynamic configuration plugin for the Genkit framework, designed to work with the Ollama API.

What the Plugin Does:

1. Dynamic Configuration:

   • The core purpose of the plugin over the existing Ollama plugin is to dynamically configure the model, server address, and request headers based on the incoming request. This allows for greater flexibility, enabling the use of different models, API servers, and authentication tokens depending on the context of the request.

2. Dynamic Model Definition:

   • The ollamadynamicModel function defines a model that can be configured dynamically. The model's configuration, such as server address, model name, and type, is determined by the dynamicConfig function, which is provided as part of the OllamaDynamicPluginParams.

3. Handling API Requests:

   • The plugin constructs an API request to the specified server address using the provided model name and other parameters (like temperature, top-p, etc.). It supports both chat and generate types of API calls.

4. Streaming and Non-Streaming Responses:

   • The plugin can handle both streaming and non-streaming API responses. For streaming responses, it reads the response chunks and processes them as they arrive, while for non-streaming responses, it processes the entire response at once.

5. Tool Support:

   • The plugin supports multiturn conversations, media, tools, and system roles. This means it can be used in complex scenarios where these features are needed.

6. Metadata Usage:

   • The plugin uses metadata from the input's context to dynamically set parameters like requestId and modelName. These are used to retrieve the appropriate configuration for the model execution.

How It Works:

• Dynamic Configuration:

  • When a request is made to this model, the dynamicConfig function is called with the modelName and requestId extracted from the input's context. This function returns the server address, model name, type, and authentication tokens required to make the API call.

• Request Handling:

  • The plugin constructs the API request with the necessary options and headers, then sends this request to the dynamically determined server address.

• Response Handling:

  • The response from the API is processed, whether it's a streaming response (handled in chunks) or a non-streaming response (handled as a whole).

Usage Scenario:

This plugin is ideal for situations where you need to interact with different instances of an API (like Ollama) or use different models depending on the context of the request. It offers the flexibility to change server addresses, models, and headers dynamically, making it suitable for advanced use cases like multi-tenant applications, user-specific configurations, or adaptive AI systems where the underlying model may need to change based on specific criteria.

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[MichaelDoyle]

Hey Ashley! Thanks for all your efforts here. We are not accepting 3P plugins into the main repository at this time. We encourage you to publish to genkitx- on NPM, just as you've already done, so that folks can easily find your plugin. If you don't want to be in the business of maintaining the repository itself, you could consider collaborating with other members of the community to share the load. As of today, the biggest collection of plugins is currently maintained by TheFireCo: https://github.com/TheFireCo/genkit-plugins.

Alternatively, if you feel any of these features should be part of the official ollama plugin supported by Genkit, we're happy to discuss those points (either on GH or Discord) and accept any PRs that come out of it.