Skyvern-AI
Integration with Ollama
Great work on this! For testing purposes and local development, potentially can we integrate this with Ollama Chat Completions?
https://github.com/ollama/ollama/blob/main/docs/openai.md
When I set mine up and changed the OpenAPIWrapper to a localhost base_url
class OpenAIKeyClientWrapper:
client: AsyncOpenAI
key: str
remaining_requests: int | None
def __init__(self, key: str, remaining_requests: int | None) -> None:
self.key = key
self.remaining_requests = remaining_requests
self.updated_at = datetime.utcnow()
self.client = AsyncOpenAI(api_key=self.key, base_url = 'http://localhost:11434/v1',)
and changing the its model to point towards llama2
json_response = await app.OPENAI_CLIENT.chat_completion(
model="llama2",
step=step,
prompt=extract_information_prompt,
screenshots=scraped_page.screenshots,
)
It doesn't seem to like the request from the Geico.com boilerplate Task
Error message:
Error code: 400 - {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'http://localhost:11434/v1/chat/completions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
During handling of the above exception, another exception occurred:
File "/Users/bryankho/Code/skyvern/skyvern/forge/sdk/api/open_ai.py", line 154, in chat_completion
response = await available_client.client.chat.completions.with_raw_response.create(**chat_completion_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bryankho/Code/skyvern/skyvern/forge/agent.py", line 335, in agent_step
json_response = await app.OPENAI_CLIENT.chat_completion(
File "/Users/bryankho/Code/skyvern/skyvern/forge/agent.py", line 211, in execute_step
step, detailed_output = await self.agent_step(task, step, browser_state, organization=organization)
File "/Users/bryankho/Code/skyvern/skyvern/forge/sdk/agent.py", line 97, in __call__
await self.app(scope, receive, send)
File "<string>", line 1, in <module>
openai.BadRequestError: Error code: 400 - {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Not sure is this more of a setup issue or Ollama unable to support all use cases of OpenAI Chat Completion API
We're going to release some LLM router changes really soon (testing's almost complete!!!)
We used LiteLLM to implement a configurable, easily extensible LLM manager. I just checked it out and it has support for Ollama (https://litellm.vercel.app/docs/providers/ollama).
We'd appreciate it if you could open a PR to add support for Ollama after we merge the LLM router changes. I'll let you know when that PR is merged.
Our first release with the brand new LLM module is out. Feel free to check out the release notes for more details: https://github.com/Skyvern-AI/skyvern/releases/tag/0.1.1
Closing this as litellm supports ollama https://docs.litellm.ai/docs/providers/ollama
@suchintan apologies for commenting on a closed PR, but wondering how do we use this to use local (Ollama) models? Can you provide a simple example...thanks!
@arsaboo https://github.com/Skyvern-AI/skyvern/issues/242
I closed this a little bit hastily -- we would love a contribution in #242 -- just need to add a new model provider type in the code and LiteLLM would take care of the rest
@arsaboo If you're open to writing a little bit of code -- we'd love some help here! :)
@suchintan I will take a look at this, but I think it is above my pay grade 😜
I would love to help you implement this (testing or docs). Given how expensive GPT4 is, local models are a must. I have a very capable device that can run large models and would love to test this with local models. Litellm devs are very active on GH, and you may have some luck with them.