[Bug]: Can't run Ollama with stream=True

[FlafyDev]

What happened?

When trying to use Ollama with LiteLLM and with Stream=True, an exception is thrown.

litellm version: 1.15.0

1. Serve ollama locally on port 11434 (or replace the port in the python file)
2. pip install litellm
3. Run this:

from litellm import completion
import litellm

litellm.set_verbose=True

response = completion(
    model="ollama/llama2", 
    messages=[{ "content": "respond in 20 words. who are you?","role": "user"}], 
    api_base="http://localhost:11434",
    stream=True,
)
for part in response:
    print(part.choices[0].delta.content or "")

Relevant log output

❯ python3 main.py
kwargs[caching]: False; litellm.cache: None

LiteLLM completion() model= llama2; provider = ollama

LiteLLM: Params passed to completion() {'functions': [], 'function_call': '', 'temperature': None, 'top_p': None, 'n': None, 'stream': True, 'stop': None, 'max_tokens': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'model': 'llama2', 'custom_llm_provider': 'ollama', 'response_format': None, 'seed': None, 'tools': None, 'tool_choice': None, 'max_retries': None}

LiteLLM: Non-Default params passed to completion() {'stream': True}
self.optional_params: {'stream': True}
PRE-API-CALL ADDITIONAL ARGS: {'api_base': 'http://localhost:11434/api/generate', 'complete_input_dict': {'model': 'llama2', 'prompt': 'respond in 20 words. who are you?', 'stream': True}, 'headers': {}, 'acompletion': False}


POST Request Sent from LiteLLM:
curl -X POST \
http://localhost:11434/api/generate \
-d '{'model': 'llama2', 'prompt': 'respond in 20 words. who are you?', 'stream': True}'


value of chunk: {'role': 'assistant', 'content': 'I'} 
PROCESSED CHUNK PRE CHUNK CREATOR: {'role': 'assistant', 'content': 'I'}

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Logging Details: logger_fn - None | callable(logger_fn) - False
HITS AN ERROR: the JSON object must be str, bytes or bytearray, not dict

 Traceback (most recent call last):
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5828, in chunk_creator
    response_obj = self.handle_ollama_stream(chunk)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5651, in handle_ollama_stream
    raise e
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5632, in handle_ollama_stream
    json_chunk = json.loads(chunk)
                 ^^^^^^^^^^^^^^^^^
  File "/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5932, in __next__
    response = self.chunk_creator(chunk=chunk)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5919, in chunk_creator
    raise exception_type(model=self.model, custom_llm_provider=self.custom_llm_provider, original_exception=e)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5192, in exception_type
    raise e
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5169, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: the JSON object must be str, bytes or bytearray, not dict

Logging Details LiteLLM-Failure Call
Traceback (most recent call last):
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5828, in chunk_creator
    response_obj = self.handle_ollama_stream(chunk)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5651, in handle_ollama_stream
    raise e
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5632, in handle_ollama_stream
    json_chunk = json.loads(chunk)
                 ^^^^^^^^^^^^^^^^^
  File "/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/lib/python3.11/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/flafy/repos/random/python-test2/main.py", line 13, in <module>
    for part in response:
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5946, in __next__
    raise e
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5932, in __next__
    response = self.chunk_creator(chunk=chunk)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5919, in chunk_creator
    raise exception_type(model=self.model, custom_llm_provider=self.custom_llm_provider, original_exception=e)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5192, in exception_type
    raise e
  File "/nix/store/y915lwwdggw5a2zs5zfissa3k2ia9jjg-python3-3.11.6-env/lib/python3.11/site-packages/litellm/utils.py", line 5169, in exception_type
    raise APIConnectionError( 
    ^^^^^^^^^^^^^^^^^^^^^^^^^
litellm.exceptions.APIConnectionError: the JSON object must be str, bytes or bytearray, not dict

Twitter / LinkedIn details

No response

[krrishdholakia]

try with ollama_chat/ and let me know if the issue persists

[FlafyDev]

ollama_chat/

I tried both model="ollama_chat/", and model="ollama_chat/llama2",

Both failed with

    raise litellm.exceptions.NotFoundError( # type: ignore
litellm.exceptions.NotFoundError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=ollama_chat/

 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers

Unless that's not what you meant?

[FlafyDev]

try with ollama_chat/ and let me know if the issue persists

@krrishdholakia just making sure you saw my response..

[FlafyDev]

any updates on this?

[nachouve]

I've tried with ollama/llama3:

from litellm import completion
import litellm

litellm.set_verbose=True

response = completion(
    model="ollama/llama3", 
    messages=[{ "content": "respond in 20 words. who are you?","role": "user"}], 
    api_base="http://localhost:11434",
    stream=True,
)
txt_resp = ""
for part in response:
    txt = part.choices[0].delta.content or ""
    txt_resp += txt
    print(txt)
print(txt_resp)

and it worked.