Workaround for litellm proxy -> ollama issue

[clevcode]

When streaming responses from a model served by ollama, through a litellm proxy that provides an OpenAI compatible API, the last chunk (i.e. choice["delta"]["content"]) of the streaming response will be set to None

This causes an exception to be thrown during the following, since None can't be appended to a string: content += choice["delta"]["content"]

To avoid this, I simply added a check so that choice["delta"]["content"] is only appended to content if it's not None.

[clevcode]

I've researched the issue more now, and it's the very last chunk, containing the finish_reason field, that in the case of at least LiteLLM:s implementation of the OpenAI API, also sets the 'delta' field to {content: null, role: null} rather than just not setting the delta field at all in this case.

I think it's a good idea for this project to be resilient to that type of slight deviation from how the real OpenAI endpoints might respond, but I also think it should be fixed upstream in the litellm repo. I've already made some pull requests for a couple of other issues there when using litellm with ollama.

[cmungall]

@simonw - any chance of getting this PR merged? I think it makes sense for llm to be defensive in this way. I'm currently vendoring my llm as I really need this change.

@clevcode - do you have plans to add a fix that eliminates the Nones to litellm directly? Lmk if I can help with this

[simonw]

Sorry for not getting this into 0.13 - this will need some tweaks now that I've upgraded the OpenAI Python library.

[krrishdholakia]

Hey @clevcode i'm the maintainer of litellm

Tracking this issue on our end as well - https://github.com/BerriAI/litellm/issues/2010

Thanks for raising it.