[Bug]: Watsonx embeddings `authentication_token_expired`

[Darxeal]

What happened?

It seems that Watsonx embeddings endpoint does not refresh the token. Initially, the endpoint works fine, but after a while the token expires and results in an error: litellm.exceptions.AuthenticationError: litellm.AuthenticationError: watsonxException - {"errors":[{"code":"authentication_token_expired","message":"Failed to authenticate the request due to an expired token","more_info":"https://cloud.ibm.com/apidocs/watsonx-ai"}],"trace":"redacted","status_code":401}. Received Model Group=granite-embedding-278m-multilingual

Using proxy server. Chat completions for Watsonx does not have this issue.

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

latest

Twitter / LinkedIn details

No response

[araujof]

I'm experiencing the same issue. Started happening after I updated to the latest litellm (1.67.2).

litellm.exceptions.AuthenticationError: litellm.AuthenticationError: watsonxException - {"errors":[{"code":"authentication_token_expired","message":"Failed to authenticate the request due to an expired token","more_info":"https://cloud.ibm.com/apidocs/watsonx-ai"}],"trace":"...","status_code":401}

[terylt]

@Darxeal Nice to meet you. I dug into this issue more deeply as I was having the same issue. I created a PR with a fix here: https://github.com/BerriAI/litellm/pull/10728

Are you able to test just to make sure it's working for you?

cc: @araujof