Fix/issue 10113 embeddings use non default tokenizer

[camfarineau]

Title

Embeddings: Not use default gpt-3.5-turbo's tokenizer

Relevant issues

Fixes https://github.com/BerriAI/litellm/issues/10113

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• [x] I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• [x] I have added a screenshot of my new test passing locally
• [x] My PR passes all unit tests on make test-unit
• [x] My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🐛 Bug Fix

Changes

• embeddings: allow for passthrough of list of lists of tokens to self hosted vllm models

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 13, 2025 8:46am

[CLAassistant]

All committers have signed the CLA.

[camfarineau]

Screenshot of the new test passing locally