(feat) NVIDIA fixes for chat completion impl

[raspawar]

NVIDIA Integration Enhancements for Chat Completion

This PR extends and improves the NVIDIA integration in LiteLLM by adding support for more standardised parameter naming and additional functionality.

Key Changes

• Added support for provider=nvidia (in addition to existing provider=nvidia_nim)
• Added support for NVIDIA_API_KEY environment variable (maintaining backward compatibility with NVIDIA_NIM_API_KEY)
• Improved model validation through NvidiaConfig().available_models()
• Added comprehensive test coverage for NVIDIA endpoints
• Maintained backward compatibility with existing NVIDIA integration

Type

🆕 New Feature 📖 Documentation ✅ Test

[REQUIRED] Testing - Attach a screenshot of any new tests passing locally

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 17, 2025 9:34am

[joaomdmoura]

+1 @mattf We could definitely use this 🙏 lmk if there is anything I cna do to get it merged

[ishaan-jaff]

hi we already have nvidia_nim support - what exactly is the issue ?

Can I see a example request to repro the issue @raspawar ?

[ishaan-jaff]

@joaomdmoura - what provider / model are you waiting on us to add support for ?

(python code would help so we can validate)

[raspawar]

Hi @ishaan-jaff, I am Rashmi from NVIDIA. Thanks for the integration and the existing integration works perfectly fine! This PR extends support to :

• parameters NVIDIA_API_KEY, provider=nvidia, etc. to maintain the uniform setup for nvidia endpoints across multiple frameworks.
• Model validation, checking available models through NvidiaConfig().available_models()
• test case coverage and more

[joaomdmoura]

hey @ishaan-jaff thanks for the answer! It's exactly what @raspawar said, I'm building a crew that uses nvidia so this will come super in handy :)

[raspawar]

@ishaan-jaff ptal

[raspawar]

@krrishdholakia would you be able to review changes in this PR too? Thanks in advance!

[krrishdholakia]

Hi @raspawar, thank you for your work. Currently your PR does not follow the conventions for an LLM integration.

For example - your folder structure needs to be <provider>/<endpoint>/transformation.py

Since your PR seems to be openai-compatible, you can see vllm for reference -

https://github.com/BerriAI/litellm/tree/e00d4fb18c2fe8400cc949bf8afb64e38e8f3927/litellm/llms/hosted_vllm

[raspawar]

Hey @krrishdholakia, @ishaan-jaff, just following up on this PR. It's been a couple of months, and it'd be great to get it merged soon so we don’t have to keep resolving conflicts. Let me know if there’s anything blocking it.

cc: @dglogo Thanks!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.