Fix CUDA Compatibility Issues with PyTorch and Large Models

[groxaxo]

This PR resolves critical CUDA compatibility issues that were preventing the application from running with large models on GPU. The main problem was the use of CUDA 12.9.1, which is not officially supported by PyTorch.

Summary of Changes

• Dockerfile Updates: Changed base images to CUDA 12.1.1 (officially supported by PyTorch) and updated PyTorch installations to use CUDA 12.1 compatible packages
• Documentation Improvements: Added comprehensive CUDA setup guide and updated README with driver requirements and usage instructions
• Dependency Management: Added minimum version requirements for torch (>=2.1.0) and torchaudio (>=2.1.0) in pyproject.toml
• Testing Infrastructure: Added test files to verify CUDA compatibility in containerized environments

Why CUDA 12.1?

• Officially supported by current PyTorch versions
• Compatible with widely available NVIDIA drivers (515+)
• Provides good performance for Whisper models
• Balances compatibility with performance

Testing

To verify the fixes:

# Test CUDA compatibility
docker build -t cuda-test -f Dockerfile.test .
docker run --gpus all cuda-test

# Test with actual GPU container
docker build -t whisperlivekit-gpu -f Dockerfile.gpu .
docker run --gpus all -p 8000:8000 whisperlivekit-gpu --model large-v3

These changes ensure WhisperLiveKit works reliably with GPU acceleration, including support for large models.

Related Documentation

• CUDA Fixes Summary
• GPU Setup Guide

[ronaldvh95]

Thnx

[Vladimir-Anfimov]

Hi, when will this be deployed?

[QuentinFuxa]

Hi, when will this be deployed?

Hi, you probably used some AI coding for the fix since the statement that torch only supports cuda up to 12.1.1 is false:

So that is not True anymore, no need for the downgrade