HuggingFaceModelDownloader
HuggingFaceModelDownloader copied to clipboard
refactor: improve download performance on networked drives by pre-allocating instead of using temp files
Improve download performance and progress reporting
Problem
The previous implementation used temporary files for each chunk of a download, which were later merged into the final file. This approach had several issues:
- Extra disk I/O from writing to temp files and then copying to the final location
- Poor performance on network drives due to unnecessary file operations
- Inaccurate download speed reporting due to how progress was tracked
- Wasted disk space from temporary files
Solution
This PR refactors the download mechanism to:
- Write directly to the final file using
WriteAt()with correct offsets - Pre-allocate the full file size to reduce fragmentation
- Track actual bytes downloaded per chunk for accurate progress reporting
- Eliminate the merge step entirely
Key Changes
- Added
downloadProgressstruct to properly track bytes read per chunk - Modified
downloadChunkto write directly to the target file using offsets - Improved progress monitoring to calculate actual download speed
- Removed temporary file handling and merging code
- Added elapsed time check to prevent division by zero in speed calculation
Benefits
- Faster downloads, especially on network drives
- Lower disk I/O
- More accurate progress reporting
- Reduced disk space usage
- Better handling of large files