nydus icon indicating copy to clipboard operation
nydus copied to clipboard

feat(nydusify): add zero-disk streaming image transfer

Open daiyongxuan opened this issue 5 months ago • 1 comments

Implement a new streaming copy mechanism for image transfer that enables direct source-to-target streaming without using intermediate disk storage.

Key features:

  • Add StreamCopy for concurrent pull/push operations
  • Introduce streamTransferManager for orchestrating streaming workflows
  • Support chunked transfer with progress tracking
  • Add HTTP/HTTPS fallback with automatic retries
  • Support concurrent layer processing with configurable concurrency
  • Retain compatibility with traditional transfer mode

Performance improvements:

  • Eliminate need for intermediate disk storage
  • Lower memory footprint via streaming buffers
  • Enable concurrent image layer processing
  • Add progress logging for large transfers

Implementation details:

  • Introduce enableStreamTransfer configuration option
  • Add httpModeManager for connection switching logic
  • Optimize for both small and large file transfers
  • Add robust error handling and retry logic
  • Integrate with existing provider/resolver framework

This significantly improves performance for large image transfers by avoiding disk I/O bottlenecks.

daiyongxuan avatar Jun 19 '25 15:06 daiyongxuan