ComfyUI-Custom-Scripts
ComfyUI-Custom-Scripts copied to clipboard
use memory mapping to accelerate hashing of large models.
This change optimizes the hashing of checkpoint files using memory mapping.
Essentially this makes file I/O far more efficient and allows the hasher to run extremely quickly.
A SDXL checkpoint was taking roughly 30 seconds to hash using the original implementation, this is reduced to about 6 seconds with memory mapping.
This could be accelerated further by switching to a non cryptographic hash like blake3 which is extremely fast, and supports multithreading.