ComfyUI-Custom-Scripts icon indicating copy to clipboard operation
ComfyUI-Custom-Scripts copied to clipboard

use memory mapping to accelerate hashing of large models.

Open rfletchr opened this issue 5 months ago • 0 comments

This change optimizes the hashing of checkpoint files using memory mapping.

Essentially this makes file I/O far more efficient and allows the hasher to run extremely quickly.

A SDXL checkpoint was taking roughly 30 seconds to hash using the original implementation, this is reduced to about 6 seconds with memory mapping.

This could be accelerated further by switching to a non cryptographic hash like blake3 which is extremely fast, and supports multithreading.

rfletchr avatar Sep 06 '24 17:09 rfletchr