iscc-specs icon indicating copy to clipboard operation
iscc-specs copied to clipboard

Change Data-ID to chunking algorithm compatible with other implementations.

Open titusz opened this issue 4 years ago • 0 comments

Recently some implementations of FastCDC in different programming languages have shown up that are compatible to each other:

https://github.com/ronomon/deduplication https://github.com/nlfiedler/fastcdc-rs https://github.com/titusz/fastcdc-py

We should join this development and update the Data-ID chunking algorithm to be in line with those implementations. We also should implement an optional accelerated version as the 5 MB/s of a pure python version is a major performance bottleneck.

titusz avatar Jun 30 '20 11:06 titusz