Assert in ZSTD_buildSeqStore when using ZSTD_compressContinue with a dictionary

[gdhuse-ocient]

Hello, I'm hitting the assert on zstd_compress.c:2837 (v1.5.2) when calling ZSTD_compressContinue with a dictionary (attached using ZSTD_compressBegin_usingCDict).

This code uses a manually-allocated circular buffer of 1 << (windowLog+1) (I've reproduced with multiple window sizes including windowLog == 17) and compresses no more than half the buffer in each call to ZSTD_compressContinue to ensure that 1 << windowLog bytes of previous data are always available. The assert is only hit when a dictionary is attached (in this case the dictionary varies in size <= 128KB).

Does this sound like correct use of the buffer-less streaming API? Are there any additional requirements/caveats around using this API with a dictionary? Tracing through the code around this assert it looks like there are cases where the dictionary cannot be used -- when that happens, is there a potential impact on compression?

Thanks in advance for your help. I've attached a log captured with DEBUGLEVEL=6 and can work on trying to come up with a minimal test case to reproduce if needed.

Cheers, Greg

test.log.gz

[terrelln]

An example of your usage of the compression API would be useful for debugging, though probably not 100% necessary.

What is your use-case for the buffer-less API? Could you use the streaming API ZSTD_compressStream2() instead? The buffer-less API is not super well supported, and I generally wouldn't recommend using it.

At some point in the future, I would like to deprecate the buffer-less API. And capture those use cases with the streaming API. So hearing how & why you're using this API would be super helpful.

[gdhuse-ocient]

Hi, and thanks for the response. My eventual goal is to avoid large internal allocations and use a collection of externally provided buffers where I don't control the size, so it wouldn't be possible to just configure a custom memory allocator with ZSTD_compressStream2.