fst icon indicating copy to clipboard operation
fst copied to clipboard

Optional connection argument to compress_fst

Open wlandau opened this issue 6 years ago • 2 comments

Writing directly to a connection could help conserve memory. Suggested at https://github.com/richfitz/storr/pull/111#issuecomment-509136783.

wlandau avatar Jul 12 '19 16:07 wlandau

Hi @wlandau, great, thanks for submitting!

We could go a little further and adapt compress_fst() to:

compress_fst(con_in, con_out = NULL, compressor = "ZSTD", compression = 0, hash = FALSE)

So, it would also accept a connection as the input (con_in) and/or the output (con_out).

That makes the function much more flexible. For example, with a connection selected for the input and for the output, a file can be compressed with very little RAM overhead.

When a raw vector is selected for the input and a connection for the output, we get to your scenario, where the compressed stream is directly written to disk...

MarcusKlik avatar Jul 12 '19 20:07 MarcusKlik

Nice! An optional output connection would certainly enhance versatility.

wlandau avatar Jul 13 '19 15:07 wlandau