chatdocs icon indicating copy to clipboard operation
chatdocs copied to clipboard
verified publisher icon marella

Error when parquet files get too big / function for splitting?

Open Ananderz opened this issue 2 years ago • 0 comments

Hi!

I have been uploading a lot of data and ran into a snappy compress error after reaching around 3,6GB of data in the parquet file.

Error: Invalid Error: Snappy decompression failure

I saw that there was a limit for parquetfiles and that limit is 4GB. Could we add functionality to split the parquet files when they reach 1 GB of data to get rid of this issue. Does anyone know how to do it ?

Ananderz avatar Jun 30 '23 23:06 Ananderz