zui icon indicating copy to clipboard operation
zui copied to clipboard

Add Parquet as an Export format

Open philrz opened this issue 2 years ago • 0 comments

At the time of the opening of this issue, Brim is at commit 9d0a742.

Since the backend already supports it, like we did for ZSON with #1486, it should be trivial to add Parquet as an Export format. However, as I was about to impulsively do this, I remembered that the backend still has this limitation:

$ zq -version
Version: v1.2.0-130-g8dd27d98

$ zq -f parquet 'fuse' ./src/test/shared/data/sample.tsv 
union type unsupported

Since union types often show up in Zed formats (particularly as a side effect of things like applying fuse) my inclination is to wait on an improvement in the backend before we add it as an Export option.

For now, I've got a branch https://github.com/brimdata/brim/tree/parquet-export where I've got it working for the happy path. Separately I opened up https://github.com/brimdata/zed/issues/4215 to track what I'm observing in the API interactions that are currently preventing the app from reporting the failure scenarios to the user.

philrz avatar Nov 16 '22 20:11 philrz