datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

docs: add example for custom file format with `COPY TO`

Open tshauck opened this issue 1 year ago • 1 comments

Which issue does this PR close?

Closes #11079

Rationale for this change

Adds an example of for how to COPY table TO a custom file format.

What changes are included in this PR?

Created sort of a mock file format factory and file format that just wrap the CSV one but for a TSV.

Are these changes tested?

yes, I've run the example

Are there any user-facing changes?

no

tshauck avatar Jun 29 '24 03:06 tshauck

~Seem to be failing in CI due to being out space on the worker.~ Potentially fixed with a change to the build script.

image image

tshauck avatar Jun 29 '24 14:06 tshauck

Yeah, I started down the road of something custom, but it turned out to be a non-trivial amount of code.

Perhaps adding a link to something like https://github.com/datafusion-contrib/datafusion-orc would show a more complex example, though it doesn't implement create_writer_physical_plan yet.

tshauck avatar Jul 01 '24 14:07 tshauck

Thanks again @tshauck and @devinjdangelo

alamb avatar Jul 02 '24 11:07 alamb