Add support for compression levels to file sink

[bruceg]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The file sink has configuration for the algorithm used to compress the data before writing to the files, but not for the associated parameters of the algorithm, notably the compression "level". Users of the file sink want to choose the trade offs of CPU time versus file size, which can only be accomplished by allowing for customizing the compression level.

Attempted Solutions

A user contribution was provided to solve this issue in #14349, but we decided to go with a different approach.

Proposal

The sink batch buffer handling has support for compression level configuration. We can use that structure to configure the file sink as well to ensure a unified UX for these components. That the batch buffer code is missing the Zstandard compression that is already supported here, so this depends on the completion of #2302.

References

No response

Version

No response