flink icon indicating copy to clipboard operation
flink copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-37184][connector/filesystem] Add ZStandard to supported standard decompressors

Open JoeryH opened this issue 10 months ago • 7 comments

What is the purpose of the change

  • Support ZStandard decompression for input files of the new File API. When ZStandard was first added it was only added to the old File API, the new File API was left out.

Brief change log

  • Added ZStandard to the list of decompressors in StandardDeCompressors

Verifying this change

  • Added unit test for StandardDeCompressors

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

JoeryH avatar Jan 20 '25 21:01 JoeryH

CI report:

  • 3ee951c5d65525e9e187e42f1a01a55aa8de3d24 Azure: FAILURE
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jan 20 '25 22:01 flinkbot

Reviewed by Chi on 23/01/2025 Go back to the submitter with review comments.

davidradl avatar Jan 24 '25 16:01 davidradl

@davidradl Thanks for looking at my PR. I'll add some sort of unit test tomorrow, if you have anything specific in mind, please let me know.

On the documentation I wholeheartedly agree. But I am pretty new to Flink and I don't know how to change the documentation or really even what to document. I'm still a little bit confused about the status of https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface. It's been quite a while, but the old API has better support for compressions: it has zstd added and it has an API to register more compressions for Flink users, while the new API has neither.

JoeryH avatar Jan 24 '25 17:01 JoeryH

@davidradl I added an unit test

JoeryH avatar Jan 25 '25 18:01 JoeryH

Would still like to have this merged; let me know if I need to do something

JoeryH avatar Feb 07 '25 23:02 JoeryH

This PR is being marked as stale since it has not had any activity in the last 90 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

github-actions[bot] avatar May 10 '25 06:05 github-actions[bot]

Would still like to merge this one!

JoeryH avatar May 28 '25 17:05 JoeryH

This PR is being marked as stale since it has not had any activity in the last 90 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

github-actions[bot] avatar Oct 23 '25 06:10 github-actions[bot]