apache
[Umbrella] Flink Support (Help Wanted)
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Search before asking
- [X] I have searched in the issues and found no similar issues.
Describe the proposal
Flink Support is complex topic. We set up an issue here. In Flink Community, Flink Shuffle will have much change because of https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage. We should discuss actively with Flink community. Their discussion thread https://lists.apache.org/thread/nwrqd5jtqwks89tbxpcrgto6r2bhdhno
Task list
xx
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
This document can also be used as a reference.
FLIP-31: Pluggable Shuffle Service
FLIP-148: Introduce Sort-Based Blocking Shuffle to Flink
FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework
FLIP-199: Change some default config values of blocking shuffle for better usability
FLIP-209: Support to run multiple shuffle plugins in one session cluster
FLIP-301: Hybrid Shuffle supports Remote Storage
Vedio: Pluggable Shuffle Service and Unaligned Checkpoints
https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
@jerqi I will follow up the issue with chenglong(Big Data Development Engineer In Lixiang Car).
Hi, @jerqi and the team,
I am interested in this project due to investigating Flink batch shuffle service in my previous job.
Would I still be able to join this project?
Hi, @jerqi and the team,
I am interested in this project due to investigating Flink batch shuffle service in my previous job.
Would I still be able to join this project?
You're welcome. Currently, we have a regular meeting. If you have interest and can Speak Chinese, you can join the Wechat Group, the meeting link is the Wechat Group.
cc @slfan1989 Do you already have a draft implementation? I remember you mentioned that in the online meeting
cc @slfan1989 Do you already have a draft implementation? I remember you mentioned that in the online meeting
@zuston My draft version code has been basically completed (but some details still need to be optimized), I will submit PR to the community as soon as possible.