flink-http-connector
flink-http-connector copied to clipboard
getindata
Support Flink 2
There may be a different java version. Also there is a Flink lineage tracing capability that we may want to take advantage of. The dev list has a post proposing lots of deliverables for v1 pre lineage and V1 with lineage and V2 Kafka connector.
Sure @davidradl ! If you needed help with lineage APIs and/or OpenLineage, our colleagues from Getindata actually implemented it, like @pawel-big-lebowski :) I will also help with getting reviewers for that.
Sure @davidradl ! If you needed help with lineage APIs and/or OpenLineage, our colleagues from Getindata actually implemented it, like @pawel-big-lebowski :) I will also help with getting reviewers for that.
sounds good. I will see how the Kafka connector plays out, in terms of java levels. before looking at this.
@davidradl I think we should be ready soon with the first version of job listener that converts Flink lineage into OpenLineage events. We'll bring more details during Flink Forward talk https://www.flink-forward.org/berlin-2024/agenda#data-lineage-for-apache-flink-with-openlineage
@davidradl I think we should be ready soon with the first version of job listener that converts Flink lineage into OpenLineage events. We'll bring more details during Flink Forward talk https://www.flink-forward.org/berlin-2024/agenda#data-lineage-for-apache-flink-with-openlineage
sounds good - I am attending so will come along and say hello!
I tried to estimate how much work is needed to migrate to Flink 2.0, so I updated the dependency version to see how much code need to be refactored. It turns out that only a few changes are needed (draft PR). Another iteration will definitely be needed. I’ll try to go through the release notes to see what else needs to be adjusted.
