apache
Remove temporary KafkaSourceDescriptor.create function
In https://github.com/apache/beam/commit/b609aab9e2bd9e8d86c8e3a511c22a4402c5704e (https://github.com/apache/beam/pull/12518), it is added to make a workaround for bug in AutoValueSchema. It seems fixed now (https://github.com/apache/beam/pull/12520), so I suggest to remove this function.
Feel free to close this PR if I got something wrong.
I found this code change by investigating my issue, which is #36356. When updating Dataflow job, it occurred NullPointerException because of topic is null. (Drain and re-deploy worked well, but it happens only in existing job update scenarioes)
I doubt this code change would resolve the original issue, but I believe this change might be needed in any way.
Related PRs -> https://github.com/apache/beam/pull/36295
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead. - [x] Update
CHANGES.mdwith noteworthy changes. - [x] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.
Assigning reviewers:
R: @kennknowles for label java. R: @Dippatel98 for label kafka.
Note: If you would like to opt out of this review, comment assign to next reviewer.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Note for reviewers: I can't recall if these objects get serialized into the pipeline graph, but if that's the case then we can't remove this static method because the auto-generated serialVersionUID would change (see https://docs.oracle.com/javase/8/docs/platform/serialization/spec/class.html#a4100) and break update compatibility. 🙃
I think they may be both serialized to the graph and shipped in Kafka SDF using SerializableCoder (haven't checked). So, yes, this could be an issue.
@kennknowles So, could you please tell me what should I do now?
If this change breaks something, it would be better to just comment not to change this due to backward-compatibility, and just keep use KafkaSourceDescriptor.create function.
This PR might mitigate potential issues in any way.
Please feel free to close this PR if needed.
Yea it is pretty hard to test. You would have to start a pipeline with the released version of the SDK and then use this version of the SDK to update. Right now we don't have good automation around it in the Beam project.
It is OK to break update compatibility but we have to put in a flag like here: https://github.com/apache/beam/pull/28853/files#diff-b8cf6c3051a36c566f2f28f525449f456a88b05b3b4c17c814e6a55ba2ce36e9R77-R90
Reminder, please take a look at this pr: @kennknowles @Dippatel98
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:
R: @Abacn for label java. R: @johnjcasey for label kafka.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:
R: @chamikaramj for label java. R: @Abacn for label kafka.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:
R: @kennknowles for label java. R: @johnjcasey for label kafka.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
Reminder, please take a look at this pr: @kennknowles @johnjcasey
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:
R: @ahmedabu98 for label java. R: @chamikaramj for label kafka.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
Reminder, please take a look at this pr: @ahmedabu98 @chamikaramj
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:
R: @Abacn for label java. R: @johnjcasey for label kafka.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:
R: @kennknowles for label java. R: @damccorm for label kafka.
Available commands:
stop reviewer notifications- opt out of the automated review toolingremind me after tests pass- tag the comment author after tests passwaiting on author- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)