kafka-connect-jdbc icon indicating copy to clipboard operation
kafka-connect-jdbc copied to clipboard

feat: adding pollingtimestamp metadata in bulk mode

Open sachin-kansal opened this issue 11 months ago • 4 comments

Problem

While using bulk mode in JDBC source connector, polling time is not sent to the records metadata. Hence insert field smt does not work to add polling time in the records.

Also whenever the jdbc is restarted polling is started again results in reading the table again or duplicate data and make difficult to bifurcate data which is added in the queue at each poll.

Solution

adding timestamp helps in the bifurcation.

Does this solution apply anywhere else?
  • [ ] yes
  • [ ] no
If yes, where?

Test Strategy

end to end dataflow check

Testing done:
  • [ ] Unit tests
  • [ ] Integration tests
  • [ ] System tests
  • [ ] Manual tests

Release Plan

sachin-kansal avatar Dec 21 '24 19:12 sachin-kansal

:tada: All Contributor License Agreements have been signed. Ready to merge.
:white_check_mark: sachin-kansal
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

@confluent-antonio please review my Pull Request

sachin-kansal avatar Mar 10 '25 02:03 sachin-kansal

@aniketshrimal , could please review my Pull Request

sachin-kansal avatar Mar 10 '25 02:03 sachin-kansal

@confluent-antonio please review my Pull Request

I'm sorry, I'm not a developer. I can't review your PR.

confluent-antonio avatar Mar 10 '25 07:03 confluent-antonio

Hi @sangeet259, @rhauch Could you please review my Pull request.

sachin-kansal avatar Aug 12 '25 17:08 sachin-kansal

hi @petersilverwood, could you plase review my Pull Request.

sachin-kansal avatar Aug 22 '25 18:08 sachin-kansal