#1360 Support for poll maximum wait time

[jakubmalek]

trafficstars

Problem

The JDBC source task stoppage often results in the ungraceful shutdown, due to a long duration time for the polling operation which blocks the worker task thread. If the connector is restarted and re-deployed on the same node, it may result in a missing JMX bean for the connector as the newly registered metrics will be removed once the task is finally stopped after graceful timeout. This can case issues with monitoring as the metrics needs to be exported from the JMX.

Solution

The proposed solution is to introduce a new configuration property poll.max.wait.time.ms which limits the duration for which the worker task is blocked for a single polling call. To make it possible, the polling operation should be run in a separated thread, where the task awaits for it to finish w up to the configured poll.max.wait.time.ms duration. If the operation is not finished within poll.max.wait.time.ms the source task will return null list of records signaling no-data to the worker. As the worker task runs in a loop, where its continuously polling the source task, the task will check if there is a previously started poll operation, and try to wait for it to finish each time. If there is no previous operation started, it will start new one, and apply the same logic. This in consequence should limit the total duration for which the worker task thread can be blocked, thus allowing the graceful shutdown, regardless of the poll.interval.ms value nor the records fetching time.

For the backward compatibility it should be possible to set the poll.max.wait.time.ms to 0 where the task waits indefinitely for the polling result, without a separated thread.

Does this solution apply anywhere else?

• [x] yes
• [ ] no

If yes, where?

Similar strategy can be applied to other types of source connectors where worker thread is blocked for extended time period during the SourceTask::poll call.

Test Strategy

To allow easier testing, I've extracted the related logic into JdbcSourceTaskPollExecutor class.

Testing done:

• [x] Unit tests
• [ ] Integration tests
• [ ] System tests
• [ ] Manual tests

Release Plan

[cla-assistant[bot]]

All committers have signed the CLA.

[cla-assistant[bot]]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}