kcctl icon indicating copy to clipboard operation
kcctl copied to clipboard

Support new restart semantics in Kafka 3.0

Open gunnarmorling opened this issue 3 years ago • 3 comments

Kafka Connect 3.0 has the ability to restart all the failed tasks of a connector:

POST /connectors/{name}/restart?includeTasks=<true|false>&onlyFailed=<true|false>

More info:

  • https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
  • https://github.com/apache/kafka/pull/10975

The existing kcctl restart connector <name> command should be enhanced with a parameter for leveraging this: --tasks=<all|failed>. ~~Using these parameters with Connect < 3.0 should be ignored but log a warning. Completion of those options should not be done for Connect 3.0.~~

Thinking more about it, we actually could support this for any KC version, using the new (and more efficient API) if available, otherwise falling back to multiple separate API calls.

gunnarmorling avatar Nov 19 '21 22:11 gunnarmorling

// CC @tonyfosterdev

gunnarmorling avatar Nov 19 '21 22:11 gunnarmorling

That is a nice feature. I'll look at it over the next few weeks

tonyfosterdev avatar Nov 20 '21 17:11 tonyfosterdev

I am on this

SiddiqueAhmad avatar Feb 05 '22 09:02 SiddiqueAhmad

@C0urante, do you happen to know from the top of your head whether any work is left to be done here?

gunnarmorling avatar Jul 31 '23 14:07 gunnarmorling

Yeah, I think this feature hasn't been implemented at all yet. Don't see anything in the code base or the usage summaries for restart [connector|task] that mentions support of restarting multiple tasks.

I'll throw up a PR today that at least adds support for 3.0+; would be nice to have this included in 1.0.0.

C0urante avatar Jul 31 '23 15:07 C0urante

Ah yes, thanks! I somehow thought that might have been implemented already. Thanks a lot for looking into this!

gunnarmorling avatar Jul 31 '23 16:07 gunnarmorling