ksql
ksql copied to clipboard
DESCRIBE CONNECTOR should show aggregate task status next to connector state
ksql> DESCRIBE CONNECTOR SOURCE_JDBC_MYSQL_01;
Name : SOURCE_JDBC_MYSQL_01
Class : io.confluent.connect.jdbc.JdbcSourceConnector
Type : source
State : RUNNING
WorkerId : kafka-connect:8083
Task ID | State | Error Trace
---------------------------------------------------------------------------------------------------------------------------------------------
0 | FAILED | java.lang.ClassCastException: java.lang.String cannot be cast to java.sql.Timestamp
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractOffsetTimestamp(TimestampIncrementingCriteria.java:217)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractValues(TimestampIncrementingCriteria.java:187)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:192)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:316)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:245)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:221)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The headline here is State : RUNNING
but then there's a failed task. I think it would be a good idea to take the task(s) and include in the connector state, e.g.
-
State : RUNNING (all tasks healthy)
-
State : RUNNING (note: some tasks FAILED, see below)
-
State : RUNNING (note: all tasks FAILED, see below)
So DESCRIBE
output would then be:
ksql> DESCRIBE CONNECTOR SOURCE_JDBC_MYSQL_01;
Name : SOURCE_JDBC_MYSQL_01
Class : io.confluent.connect.jdbc.JdbcSourceConnector
Type : source
State : RUNNING (note: all tasks FAILED, see below)
WorkerId : kafka-connect:8083
Task ID | State | Error Trace
---------------------------------------------------------------------------------------------------------------------------------------------
0 | FAILED | java.lang.ClassCastException: java.lang.String cannot be cast to java.sql.Timestamp
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractOffsetTimestamp(TimestampIncrementingCriteria.java:217)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractValues(TimestampIncrementingCriteria.java:187)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:192)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:316)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:245)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:221)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
just fyi, this is not incorrect. when running a connector with k tasks, the workers actually spins up (k + 1) entities. The additional one being the connector itself. the RUNNING state you are seeing there is the health of that entity. A connector can have all the tasks failed, but still would as RUNNING. There was some discussion around adding a DEGRADED state to a connector, to reflect this situation, so that users do not get confused. I think that might help this situation.