Source connector not working on Kafka 1.1.0

[scholzj]

I used the TwitterSourceConnector and it worked fine on Kafka 1.0.1 and lower. But after migrating to Kafka 1.1.0, it doesn't work anymore. When I try to deploy the connector, Kafka Connect throws following exception:

[2018-05-11 16:37:57,427] ERROR Uncaught exception in REST call to /connectors (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
org.apache.kafka.common.config.ConfigException: Must configure one of topics or topics.regex
        at org.apache.kafka.connect.runtime.SinkConnectorConfig.validate(SinkConnectorConfig.java:71)
        at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:264)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:534)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:531)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:267)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:216)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2018-05-11 16:37:57,433] INFO 0:0:0:0:0:0:0:1 - - [11/May/2018:14:37:57 +0000] "POST /connectors HTTP/1.1" 500 75  68 (org.apache.kafka.connect.runtime.rest.RestServer:60)

This seems to be cause by the TwitterSourceConnector type not being recognised - it seems Kafka 1.1.0 started suddenly treating connectors of unknown type as sink connectors and the validation of the configuration fails.

$ curl http://localhost:8083/connector-plugins | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   685  100   685    0     0   2208      0 --:--:-- --:--:-- --:--:--  2216
[
  {
    "class": "com.eneco.trading.kafka.connect.twitter.TwitterSinkConnector",
    "type": "sink",
    "version": ""
  },
  {
    "class": "com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector",
    "type": "unknown",
    "version": ""
  }
]

The problem seems to be in the TwitterSourceConnector class which extends the generic Connector class. This PR fixes this and makes the TwitterSourceConnector extend the SourceConnector instead of Connector class. Thanks to this it is now properly recognised as source connector and I can create an instance.

[rsjain1978]

Hi, I was having exactly the same issue, didn't realized that there is a pull request opened for it.

Glad you've been able to fix it.

Thanks

[scholzj]

@rsjain1978 Lets hope that someone can merge this :-/

[liliankasem]

It's been about 3 years, any chance we can merge this :D