kafka-connect-ui icon indicating copy to clipboard operation
kafka-connect-ui copied to clipboard

Correct loading sources, incorrect loading sinks

Open laurensdv opened this issue 6 years ago • 5 comments

With the latest kafka-connect-ui docker (2018-03-21):

loading the sources is OK (when creating a new Kafka Connect Source)

OK source

but results in an empty (class) when creating a new sink.

This also prevents from activating the "CREATE" button when putting a new configuration in the input field (the validation of the input is broken).

Empty sink

I also get the following HTTP 500 error on /api/cluster/connector-plugins/com.datamountaineer.streamreactor.connect.mqtt.sink.MqttSinkConnector/config/validate (probably due to the empty properties template): {error_code: 500, message: "Must configure one of topics or topics.regex"} error_code : 500 message : "Must configure one of topics or topics.regex"

laurensdv avatar Apr 25 '18 14:04 laurensdv

@laurensdv I imagine you moved to Kafka 1.1. A bug has been introduced in Kafka 1.1 for connect. the validation requires an entry topics or topics.regex to be provided. Unfortunately the framework doesn't return the entries as required like it does with anything else specified in the connector ConfigDef. Unfortunate design approach. Anyway....

You need to add topics or topics.regex as a key in the connector when you create it.

stheppi avatar Apr 25 '18 14:04 stheppi

@stheppi That's correct, we moved tot Kafka 1.1.

I think this is not the only error, when I fill in 'topics', it still complains that the 'class' is missing, it results in an error that no class can be found that matches 'undefined' which makes sense because after clicking new on e.g. the MQTT sink, no class appears in the class: (no icon either). It's also strange that the sample configuration is not preloaded either (it used to before)

However as mentioned before, all of this works (filling in the class name and loading the sample configdef) with any (e.g. MQTT) source connector with the same Kafka deployment.

second

{"error_code":500,"message":"Failed to find any class that implements Connector and which name matches undefined, available connectors are: (a list of all available connectors ... ) }

laurensdv avatar Apr 26 '18 07:04 laurensdv

facing exactly same issue. any work around for now?

krishghata avatar May 16 '18 14:05 krishghata

Currently this line is forbidding you to get all the configs since they are collected through validate REST endpoint as @stheppi said https://github.com/jklukas/kafka/blob/ef1969011909bb6134af77ac03fc7d7c64eed3aa/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L264

This is new at Kafka v1.1 and here is issue on Kafka jira https://issues.apache.org/jira/browse/KAFKA-6806?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=16512110#comment-16512110

The only workaround is to use cli in console and push connectors through it

iMajna avatar Jun 14 '18 11:06 iMajna

When the error is: {error_code: 500, message: "Must configure one of topics or topics.regex"} error_code : 500 message : "Must configure one of topics or topics.regex" This means you have to add an "s" at the end of topic in configuration file. I had this issue and in the newer version of Kafka, they change the configuration from "topic" to "topics" When the error is: {"error_code":500,"message":"Failed to find any class that implements Connector and which name matches undefined, available connectors are: (a list of all available connectors ... ) } This means your plugin.path in connect-distributed.properties is not pointing to the right location (where all the plugins stored). I had this problem, and I tried different paths. It worked for me. For example: plugin.path=/connectors,C:/Users/1412502/Downloads/kafka_2.11-1.1.0/connectors.

Thalia-Long avatar Jul 11 '18 21:07 Thalia-Long