Class not found although plugin loaded

[mmajis]

Thanks for your work on this plugin! I'm trying the XML transform but am unable to get it running. Any ideas what I might be doing wrong?

I POST this to http://my-connect:8083/connectors:

{
    "name": "file-source",
    "config": {
		"connector.class": "FileStreamSource",
		"tasks.max": "1",
		"file": "/tmp/test.xml",
		"name": "file-source",
                 "topic": "mytopic",
                 "value.converter": "org.apache.kafka.connect.storage.StringConverter",
                 "key.converter": "org.apache.kafka.connect.storage.StringConverter",
                 "transforms": "FromXml",
		"transforms.FromXml.type": "com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value",
		"transforms.FromXml.schema.path": "https://my-schema/schema.xsd"
    }
}

This results in the following response:

{
    "error_code": 400,
    "message": "Connector configuration is invalid and contains the following 2 error(s):\nInvalid value com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value for configuration transforms.FromXml.type: Class com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value could not be found.\nInvalid value null for configuration transforms.FromXml.type: Not a Transformation\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`"
}

I'm installing the plugin like this in a Dockerfile:

FROM confluentinc/cp-kafka-connect:5.3.1

RUN confluent-hub install --no-prompt jcustenborder/kafka-connect-transform-xml:0.1.0.12

In the logs, I can see the plugin loading:

[2019-10-10 10:23:11,721] INFO Loading plugin from: /usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-xml (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-xml/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.PatternRename$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.BytesToString$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ToJSON$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,994] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ToJSON$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,994] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.BytesToString$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,994] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ExtractTimestamp$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,994] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,994] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.ChangeTopicCase' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,994] INFO Added plugin 'com.github.jcustenborder.kafka.connect.transform.common.PatternRename$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:13,227] INFO Added alias 'ChangeTopicCase' to plugin 'com.github.jcustenborder.kafka.connect.transform.common.ChangeTopicCase' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

If I try to validate the config, I get the transform listed as a recommended value so looks like the transform is loaded:

            "value": {
                "name": "transforms.FromXml.type",
                "value": null,
                "recommended_values": [
                    "com.github.jcustenborder.kafka.connect.transform.common.BytesToString$Key",
                    "com.github.jcustenborder.kafka.connect.transform.common.BytesToString$Value",
                    "com.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Key",
                    "com.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Value",
                    "com.github.jcustenborder.kafka.connect.transform.common.ChangeTopicCase",
                    "com.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Key",
                    "com.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Value",
                    "com.github.jcustenborder.kafka.connect.transform.common.ExtractTimestamp$Value",
                    "com.github.jcustenborder.kafka.connect.transform.common.PatternRename$Key",
                    "com.github.jcustenborder.kafka.connect.transform.common.PatternRename$Value",
                    "com.github.jcustenborder.kafka.connect.transform.common.ToJSON$Key",
                    "com.github.jcustenborder.kafka.connect.transform.common.ToJSON$Value",
                    "com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Key",
                    "com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value",
                    "org.apache.kafka.connect.transforms.Cast$Key",
                    "org.apache.kafka.connect.transforms.Cast$Value",
                    "org.apache.kafka.connect.transforms.ExtractField$Key",
                    "org.apache.kafka.connect.transforms.ExtractField$Value",
                    "org.apache.kafka.connect.transforms.Flatten$Key",
                    "org.apache.kafka.connect.transforms.Flatten$Value",
                    "org.apache.kafka.connect.transforms.HoistField$Key",
                    "org.apache.kafka.connect.transforms.HoistField$Value",
                    "org.apache.kafka.connect.transforms.InsertField$Key",
                    "org.apache.kafka.connect.transforms.InsertField$Value",
                    "org.apache.kafka.connect.transforms.MaskField$Key",
                    "org.apache.kafka.connect.transforms.MaskField$Value",
                    "org.apache.kafka.connect.transforms.RegexRouter",
                    "org.apache.kafka.connect.transforms.ReplaceField$Key",
                    "org.apache.kafka.connect.transforms.ReplaceField$Value",
                    "org.apache.kafka.connect.transforms.SetSchemaMetadata$Key",
                    "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
                    "org.apache.kafka.connect.transforms.TimestampConverter$Key",
                    "org.apache.kafka.connect.transforms.TimestampConverter$Value",
                    "org.apache.kafka.connect.transforms.TimestampRouter",
                    "org.apache.kafka.connect.transforms.ValueToKey"
                ],
                "errors": [
                    "Invalid value com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value for configuration transforms.FromXml.type: Class com.github.jcustenborder.kafka.connect.transform.xml.FromXml$Value could not be found.",
                    "Invalid value null for configuration transforms.FromXml.type: Not a Transformation"
                ],
                "visible": true
            }
        },

Thanks!

[jcustenborder]

hmmm do you have more than one node in the connect cluster?

[mmajis]

I’m running this with the Confluent cp-all-in-one docker-compose setup with the connect container switched to the one that has the plugin installed. Only one connect container.

I’ll see if I can push a git repo where this can be reproduced.

[mmajis]

Here's a repro environment: https://github.com/mmajis/xml-transform-class-not-found

[med4it]

I have the same issue. is there any update on this? Plugin installed with confluent-hub install jcustenborder/kafka-connect-transform-xml:0.1.0.18

[jcustenborder]

@mmajis @med4it I'm sorry. I totally missed this issue. Which version of kafka connect are you running? Where did confluent-hub place the connector? What is your plugin.path setting on your worker(s)? Did you install this on all of your workers and restart all of your workers?

[med4it]

@jcustenborder I use confluentinc/cp-kafka-connect:5.5.0 The connector is placed in /usr/share/confluent-hub-components/ directory. I verified plugin.path in both files connect-standalone.properties and connect-distributed.properties : plugin.path=/usr/share/java,/usr/share/confluent-hub-components

Thank you for the reply

[med4it]

confluent-hub install --no-prompt jcustenborder/kafka-connect-transform-xml:latest Worked for me.

[jcustenborder]

@med4it Can you give me the output of this command? Also can I have the logs when it's loading the plugins.

find /usr/share/confluent-hub-components/kafka-connect-transform-xml

[2019-10-10 10:23:11,721] INFO Loading plugin from: /usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-xml (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2019-10-10 10:23:11,993] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-xml/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

[jcustenborder]

@med4it Wait is it working for you? @mmajis Can you try deleting the directory and downloading the latest version?

[med4it]

Yes, it does work! It is on /usr/share/confluent-hub-components/jcustenborder-kafka-connect-transform-xml/ directory.