Externalize the secrets and other info not working

[smasilamani-cfins]

trafficstars

Hello

I am working on externalize the secret strings from the connectors config to AWS secret manager. But for some reason I keep getting below error just for URL. From the log, I can see that the server name is actually replaced but still getting the same error. However, If replace just the URL with the actual value, then it works. Please advise.

connector config:

{
    "name": "test.es.es",
    "config": {
        "config.providers": "secretManager",
        "config.providers.secretManager.class": "com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider",
        "config.providers.secretManager.param.aws.region": "us-east-1",
        "config.providers.secretManager.param.secret.prefix": "",
        "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
        "tasks.max": "1",
        "connection.url": "https://${secretManager:dev/kafka/es:host}",
        "connection.username": "${secretManager:dev/kafka/es:username}",
        "connection.password": "${secretManager:dev/kafka/es:password}",
        "elastic.security.protocol": "SSL",
        "elastic.https.ssl.truststore.location": "/usr/lib/jvm/zulu11/lib/security/cacerts",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter.schema.registry.url": "${secretManager:dev/kafka/others:schema_reg_url}",
        "type.name": "_doc",
        "topics": "test.es",
        "transforms": "renameTopicToIndex,createKey,extractKey",
        "transforms.renameTopicToIndex.type": "org.apache.kafka.connect.transforms.RegexRouter",
        "transforms.renameTopicToIndex.regex": ".*",
        "transforms.renameTopicToIndex.replacement": "test.es",
        "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
        "transforms.createKey.fields": "pk",
        "transforms.extractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
        "transforms.extractKey.field": "pk",
        "schema.ignore": true,
        "consumer.security.protocol": "SASL_SSL",
        "consumer.sasl.mechanism": "AWS_MSK_IAM",
        "consumer.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;",
        "consumer.sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    }
}

REST API response:

{ "error_code": 400, "message": "Connector configuration is invalid and contains the following 1 error(s):\nInvalid value [https://${secretManager:dev/ssl/kafka/es:host}] for configuration connection.url: The provided url 'https://${secretManager:dev/ssl/kafka/es:host}' is not a valid url.\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate" }

Connect log:

2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,142] INFO SecretsManagerConfigProviderConfig values: 2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 aws.access.key = 2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 aws.region = us-east-1 2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 aws.secret.key = [hidden] 2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 secret.prefix = 2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 secret.ttl.ms = 300000 2021-06-05T16:21:16.143000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 (com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProviderConfig) 2021-06-05T16:21:16.146000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,146] INFO get() - path = 'dev/kafka/others' keys = '[schema_reg_url]' (com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider) 2021-06-05T16:21:16.219000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,219] INFO get() - path = 'dev/kafka/es' keys = '[password, host, username]' (com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider) 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,284] INFO ElasticsearchSinkConnectorConfig values: 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 batch.size = 2000 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 behavior.on.malformed.documents = FAIL 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 behavior.on.null.values = FAIL 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 compact.map.entries = true 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 connection.compression = false 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 connection.password = [hidden] 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 connection.timeout.ms = 1000 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 connection.url = [https://test-server.es.amazonaws.com] 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 connection.username = dev.ssl.kafka 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 drop.invalid.message = false 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.cipher.suites = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.enabled.protocols = [TLSv1.2, TLSv1.3] 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.endpoint.identification.algorithm = https 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.engine.factory.class = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.key.password = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.keymanager.algorithm = SunX509 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.keystore.certificate.chain = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.keystore.key = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.keystore.location = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.keystore.password = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.keystore.type = JKS 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.protocol = TLSv1.3 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.provider = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.secure.random.implementation = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.trustmanager.algorithm = PKIX 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.truststore.certificates = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.truststore.location = /usr/lib/jvm/zulu11/lib/security/cacerts 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.truststore.password = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.https.ssl.truststore.type = JKS 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 elastic.security.protocol = SSL 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 flush.timeout.ms = 180000 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 kerberos.keytab.path = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 kerberos.user.principal = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 key.ignore = false 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 linger.ms = 1 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 max.buffered.records = 20000 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 max.connection.idle.time.ms = 60000 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 max.in.flight.requests = 5 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 max.retries = 5 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 proxy.host = 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 proxy.password = null 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 proxy.port = 8080 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 proxy.username = 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 read.timeout.ms = 3000 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 retry.backoff.ms = 100 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 schema.ignore = true 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 topic.key.ignore = [] 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 topic.schema.ignore = [] 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 write.method = INSERT 2021-06-05T16:21:16.284000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 (io.confluent.connect.elasticsearch.ElasticsearchSinkConnectorConfig) 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,285] INFO SecretsManagerConfigProviderConfig values: 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 aws.access.key = 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 aws.region = us-east-1 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 aws.secret.key = [hidden] 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 secret.prefix = 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 secret.ttl.ms = 300000 2021-06-05T16:21:16.285000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 (com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProviderConfig) 2021-06-05T16:21:16.286000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,286] INFO get() - path = 'dev/kafka/others' keys = '[schema_reg_url]' (com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider) 2021-06-05T16:21:16.353000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,353] INFO get() - path = 'dev/kafka/es' keys = '[password, host, username]' (com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider) 2021-06-05T16:21:16.410000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 [2021-06-05 16:21:16,410] INFO AbstractConfig values: 2021-06-05T16:21:16.410000+00:00 ecs/dev-ssl-kafka-connect/7f38f24b4c6d4aa8935909f2ebdb3f64 (org.apache.kafka.common.config.AbstractConfig)

[dcgomesbr]

Interesting, I'm having the same issue with MongoDB source driver. Seeing your post from Jun 5 with no comments is... sad.

[vl-kp]

any update?