The plugin is not translating secrets from Secret Manager Service correctly

[koren-at-fundbox]

I'm not sure if this is a real issue or a misconfiguration. Our stack includes:

• RDS with MySql engine.
• Debezium source Kafka connector on confluent Kafka connect docker image version 5.5.12 + kafka-config-provider-aws plugin.
• MSK
• Snowflake sink Kafka connect on confluent Kafka connect docker image version 5.5.12 + kafka-config-provider-aws plugin.
• Snowflake
• We have a dedicated Kafka connect cluster for each source\sink connector in distributed mode on Amazon ECS and Fargate.

We have a secret in Secret manager Service called /testing/cdc_mysql_secrets with the value: {"username":"***","password":"***"}

We are Posting a new connector configuration for Debezium with the following configuration (this is a partial config of course)

    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.user": "${secretManager:/testing/cdc_mysql_secrets:username}",
    "database.password": "${secretManager:/testing/cdc_mysql_secrets:password}",
    "config.providers.secretManager.class": "com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider",
    "config.providers": "secretManager",
    "config.providers.secretManager.param.aws.region": "us-east-1"

The HTTP POST action uses the /connectors/ REST API endpoint and responds with the following ERROR message: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nUnable to connect: Access denied for user '${secretManager:/testing/cdc_mysql_secrets:username}'@'IP' (using password: YES)\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}[ec2-user@ip ~]$ curl -i -X GET -H "Accept:application/json" -H "Content-Type:application/json" ***.elb.amazonaws.com:****/connectors/

When calling the /connector-plugins/{connectorType}/config/validate API endpoint I see the same error in the database.host config object.

NOTE: replacing the username and password with the actual credentials as plain text just works fine. we also have a local environment in which the issue is reproduced and we've placed some debug logs. We can confirm that the method public ConfigData get(String p, Set<String> keys) returns a ConfigData object with a map that looks as follow: {"username":"***","password":"***"}. Also the print in com/github/jcustenborder/kafka/config/aws/SecretsManagerConfigProvider.java:78 shows that the plugin code gets the correct arguments.

We would love to get some help on that matter, Thanks!