Connector needs to have config validators

[ryancrawcour]

## Problem Statement When entering configuration for the connector it would be great if these config values could be validated whilst registering the connector instead of @ runtime when the connector tries to start.

## Proposed Solution Kafka Connect supports connector config validation. Implement this for the sink connector config properties. Ensure common, Sink and Source config values all have validators and recommendors as appropriate.

Validations to add -

• [x] validate database name is valid
• [x] validation container name is valid
• [ ] validate that the topic(s) are valid

## Additional context https://www.confluent.io/blog/write-a-kafka-connect-connector-with-configuration-handling/

Next Steps

• [x] Team consensus to proceed
• [ ] Schedule Design Session
• [ ] Complete Design Review

[ncliang]

@ryancrawcour I see the following in CosmosClientBuilder::key javadoc

Sets either a master or readonly key used to perform authentication for accessing resource. Params: key – master or readonly key Returns: current Builder.

This implies that a readonly key could also be used to create the client. Is there a good way to validate that the client is created with a master key containing required ALL permission to create/write to database/container?

[ryancrawcour]

This implies that a readonly key could also be used to create the client. Is there a good way to validate that the client is created with a master key containing required ALL permission to create/write to database/container?

that's a good point. i forget about the read-only key. not sure the best way to validate that the key being used is a master key, not a read-only key.

@kushagraThapar / @moderakh can you think of a way to validate the key being used has write permission to the container, without actually attempting to write a doc (and then having to delete it).

[mageshn]

@ryancrawcour I don't have a lot of knowledge on this but would the permissions API help here? https://docs.microsoft.com/en-us/rest/api/cosmos-db/permissions . This can work only if we can invoke without any additional credentials or permissions.

[moderakh]

@mageshn no permission API purpose is different.

At client side we currently do not have any API to distinguish a write key from a read key without actually attempting to perform a write operation.

I will check with our service side team if there is any workaround.

[ryancrawcour]

i have the following listed as validators to add -

validate database name is valid

• I am thinking of doing a GetDatabase call to check if the db id supplied is valid.

validation container name is valid

• as above, plan to do a GetContainer (given a valid database) to check is the container id is valid. might have to do multiple here as we support a Topic#Container mapping where it would be possible to supply multiple container names.

@moderakh / @kushagraThapar thoughts on the above 2 operations?

validate that the topic(s) are valid

• @ncliang what would be the best way to check if the Kafka topic supplied is valid?

any others?

[ryancrawcour]

think of a way to validate the key being used has write permission to the container, without actually attempting to write a doc (and then having to delete it).

doesn't look like there is a way to determine write permissions given a key without actually attempting to write a doc. so we'll just validate that the key given is a valid key, and handle a write failure if that happens.