Pass partition key when creating items in Cosmos DB

[sivamu]

Pass the partition key information when creating an item in Sink connector. This will need a new Sink config field. Would be useful to document this since it will be important at scale.

[ryancrawcour]

we do this already, don't we? you can't create a document in Cosmos DB without telling it what partition to put it in (i.e. passing the partition key).

is there some other partitioning info that we should be passing?

[ryancrawcour]

well, i'll be .... according to https://github.com/microsoft/kafka-connect-cosmosdb/blob/94a1a3b492a9b99b08550f0d9d52ae6970a37642/src/main/java/com/azure/cosmos/kafka/connect/sink/CosmosDBSinkTask.java#L93-L96 we don't explicitly pass the partition key info. assuming then that Cosmos DB figures this out because it knows what the key for the container is, and can read it out of the document.

is that more or less efficient that us digging in to the document to extract the value? i wonder. be worth testing to see.

[ryancrawcour]

either you know what the PK value* is at runtime, or you don't if you don't, the SDK will go figure it out and extract it from the recordValue if you do know the PK, then it is better to pass that with the recordValue. less work for the SDK to do.

*to pass the PK value you need to know what the PK path is