Nice to have worker task:topic mapping in Sink connector

[jcocchi]

Currently the sink connector spins up as many workers as the maxTasks configuration and assigns all topics to all workers. Because of the topic -> collection mapping it would be more efficient to assign specific works to a subset of topics instead of all having all workers read from all topics.

Investigate the implications of this and implement the best solution.

This refers to the taskConfigs function of the CosmosDBSinkConnector

[ryancrawcour]

does it work as is? i.e. is this a feature we need to have, or a nice to have enhancement that could improve efficiency?

is it "should have" or "nice to have"?

[jcocchi]

@ryancrawcour Yep, it does work as is and is a "nice to have."

Currently all workers listen to all topics, but because of the way we are writing records looping through messages for each topic then writing them it would be more efficient to have certain workers dedicated to certain topics so there are fewer loops per processing chunk.

[ryancrawcour]

Makes sense. Thanks. Will mark it as a future enhancement.

[ryancrawcour]

Out of interest, how does the Mongo connector do this?

[brandynbrown]

Follow up: do a spike to see if it's feasible to configure a worker to a specific topic

If so, Are there performance improvements?

[brandynbrown]

Spike Investigate what Cassandra connector does when monitoring multiple topics.

[brandynbrown]

blocked by #292