cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

crosscluster/logical: consolidate batching decisions in flushChunk

Open dt opened this issue 1 year ago • 2 comments

Previously we would choose to batch rows once in flushBuffer/flushChunk and then again in HandleBatch. This change moves the batching decision making only into the flushBuffer/flushChunk functions, with HandleBatch returning to be solely a single batch application. To preserve the implicit txn optimization that had been why HandleBatch would re-batch differently, HandleBatch is now free to use an implicit txn when the batch is a single KV, and the flushChunk func will pass it a single KV when the optimization is enabled.

This lays the groundwork for flushChunk being extended to do bookeeping for which updates fail to apply and which apply.

Release note: none. Epic: none.

dt avatar Jun 30 '24 15:06 dt

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] avatar Jun 30 '24 15:06 blathers-crl[bot]

This change is Reviewable

cockroach-teamcity avatar Jun 30 '24 15:06 cockroach-teamcity

TFTR!

bors r+

dt avatar Jul 01 '24 17:07 dt

intentional that minChunkSize is increased from 32

Yeah, it doesn't really need to be anything in particular just don't want to fire up 32 threads if we don't have enough rows to justify them, so I want each thread to have some amount of work to do independent of the batch size at which it will do it.

dt avatar Jul 01 '24 17:07 dt