cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

streamingccl: add new logical package

Open stevendanna opened this issue 1 year ago • 2 comments

This adds a new LogicalReplication job and LogicalReplicationWriter processor. This is a stub implementation that will be changed substantially in the coming weeks.

The idea of the LogicalReplication job is that it can be used to ingest data from one cluster into another while both clusters are still accepting writes.

When two LogicalReplication jobs are started operating in opposite directions on a pair of servers, the end effect will be an active-active asynchronous replication process.

The overall architecture is similar to physical cluster replication.

Namely, for one flow, the destination cluster asks the source cluster to establish a protected timestamp over the requested rows and to return an ideal ingestion plan to the source.

The source then starts a set of workers each of which is responsible for streaming a subset of the source cluster's keyspace. Rows are returned over a pgwire connection.

The implementation here is contains almost none of the pieces required to make this production ready.

It is missing, among many other things:

  1. Robust handling of schema descriptors to ensure that the user cannot make incompatible schema changes during replication.

  2. Correct handling of table's related by FK constraints.

  3. Easy-to-use SQL syntax.

  4. Correct last-write-wins semantics for DELETEs.

  5. Last-write-wins semantics without changes to the user's schema.

The goal of this PR is simply to get the job and processor in place so that we can start filling out the production implementation.

Epic: none Release note: None

stevendanna avatar May 23 '24 12:05 stevendanna

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] avatar May 23 '24 12:05 blathers-crl[bot]

This change is Reviewable

cockroach-teamcity avatar May 23 '24 12:05 cockroach-teamcity

bors r=msbutler

stevendanna avatar May 30 '24 11:05 stevendanna

blathers backport 24.1

nicktrav avatar Jun 12 '24 20:06 nicktrav

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from df6175d0b82e81892fb473fb70614e0026b7bab7 to blathers/backport-release-24.1-124606: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 24.1 failed. See errors above.


:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] avatar Jun 12 '24 20:06 blathers-crl[bot]

Ignore my derping 😂 ... wrong window.

nicktrav avatar Jun 12 '24 20:06 nicktrav