cockroach
cockroach copied to clipboard
streamingccl: add new logical package
This adds a new LogicalReplication job and LogicalReplicationWriter processor. This is a stub implementation that will be changed substantially in the coming weeks.
The idea of the LogicalReplication job is that it can be used to ingest data from one cluster into another while both clusters are still accepting writes.
When two LogicalReplication jobs are started operating in opposite directions on a pair of servers, the end effect will be an active-active asynchronous replication process.
The overall architecture is similar to physical cluster replication.
Namely, for one flow, the destination cluster asks the source cluster to establish a protected timestamp over the requested rows and to return an ideal ingestion plan to the source.
The source then starts a set of workers each of which is responsible for streaming a subset of the source cluster's keyspace. Rows are returned over a pgwire connection.
The implementation here is contains almost none of the pieces required to make this production ready.
It is missing, among many other things:
-
Robust handling of schema descriptors to ensure that the user cannot make incompatible schema changes during replication.
-
Correct handling of table's related by FK constraints.
-
Easy-to-use SQL syntax.
-
Correct last-write-wins semantics for DELETEs.
-
Last-write-wins semantics without changes to the user's schema.
The goal of this PR is simply to get the job and processor in place so that we can start filling out the production implementation.
Epic: none Release note: None
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.
:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.
bors r=msbutler
Build succeeded:
blathers backport 24.1
Encountered an error creating backports. Some common things that can go wrong:
- The backport branch might have already existed.
- There was a merge conflict.
- The backport branch contained merge commits.
You might need to create your backport manually using the backport tool.
error creating merge commit from df6175d0b82e81892fb473fb70614e0026b7bab7 to blathers/backport-release-24.1-124606: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []
you may need to manually resolve merge conflicts with the backport tool.
Backport to branch 24.1 failed. See errors above.
:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.
Ignore my derping 😂 ... wrong window.