gpdb
gpdb copied to clipboard
greenplum-db
Confusing logical_replication_slot behavior
Greenplum version or build
gpdb master latest
Step to reproduce the behavior
gpconfig -c wal_level -v logical
gpstop -r
cd contrib/test_decoding; make install
postgres=# SELECT * FROM pg_create_logical_replication_slot('my_slot', 'test_decoding');
slot_name | xlog_position
-----------+---------------
my_slot | 0/C3C6D58
(1 row)
postgres=# create table test(a int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
postgres=# SELECT * FROM pg_logical_slot_get_changes('my_slot', NULL, NULL);
ERROR: unexpected RM_XLOG_ID record type: 176 (decode.c:199)
on master i get this error, seems reasonable to get the error message "replication slot "my_slot" does not exist"
bhuvnesh=# create table test(a int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
Time: 67.558 ms
bhuvnesh=# SELECT * FROM pg_logical_slot_get_changes('my_slot', NULL, NULL);
ERROR: replication slot "my_slot" does not exist
Time: 57.045 ms
bhuvnesh=#
I think original issue is related to commit 1fd11387d2b9f250c14f0ccb893c0956b1bf1487 and not updating accordingly for logical decoding. That commit defines XLOG_NEXTRELFILENODE as 0xB0 which corresponds to 176 in error message. I think this will continue to be an issue until we add GPDB specific types to DecodeXLogOp.
I think this is not an issue as long as we're fine with only physical replication slots (no logical replication slots). So maybe we can push this off until 7X.
@bhuvnesh2703 ,sorry I left an sql command that need to reproduce, SELECT * FROM pg_create_logical_replication_slot('my_slot', 'test_decoding');, I have updated in desciption in github issue.
@dgkimura , thanks for your analysis, still seems not fully reasonable to me, because even update cause similar error
postgres=# create table aa(a int, b int);
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
postgres=# insert into aa values (1, 2);
INSERT 0 1
postgres=# SELECT * FROM pg_create_logical_replication_slot('sbai_slot', 'test_decoding');
slot_name | xlog_position
-----------+---------------
sbai_slot | 0/C471738
(1 row)
postgres=# update aa set a=3 where a=1;
UPDATE 1
postgres=# SELECT * FROM pg_logical_slot_get_changes('sbai_slot', NULL, NULL);
ERROR: unexpected RM_XACT_ID record type: 112 (decode.c:337)
We have not supported logical decoding/replication yet in GPDB, hence failures from trying to get changes from logical slot to seem reasonable. I think as an interim step we should disable pg_create_logical_replication_slot() to avoid such problems from stemming further.
Enabling logical replication/decoding has its own set of challenges in GPDB, at high-level
- How to combine transactions from different segments into a single source (mostly will be using distributed transaction ID to connect back WAl across segments to form and set)
- Need to make sure all the segments data are included in the combined set (in some cases we only have WAL on some segments (direct dispatch case) vs many transactions will have WAL from all the segments)
- For decoding AO/CO records:
- we will have to construct back full row from different CO records
- Given we write blocks to WAL, we will have to decompress, decode (for RLE, DELTA) to get the full rows
- plus need to find a strategy for update/deletes for AO/CO tables as we mostly only update visimap table (not the actual base table) Hence, enabling logical decoding is a very long-term feature and 100% outside the scope of this issue.
The concept of 'logical replication' has a bit confusion. Postgres has a logical replication feature depends on logical decoding framework but in general 'logical replication' means the another replication way compared with physical (WAL) way.
The logical replication feature of Postgres is not suitable for Greenplum. However, Greenplum can use logical decoding framework to decode the WAL to the logical format. it may or may not depending on replication slot. I tend to +1 for " disable pg_create_logical_replication_slot()" as well.
How to combine transactions from different segments into a single source (mostly will be using distributed transaction ID to connect back WAl across segments to form and set)
An option is to streaming such message to kafka and then replay through gpss which could also support update/delete besides insert. DDL and truncate are not supported.
Need to make sure all the segments data are included in the combined set (in some cases we only have WAL on some segments (direct dispatch case) vs many transactions will have WAL from all the segments)
gp_create_restore_point will create consistent point in WAL and we can use this in its logical format as well. it is enough as CDC source but still need to improve how to replay efficient (in parallel) on target.
Plan is we would spend sometime to make the feature work on GP7
