gpdb
gpdb copied to clipboard
Published
20 hours ago •
greenplum-db
greenplum-db
Use f_smgr interface for storage operations
The purpose of this Pull Request is to bring f_smgr interface back to Greenplum storage.
SMGR interface present in PostgreSQL source code since its first git commit . It's very useful to anybody who wants to re-implement some of storage-related things. This is what first commit of this PR is about.
Second commit is taken from here with little changes for Greenplum compatibilty and is actually moves f_smgr definition in header file and defines some hooks, which would be very useful for extension we develop. We are developing extension that defines custom smgr and moves some AO segment to s3 (deletes them from local storage). When smgrread, smgrwrite or something else wanting to access those (non-local) AO segment we simply bring them back using thing-party s3 tools(walg). This extension is under development and not yet open-sourced, so one could-not check sources to understand better what we want. Also check out gpdb-dev thread about automatic data offload to S3 in conjunction with backups.
What do you think about these changes?
Here are some reminders before you submit the pull request
- [x] Add tests for the change
- [x] Document changes
- [x] Communicate in the mailing list if needed
- [x] Pass
make installcheck - [x] Review a PR in return to support the community
I see the PR targetted against 6X_STABLE, would be good to have the implementation proposed for the master branch and flow down from there. The Master branch implements two SMGR implementations currently one for heap and the other for AO.
Also, curious to learn on the master branch the need for enhancing the WAL record to provide SMGR type is felt refer to https://github.com/greenplum-db/gpdb/pull/13507 won't such a need exist with this change?
Hi Ashwin! many thanks for looking in the pr!
As stated in commit message the first commit brings smgrs code, and it is kept in sync with master branch as much as possible. As you can see here, for 6X_STABLE branch smgr implementation for heap and for AO are the same (same set of functions), but in master they differ (see this). mdunlink_ao is AO-optimised unlink function as i understand, and is a new feature of GP7. Because of this difference we have a problem with wal redo (as far as i understand), and proposed solution for this is custom wal records added here. Well, custom wal records is not very good for us and for smgr-as-extention approach. We cannot write anything in wal, because the only thing we have is pointer to memory where custom smgr is located. in #13507 there is no such a problem because sgmr_which can be only one of 2 things (it is index inside static array in source code).
Well, in this case i will suggest such a solution:
- do not make WAL changes ;)
- change logic here to be like
if (need_smgr_for_ao)
return smgr_for_ao;
else
return smgr_for_heap;
need_smgr_for_ao is a bool flag we can pass to function or something 3) commit similar changes to both 6X_STABLE and for master
there is no problem in this case for mdunlink_ao, because it has compatible signature with basic mdunlink, but we need to do somthing with smgrcreate_ao. We can either add smgrcreate_ao function to f_smgr interface or change smgrcreate_ao signature to be compatible with basic smgrcreate.
so, what do you thing? is that approach ok?
@QingMa-gp can you please look at this pr too? it seems that changes you proposed in #13507 will make smgr-as-extention approch in gp impossible, at least I dont see a way how to handle wal recornds in this case...
So, i have pushed 2 more commits the first is very simple, we just move some definitions into headers, which is very usefull in general and even requested in comment for there functions in source code. Having md/sync routines definitions in header files would be very helpfull for cutsom smgr, and is similar to masster branch code
the third commit adds smgrcreate_ao function to smgr interface. The approach i described in prev message was not successful, it is not possible to know type of relation (heap/ao) in the moment of sgmropen call (at least i dont see how we can compute need_smgr_for_ao flag which i proposed previously). So, 1 more function to interface.
Can someone with access to Concourse CI tell me please what is the failed test? It seems I Don't have access in Concourse CI
Can someone with access to Concourse CI tell me please what is the failed test? It seems I Don't have access in Concourse CI
Hi, @reshke
The icw planner fails:
======================================================================
DIFF FILE: ../gpdb_src/src/test/regress/regression.diffs
----------------------------------------------------------------------
--- \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/expected\/returning_gp\.out 2022-06-09 11:42:30.868291686 +0000
+++ \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/results\/returning_gp\.out 2022-06-09 11:42:30.872291965 +0000
@@ -95,8 +95,12 @@
--
CREATE TEMP TABLE returning_aotab (id int4) WITH (appendonly=true);
INSERT INTO returning_aotab VALUES (1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_24625.1 "base/16384/t_24625.1" for relation "returning_aotab": No such file or directory
DELETE FROM returning_aotab RETURNING *;
-ERROR: DELETE RETURNING is not supported on appendonly tables
+ id
+----
+(0 rows)
+
--
-- Test UPDATE RETURNING with a split update, i.e. an update of the distribution
-- key.
======================================================================
--- \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/expected\/appendonly\.out 2022-06-09 11:46:47.886187613 +0000
+++ \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/results\/appendonly\.out 2022-06-09 11:46:47.930190677 +0000
@@ -902,6 +902,7 @@
(1 row)
insert into temp_tenk_ao5(unique1, unique2) values (99998888, 99998888);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_60946.1 "base/16384/t_60946.1" for relation "temp_tenk_ao5": No such file or directory
update temp_tenk_ao5 set unique2 = 99998889 where unique2 = 99998888;
delete from temp_tenk_ao5 where unique2 = 99998889;
select count(*) from temp_tenk_ao5;
@@ -941,11 +942,11 @@
Distributed by: (unique1)
insert into temp_tenk_ao5(unique1, unique2) values (99998888, 99998888);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_61023.1 "base/16384/t_61023.1" for relation "temp_tenk_ao5": No such file or directory
select unique1 from temp_tenk_ao5;
- unique1
-----------
- 99998888
-(1 row)
+ unique1
+---------
+(0 rows)
-- TEMP TABLES w/ COMMIT DROP AND USING PREPARE
begin;
======================================================================
--- \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/expected\/aocs\.out 2022-06-09 11:48:14.684231283 +0000
+++ \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/results\/aocs\.out 2022-06-09 11:48:14.712233233 +0000
@@ -779,73 +779,55 @@
-- TEMP TABLES w/ INDEXES
create temp table temp_tenk_aocs5 with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
as select * from tenk_aocs5 distributed by (unique1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49359.128 "base/16384/t_49359.128" for relation "temp_tenk_aocs5": No such file or directory
create index temp_even_index on temp_tenk_aocs5 (even);
+ERROR: relation "temp_tenk_aocs5" does not exist
select count(*) from temp_tenk_aocs5;
- count
--------
- 10000
-(1 row)
-
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: select count(*) from temp_tenk_aocs5;
+ ^
select i.tablename, i.indexname, pg_get_indexdef(c.oid, 0, TRUE) from pg_indexes i join pg_class c on i.indexname=c.relname and i.tablename='temp_tenk_aocs5';
- tablename | indexname | pg_get_indexdef
------------------+-----------------+--------------------------------------------------------------------
- temp_tenk_aocs5 | temp_even_index | CREATE INDEX temp_even_index ON temp_tenk_aocs5 USING btree (even)
-(1 row)
+ tablename | indexname | pg_get_indexdef
+-----------+-----------+-----------------
+(0 rows)
insert into temp_tenk_aocs5(unique1, unique2) values (99998888, 99998888);
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: insert into temp_tenk_aocs5(unique1, unique2) values (999988...
+ ^
update temp_tenk_aocs5 set unique2 = 99998889 where unique2 = 99998888;
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: update temp_tenk_aocs5 set unique2 = 99998889 where unique2 ...
+ ^
delete from temp_tenk_aocs5 where unique2 = 99998889;
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: delete from temp_tenk_aocs5 where unique2 = 99998889;
+ ^
select count(*) from temp_tenk_aocs5;
- count
--------
- 10000
-(1 row)
-
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: select count(*) from temp_tenk_aocs5;
+ ^
vacuum analyze temp_tenk_aocs5;
+ERROR: relation "temp_tenk_aocs5" does not exist
truncate table temp_tenk_aocs5;
+ERROR: relation "temp_tenk_aocs5" does not exist
\d temp_tenk_aocs5
-Append-Only Columnar Table "pg_temp_#####_tenk_aocs5"
- Column | Type | Modifiers
--------------+---------+-----------
- even | integer |
- fivethous | integer |
- four | integer |
- hundred | integer |
- odd | integer |
- string4 | name |
- stringu1 | name |
- stringu2 | name |
- ten | integer |
- tenthous | integer |
- thousand | integer |
- twenty | integer |
- two | integer |
- twothousand | integer |
- unique1 | integer |
- unique2 | integer |
-Checksum: t
-Indexes:
- "temp_even_index" btree (even)
-Distributed by: (unique1)
-
insert into temp_tenk_aocs5(unique1, unique2) values (99998888, 99998888);
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: insert into temp_tenk_aocs5(unique1, unique2) values (999988...
+ ^
select unique1 from temp_tenk_aocs5;
- unique1
-----------
- 99998888
-(1 row)
-
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: select unique1 from temp_tenk_aocs5;
+ ^
-- TEMP TABLES w/ COMMIT DROP AND USING PREPARE
begin;
prepare tenk_aocs5_prep(int4) as select * from tenk_aocs5 where unique1 > 8000;
create temp table tenk_aocs5_temp_drop with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
on commit drop as execute tenk_aocs5_prep(8095);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49363.128 "base/16384/t_49363.128" for relation "tenk_aocs5_temp_drop": No such file or directory
select count(*) from tenk_aocs5_temp_drop;
- count
--------
- 1999
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) from tenk_aocs5_temp_drop;
ERROR: relation "tenk_aocs5_temp_drop" does not exist
@@ -855,36 +837,26 @@
begin;
create temp table tenk_aocs5_temp_delete_rows with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
on commit delete rows as select * from tenk_aocs5 where unique1 > 8000 distributed by (unique1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_73943.128 "base/16384/t_73943.128" for relation "tenk_aocs5_temp_delete_rows": No such file or directory
select count(*) from tenk_aocs5_temp_delete_rows;
- count
--------
- 1999
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) from tenk_aocs5_temp_delete_rows;
- count
--------
- 0
-(1 row)
-
+ERROR: relation "tenk_aocs5_temp_delete_rows" does not exist
+LINE 1: select count(*) from tenk_aocs5_temp_delete_rows;
+ ^
-- TEMP TABLES w/ COMMIT PRESERVE ROWS
begin;
create temp table tenk_aocs5_temp_pres_rows with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
on commit preserve rows as select * from tenk_aocs5 where unique1 > 8000 distributed by (unique1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49371.128 "base/16384/t_49371.128" for relation "tenk_aocs5_temp_pres_rows": No such file or directory
select count(*) from tenk_aocs5_temp_pres_rows;
- count
--------
- 1999
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) from tenk_aocs5_temp_pres_rows;
- count
--------
- 1999
-(1 row)
-
+ERROR: relation "tenk_aocs5_temp_pres_rows" does not exist
+LINE 1: select count(*) from tenk_aocs5_temp_pres_rows;
+ ^
-- RULES
insert into tenk_aocs5(unique1, unique2) values (1, 99998889);
create rule ao_rule_update as on insert to tenk_aocs5 do instead update tenk_aocs5 set two=2;
======================================================================
--- \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/expected\/bfv_aggregate\.out 2022-06-09 11:49:11.364177857 +0000
+++ \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/results\/bfv_aggregate\.out 2022-06-09 11:49:11.420181757 +0000
@@ -338,6 +339,7 @@
('foo', '2015-09-1.1', 7),
('foo', '2015-09-1.1', 8),
('foo', '2015-09-1.1', 9);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_79404.1 "base/16384/t_79404.1" for relation "mtup1": No such file or directory
-- The number of SUMs is a chosen so that it it's enough to trigger the bug with NULL
-- attributes, but not much more than that. There's some O(n^2) code in ExecInitAgg
-- to detect duplicate AggRefs, so this starts to get really slow as you add more
@@ -1479,10 +1481,9 @@
SUM(c4 % 5665), SUM(c4 % 5666), SUM(c4 % 5667), SUM(c4 % 5668), SUM(c4 % 5669),
SUM(c4 % 5670), SUM(c4 % 5671)], 1)
from mtup1 where c0 = 'foo' group by c0, c1 limit 10;
- c0 | c1 | array_length
------+-------------+--------------
- foo | 2015-09-1.1 | 5670
-(1 row)
+ c0 | c1 | array_length
+----+----+--------------
+(0 rows)
-- MPP-29042 Multistage aggregation plans should have consistent targetlists in
-- case of same column aliases and grouping on them.
======================================================================
--- \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/expected\/uao_ddl\/temp_on_commit_delete_rows_row\.out 2022-06-09 11:52:52.375566684 +0000
+++ \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/results\/uao_ddl\/temp_on_commit_delete_rows_row\.out 2022-06-09 11:52:52.379566962 +0000
@@ -17,83 +17,72 @@
create temp table temp_delrows(a int, b int)
with (appendonly=true) on commit delete rows distributed by(a);
insert into temp_delrows select i, i from generate_series(1,20)i;
+ERROR: Append-only Storage Write could not open segment file base/16384/t_93501.1 "base/16384/t_93501.1" for relation "temp_delrows": No such file or directory
create index idelrows on temp_delrows using bitmap(a);
+ERROR: current transaction is aborted, commands ignored until end of transaction block
update temp_delrows set b = -a;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Aoseg table should report at least one non-empty segfile.
select distinct nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select distinct nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
-- DML after delete rows action.
begin;
insert into temp_delrows select i, i from generate_series(1,50)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i, i from generate_series(1,...
+ ^
delete from temp_delrows where a > 25;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select count(*) = 25 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Abort should leave no tuples in the table.
abort;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- No begin/end block, each insert is a separate transaction
insert into temp_delrows values (1,1), (2,2), (3,3);
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows values (1,1), (2,2), (3,3);
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
insert into temp_delrows select i,i from generate_series(1,10)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i,i from generate_series(1,1...
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
======================================================================
--- \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/expected\/uao_ddl\/temp_on_commit_delete_rows_column\.out 2022-06-09 11:52:52.487574483 +0000
+++ \/tmp\/build\/d62a0504\/gpdb_src\/src\/test\/regress\/results\/uao_ddl\/temp_on_commit_delete_rows_column\.out 2022-06-09 11:52:52.487574483 +0000
@@ -17,83 +17,72 @@
create temp table temp_delrows(a int, b int)
with (appendonly=true) on commit delete rows distributed by(a);
insert into temp_delrows select i, i from generate_series(1,20)i;
+ERROR: Append-only Storage Write could not open segment file base/16384/t_68916.1 "base/16384/t_68916.1" for relation "temp_delrows": No such file or directory
create index idelrows on temp_delrows using bitmap(a);
+ERROR: current transaction is aborted, commands ignored until end of transaction block
update temp_delrows set b = -a;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Aoseg table should report at least one non-empty segfile.
select distinct nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select distinct nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
-- DML after delete rows action.
begin;
insert into temp_delrows select i, i from generate_series(1,50)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i, i from generate_series(1,...
+ ^
delete from temp_delrows where a > 25;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select count(*) = 25 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Abort should leave no tuples in the table.
abort;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- No begin/end block, each insert is a separate transaction
insert into temp_delrows values (1,1), (2,2), (3,3);
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows values (1,1), (2,2), (3,3);
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
insert into temp_delrows select i,i from generate_series(1,10)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i,i from generate_series(1,1...
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
======================================================================
The orca icw fails:
======================================================================
DIFF FILE: ../gpdb_src/src/test/regress/regression.diffs
----------------------------------------------------------------------
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/returning_gp\.out 2022-06-09 11:45:52.789457117 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/returning_gp\.out 2022-06-09 11:45:52.793457510 +0000
@@ -95,8 +95,12 @@
--
CREATE TEMP TABLE returning_aotab (id int4) WITH (appendonly=true);
INSERT INTO returning_aotab VALUES (1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_24617.1 "base/16384/t_24617.1" for relation "returning_aotab": No such file or directory
DELETE FROM returning_aotab RETURNING *;
-ERROR: DELETE RETURNING is not supported on appendonly tables
+ id
+----
+(0 rows)
+
--
-- Test UPDATE RETURNING with a split update, i.e. an update of the distribution
-- key.
======================================================================
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/appendonly\.out 2022-06-09 11:51:56.149175526 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/appendonly\.out 2022-06-09 11:51:56.193179852 +0000
@@ -902,6 +902,7 @@
(1 row)
insert into temp_tenk_ao5(unique1, unique2) values (99998888, 99998888);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_60867.1 "base/16384/t_60867.1" for relation "temp_tenk_ao5": No such file or directory
update temp_tenk_ao5 set unique2 = 99998889 where unique2 = 99998888;
delete from temp_tenk_ao5 where unique2 = 99998889;
select count(*) from temp_tenk_ao5;
@@ -941,11 +942,11 @@
Distributed by: (unique1)
insert into temp_tenk_ao5(unique1, unique2) values (99998888, 99998888);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_60942.1 "base/16384/t_60942.1" for relation "temp_tenk_ao5": No such file or directory
select unique1 from temp_tenk_ao5;
- unique1
-----------
- 99998888
-(1 row)
+ unique1
+---------
+(0 rows)
-- TEMP TABLES w/ COMMIT DROP AND USING PREPARE
begin;
======================================================================
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/aocs\.out 2022-06-09 11:53:49.868354168 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/aocs\.out 2022-06-09 11:53:49.896356920 +0000
@@ -779,73 +779,55 @@
-- TEMP TABLES w/ INDEXES
create temp table temp_tenk_aocs5 with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
as select * from tenk_aocs5 distributed by (unique1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49361.128 "base/16384/t_49361.128" for relation "temp_tenk_aocs5": No such file or directory
create index temp_even_index on temp_tenk_aocs5 (even);
+ERROR: relation "temp_tenk_aocs5" does not exist
select count(*) from temp_tenk_aocs5;
- count
--------
- 10000
-(1 row)
-
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: select count(*) from temp_tenk_aocs5;
+ ^
select i.tablename, i.indexname, pg_get_indexdef(c.oid, 0, TRUE) from pg_indexes i join pg_class c on i.indexname=c.relname and i.tablename='temp_tenk_aocs5';
- tablename | indexname | pg_get_indexdef
------------------+-----------------+--------------------------------------------------------------------
- temp_tenk_aocs5 | temp_even_index | CREATE INDEX temp_even_index ON temp_tenk_aocs5 USING btree (even)
-(1 row)
+ tablename | indexname | pg_get_indexdef
+-----------+-----------+-----------------
+(0 rows)
insert into temp_tenk_aocs5(unique1, unique2) values (99998888, 99998888);
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: insert into temp_tenk_aocs5(unique1, unique2) values (999988...
+ ^
update temp_tenk_aocs5 set unique2 = 99998889 where unique2 = 99998888;
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: update temp_tenk_aocs5 set unique2 = 99998889 where unique2 ...
+ ^
delete from temp_tenk_aocs5 where unique2 = 99998889;
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: delete from temp_tenk_aocs5 where unique2 = 99998889;
+ ^
select count(*) from temp_tenk_aocs5;
- count
--------
- 10000
-(1 row)
-
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: select count(*) from temp_tenk_aocs5;
+ ^
vacuum analyze temp_tenk_aocs5;
+ERROR: relation "temp_tenk_aocs5" does not exist
truncate table temp_tenk_aocs5;
+ERROR: relation "temp_tenk_aocs5" does not exist
\d temp_tenk_aocs5
-Append-Only Columnar Table "pg_temp_#####_tenk_aocs5"
- Column | Type | Modifiers
--------------+---------+-----------
- even | integer |
- fivethous | integer |
- four | integer |
- hundred | integer |
- odd | integer |
- string4 | name |
- stringu1 | name |
- stringu2 | name |
- ten | integer |
- tenthous | integer |
- thousand | integer |
- twenty | integer |
- two | integer |
- twothousand | integer |
- unique1 | integer |
- unique2 | integer |
-Checksum: t
-Indexes:
- "temp_even_index" btree (even)
-Distributed by: (unique1)
-
insert into temp_tenk_aocs5(unique1, unique2) values (99998888, 99998888);
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: insert into temp_tenk_aocs5(unique1, unique2) values (999988...
+ ^
select unique1 from temp_tenk_aocs5;
- unique1
-----------
- 99998888
-(1 row)
-
+ERROR: relation "temp_tenk_aocs5" does not exist
+LINE 1: select unique1 from temp_tenk_aocs5;
+ ^
-- TEMP TABLES w/ COMMIT DROP AND USING PREPARE
begin;
prepare tenk_aocs5_prep(int4) as select * from tenk_aocs5 where unique1 > 8000;
create temp table tenk_aocs5_temp_drop with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
on commit drop as execute tenk_aocs5_prep(8095);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49361.128 "base/16384/t_49361.128" for relation "tenk_aocs5_temp_drop": No such file or directory
select count(*) from tenk_aocs5_temp_drop;
- count
--------
- 1999
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) from tenk_aocs5_temp_drop;
ERROR: relation "tenk_aocs5_temp_drop" does not exist
@@ -855,36 +837,26 @@
begin;
create temp table tenk_aocs5_temp_delete_rows with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
on commit delete rows as select * from tenk_aocs5 where unique1 > 8000 distributed by (unique1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49365.128 "base/16384/t_49365.128" for relation "tenk_aocs5_temp_delete_rows": No such file or directory
select count(*) from tenk_aocs5_temp_delete_rows;
- count
--------
- 1999
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) from tenk_aocs5_temp_delete_rows;
- count
--------
- 0
-(1 row)
-
+ERROR: relation "tenk_aocs5_temp_delete_rows" does not exist
+LINE 1: select count(*) from tenk_aocs5_temp_delete_rows;
+ ^
-- TEMP TABLES w/ COMMIT PRESERVE ROWS
begin;
create temp table tenk_aocs5_temp_pres_rows with (appendonly=true, orientation=column, compresstype=zlib, compresslevel=1)
on commit preserve rows as select * from tenk_aocs5 where unique1 > 8000 distributed by (unique1);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_49373.128 "base/16384/t_49373.128" for relation "tenk_aocs5_temp_pres_rows": No such file or directory
select count(*) from tenk_aocs5_temp_pres_rows;
- count
--------
- 1999
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) from tenk_aocs5_temp_pres_rows;
- count
--------
- 1999
-(1 row)
-
+ERROR: relation "tenk_aocs5_temp_pres_rows" does not exist
+LINE 1: select count(*) from tenk_aocs5_temp_pres_rows;
+ ^
-- RULES
insert into tenk_aocs5(unique1, unique2) values (1, 99998889);
create rule ao_rule_update as on insert to tenk_aocs5 do instead update tenk_aocs5 set two=2;
======================================================================
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/bfv_aggregate_optimizer\.out 2022-06-09 11:55:31.414336880 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/bfv_aggregate\.out 2022-06-09 11:55:31.470342386 +0000
@@ -338,6 +339,7 @@
('foo', '2015-09-1.1', 7),
('foo', '2015-09-1.1', 8),
('foo', '2015-09-1.1', 9);
+ERROR: Append-only Storage Write could not open segment file base/16384/t_79527.1 "base/16384/t_79527.1" for relation "mtup1": No such file or directory
-- The number of SUMs is a chosen so that it it's enough to trigger the bug with NULL
-- attributes, but not much more than that. There's some O(n^2) code in ExecInitAgg
-- to detect duplicate AggRefs, so this starts to get really slow as you add more
@@ -1479,10 +1481,9 @@
SUM(c4 % 5665), SUM(c4 % 5666), SUM(c4 % 5667), SUM(c4 % 5668), SUM(c4 % 5669),
SUM(c4 % 5670), SUM(c4 % 5671)], 1)
from mtup1 where c0 = 'foo' group by c0, c1 limit 10;
- c0 | c1 | array_length
------+-------------+--------------
- foo | 2015-09-1.1 | 5670
-(1 row)
+ c0 | c1 | array_length
+----+----+--------------
+(0 rows)
-- MPP-29042 Multistage aggregation plans should have consistent targetlists in
-- case of same column aliases and grouping on them.
======================================================================
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/uao_ddl\/temp_on_commit_delete_rows_row\.out 2022-06-09 12:07:18.159843716 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/uao_ddl\/temp_on_commit_delete_rows_row\.out 2022-06-09 12:07:18.159843716 +0000
@@ -17,83 +17,72 @@
create temp table temp_delrows(a int, b int)
with (appendonly=true) on commit delete rows distributed by(a);
insert into temp_delrows select i, i from generate_series(1,20)i;
+ERROR: Append-only Storage Write could not open segment file base/16384/t_68933.1 "base/16384/t_68933.1" for relation "temp_delrows": No such file or directory
create index idelrows on temp_delrows using bitmap(a);
+ERROR: current transaction is aborted, commands ignored until end of transaction block
update temp_delrows set b = -a;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Aoseg table should report at least one non-empty segfile.
select distinct nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select distinct nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
-- DML after delete rows action.
begin;
insert into temp_delrows select i, i from generate_series(1,50)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i, i from generate_series(1,...
+ ^
delete from temp_delrows where a > 25;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select count(*) = 25 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Abort should leave no tuples in the table.
abort;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- No begin/end block, each insert is a separate transaction
insert into temp_delrows values (1,1), (2,2), (3,3);
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows values (1,1), (2,2), (3,3);
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
insert into temp_delrows select i,i from generate_series(1,10)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i,i from generate_series(1,1...
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
======================================================================
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/uao_ddl\/temp_on_commit_delete_rows_column\.out 2022-06-09 12:07:18.271854728 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/uao_ddl\/temp_on_commit_delete_rows_column\.out 2022-06-09 12:07:18.275855122 +0000
@@ -17,83 +17,72 @@
create temp table temp_delrows(a int, b int)
with (appendonly=true) on commit delete rows distributed by(a);
insert into temp_delrows select i, i from generate_series(1,20)i;
+ERROR: Append-only Storage Write could not open segment file base/16384/t_68929.1 "base/16384/t_68929.1" for relation "temp_delrows": No such file or directory
create index idelrows on temp_delrows using bitmap(a);
+ERROR: current transaction is aborted, commands ignored until end of transaction block
update temp_delrows set b = -a;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Aoseg table should report at least one non-empty segfile.
select distinct nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select distinct nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
- 1
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
commit;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
-- DML after delete rows action.
begin;
insert into temp_delrows select i, i from generate_series(1,50)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i, i from generate_series(1,...
+ ^
delete from temp_delrows where a > 25;
+ERROR: current transaction is aborted, commands ignored until end of transaction block
select count(*) = 25 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: current transaction is aborted, commands ignored until end of transaction block
-- Abort should leave no tuples in the table.
abort;
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- No begin/end block, each insert is a separate transaction
insert into temp_delrows values (1,1), (2,2), (3,3);
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows values (1,1), (2,2), (3,3);
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
insert into temp_delrows select i,i from generate_series(1,10)i;
+ERROR: relation "temp_delrows" does not exist
+LINE 1: insert into temp_delrows select i,i from generate_series(1,1...
+ ^
select count(*) = 0 as passed from temp_delrows;
- passed
---------
- t
-(1 row)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select count(*) = 0 as passed from temp_delrows;
+ ^
-- All segfiles must be empty after commit.
select nonempty_segno('temp_delrows'::regclass);
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass);
+ ^
select nonempty_segno('temp_delrows'::regclass)
from gp_dist_random('gp_id');
- nonempty_segno
-----------------
-(0 rows)
-
+ERROR: relation "temp_delrows" does not exist
+LINE 1: select nonempty_segno('temp_delrows'::regclass)
+ ^
======================================================================
@reshke The new version fails one case under orca:
========================
1 of 546 tests failed.
========================
The differences that caused some tests to fail can be viewed in the
file "/tmp/build/b235ddbb/gpdb_src/src/test/regress/regression.diffs". A copy of the test summary that you see
above is saved in the file "/tmp/build/b235ddbb/gpdb_src/src/test/regress/regression.out".
make[2]: *** [installcheck-good] Error 1
make[1]: *** [installcheck-regress-recurse] Error 2
make: *** [installcheck-world-src/test-recurse] Error 2
======================================================================
DIFF FILE: ../gpdb_src/src/test/regress/regression.diffs
----------------------------------------------------------------------
--- \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/expected\/gp_lock\.out 2022-06-09 16:17:05.787244105 +0000
+++ \/tmp\/build\/b235ddbb\/gpdb_src\/src\/test\/regress\/results\/gp_lock\.out 2022-06-09 16:17:05.787244105 +0000
@@ -10,9 +10,8 @@
---------------+--------------+-----------------+---------
0 | gp_lock_test | AccessShareLock | t
1 | gp_lock_test | AccessShareLock | t
- 2 | gp_lock_test | AccessShareLock | t
-1 | gp_lock_test | AccessShareLock | t
-(4 rows)
+(3 rows)
ROLLBACK;
BEGIN;
======================================================================
So, i have pushed 2 more commits the first is very simple, we just move some definitions into headers, which is very usefull in general and even requested in comment for there functions in source code. Having md/sync routines definitions in header files would be very helpfull for cutsom smgr, and is similar to masster branch code
the third commit adds smgrcreate_ao function to smgr interface. The approach i described in prev message was not successful, it is not possible to know type of relation (heap/ao) in the moment of sgmropen call (at least i dont see how we can compute need_smgr_for_ao flag which i proposed previously). So, 1 more function to interface.
Please can you explain the flow how adding the interface helps during WAL replay without knowing the SMGR type? Would really like the working PR on master branch where clearly 2 SMGR implementations exist to see it in action and how to onboard this change. On 6X given the code flows are same for heap and AO hard to envision the working.
So, i have pushed 2 more commits the first is very simple, we just move some definitions into headers, which is very usefull in general and even requested in comment for there functions in source code. Having md/sync routines definitions in header files would be very helpfull for cutsom smgr, and is similar to masster branch code the third commit adds smgrcreate_ao function to smgr interface. The approach i described in prev message was not successful, it is not possible to know type of relation (heap/ao) in the moment of sgmropen call (at least i dont see how we can compute need_smgr_for_ao flag which i proposed previously). So, 1 more function to interface.
Please can you explain the flow how adding the interface helps during WAL replay without knowing the SMGR type? Would really like the working PR on master branch where clearly 2 SMGR implementations exist to see it in action and how to onboard this change. On 6X given the code flows are same for heap and AO hard to envision the working.
Hi, as https://github.com/greenplum-db/gpdb/pull/13682/ is merged now, can we also merge this? i have explained how it would work in master branch, so now we can backport the same to 6X_STABLE. I have rebased patch
@reshke The code changes looks good to me, but we still need SHIPIT meeting to approve it before the merge, since 6X is a production release.
@Aegeaner, hello there, i hore you are doing well ;)
So, is greenplum SHIPIT meeting scheduled weekly or monthly? When we can expect dev team feedback?
@Aegeaner, hello there, i hore you are doing well ;)
So, is greenplum SHIPIT meeting scheduled weekly or monthly? When we can expect dev team feedback?
- It's weekly
- Some points/aspects which needs review for this PR is if any kind of ABI breakage is happening or not from the header file changes. Once that aspect is looked into should be good to go
- Just note given this is enhancement and not a bug fix even if approved will have to wait for next minor version release (not patch release) to be committed