Failure during Citus 11.0 to Citus 11.1 upgrade

[mozansaka]

Using the citus_14-11.1.0.citus-0.0.git.20220810.1662729.el7.x86_64.rpm nightly package, tried Citus 11.0 to Citus 11.1 upgrade scenario in Marlin.

Error happens during alter extension step at https://msdata.visualstudio.com/Database%20Systems/_git/Marlin?path=/instance/lib/citus_upgrade.rb&version=GBmaster&line=116&lineEnd=117&lineStartColumn=1&lineEndColumn=1&lineStyle=plain&_a=contents where we run following (shellescaped) query

CREATE\ EXTENSION\ IF\ NOT\ EXISTS\ citus;ALTER\ EXTENSION\ citus\ UPDATE;UPDATE\ pg_dist_node_metadata\ SET\ metadata=jsonb_set(metadata,\ '{}',\ '{"cloud":\ true}')

Got the error message attached error_message.txt

[metdos]

Copy pasting the error message:

ERROR: function columnar_handler(internal) is already a member of extension "citus" CONTEXT: SQL statement " --#include "udfs/columnar_handler/10.0-1.sql" CREATE OR REPLACE FUNCTION columnar.columnar_handler(internal) RETURNS table_am_handler LANGUAGE C AS '$libdir/citus_columnar', 'columnar_handler'; COMMENT ON FUNCTION columnar.columnar_handler(internal) IS 'internal function returning the handler for columnar tables'; -- postgres 11.8 does not support the syntax for table am, also it is seemingly trying -- to parse the upgrade file and erroring on unknown syntax. -- normally this section would not execute on postgres 11 anyway. To trick it to pass on -- 11.8 we wrap the statement in a plpgsql block together with an EXECUTE. This is valid -- syntax on 11.8 and will execute correctly in 12 DO $create_table_am$ BEGIN EXECUTE 'CREATE ACCESS METHOD columnar TYPE TABLE HANDLER columnar.columnar_handler'; END $create_table_am$; --#include "udfs/alter_columnar_table_set/10.0-1.sql" CREATE OR REPLACE FUNCTION pg_catalog.alter_columnar_table_set( table_name regclass, chunk_group_row_limit int DEFAULT NULL, stripe_row_limit int DEFAULT NULL, compression name DEFAULT null, compression_level int DEFAULT NULL) RETURNS void LANGUAGE C AS '$libdir/citus_columnar', 'alter_columnar_table_set'; COMMENT ON FUNCTION pg_catalog.alter_columnar_table_set( table_name regclass, chunk_group_row_limit int, stripe_row_limit int, compression name, compression_level int) IS 'set one or more options on a columnar table, when set to NULL no change is made'; --#include "udfs/alter_columnar_table_reset/10.0-1.sql" CREATE OR REPLACE FUNCTION pg_catalog.alter_columnar_table_reset( table_name regclass, chunk_group_row_limit bool DEFAULT false, stripe_row_limit bool DEFAULT false, compression bool DEFAULT false, compression_level bool DEFAULT false) RETURNS void LANGUAGE C AS '$libdir/citus_columnar', 'alter_columnar_table_reset'; COMMENT ON FUNCTION pg_catalog.alter_columnar_table_reset( table_name regclass, chunk_group_row_limit bool, stripe_row_limit bool, compression bool, compression_level bool) IS 'reset on or more options on a columnar table to the system defaults'; " PL/pgSQL function inline_code_block line 8 at EXECUTE /home/mapi/lib/util.rb:5:in r': unhandled exception from /home/mapi/lib/citus_upgrade.rb:119:in alter_extension' from bin/citus-upgrade:39:in block in <main>' from /home/mapi/lib/util.rb:109:in block in exclusive' from /home/mapi/lib/util.rb:106:in open' from /home/mapi/lib/util.rb:106:in exclusive' from bin/citus-upgrade:21:in `

'

[metdos]

@jeff-davis any idea about the problem?

[mozansaka]

To give an update on this, I tried Citus 11.0 => Citus 11.1 again now with the latest nightly build citus111_14-11.1.0.citus-0.0.git.20220812.e962113.el7.x86_64

Error message is:

ERROR: schema columnar is not a member of extension "citus_columnar" DETAIL: An extension may only use CREATE ... IF NOT EXISTS to skip object creation if the conflicting object is one that it already owns. /home/mapi/lib/util.rb:5:in r': unhandled exception from /home/mapi/lib/citus_upgrade.rb:116:in alter_extension' from bin/citus-upgrade:39:in block in <main>' from /home/mapi/lib/util.rb:109:in block in exclusive' from /home/mapi/lib/util.rb:106:in open' from /home/mapi/lib/util.rb:106:in exclusive' from bin/citus-upgrade:21:in `

'

[yxu2162]

Is there a way to run the nightly build on local? I've tried upgrading Citus 11.0 => Citus 11.1 manually and haven't been able to repro the issue. Looking at the error that @metdos provided, I'm wondering if there's some possible race condition or weird upgrade path where citus still owns the columnar objects while citus_columnar is trying to create them. Which is confusing because my understanding is that citus_columnar only creates them when the user has ran create extension citus_columnar, otherwise, we should be dropping those objects in citus and then during the upgrade from citus_columnar version 11.1-0 to 11.1-1, we re-attach them to citus_columnar...