citus icon indicating copy to clipboard operation
citus copied to clipboard

error “server process was terminated by signal 11: Segmentation fault” running pg_create_logical_replication_slot using pgoutput plugin with Citus

Open AbrahimSA opened this issue 5 months ago • 0 comments

I am using docker image with PostgreSQL 15.13 (Ubuntu 15.13-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, 64-bit The Citus extension package is installed, but it is not preload on shared_preload_libraries and citus extesion is not created. I got error “server process was terminated by signal 11: Segmentation fault” using pg_create_logical_replication_slot with pgoutput plugin parameter and using test_decoding worked fine, after recreate the same docker image without Citus library I was able to use logical replication with pgoutput plugin without problem. Is there a way to use a docker image including Citus library and use logical replication with pgoutput plugin at same time? Is it a bug?

See details related of problem

Note: I also updated shm-size from 1024mb to 2g and I am using shared_buffers=1.5GB. This is a test server and there is nothing else running. It is the first time I am working with logical replication.

postgresql.conf file: wal_level = logical max_replication_slots = 10 max_wal_senders = 20 listen_addresses = '*'

psql -U postgres -h postgres -c "SELECT pg_create_logical_replication_slot('support7561_repslot', 'pgoutput');" SSL SYSCALL error: EOF detected connection to server was lost

pg_log < 2025-07-08 14:57:08.653 UTC psql postgres postgres 172.18.0.94(53414) SELECT 00000 2025-07-08 14:57:07 UTC 1096 686d31c3.448 2025-07-08 14:57:08.653 UTC LOG: Initializing CDC decoder < 2025-07-08 14:57:08.653 UTC psql postgres postgres 172.18.0.94(53414) SELECT 00000 2025-07-08 14:57:07 UTC 1096 686d31c3.448 2025-07-08 14:57:08.653 UTC STATEMENT: SELECT pg_create_logical_replication_slot('support7561_repslot', 'pgoutput'); < 2025-07-08 14:57:08.821 UTC 00000 2025-07-08 14:55:38 UTC 923 686d316a.39b 2025-07-08 14:57:08.821 UTC LOG: server process (PID 1096) was terminated by signal 11: Segmentation fault < 2025-07-08 14:57:08.821 UTC 00000 2025-07-08 14:55:38 UTC 923 686d316a.39b 2025-07-08 14:57:08.821 UTC DETAIL: Failed process was running: SELECT pg_create_logical_replication_slot('support7561_repslot', 'pgoutput'); < 2025-07-08 14:57:08.821 UTC 00000 2025-07-08 14:55:38 UTC 923 686d316a.39b 2025-07-08 14:57:08.821 UTC LOG: terminating any other active server processes < 2025-07-08 14:57:08.829 UTC 00000 2025-07-08 14:55:38 UTC 923 686d316a.39b 2025-07-08 14:57:08.829 UTC LOG: all server processes terminated; reinitializing < 2025-07-08 14:57:09.215 UTC 00000 2025-07-08 14:57:09 UTC 1098 686d31c5.44a 2025-07-08 14:57:09.215 UTC LOG: database system was interrupted; last known up at 2025-07-08 14:55:39 UTC < 2025-07-08 14:57:10.037 UTC [unknown] postgres postgres 172.18.0.217(33506) 57P03 2025-07-08 14:57:10 UTC 1101 686d31c6.44d 2025-07-08 14:57:10.037 UTC FATAL: the database system is in recovery mode < 2025-07-08 14:57:10.437 UTC 00000 2025-07-08 14:57:09 UTC 1098 686d31c5.44a 2025-07-08 14:57:10.437 UTC LOG: database system was not properly shut down; automatic recovery in progress < 2025-07-08 14:57:10.450 UTC 00000 2025-07-08 14:57:09 UTC 1098 686d31c5.44a 2025-07-08 14:57:10.450 UTC LOG: redo starts at 1FB9/C0000A0 < 2025-07-08 14:57:10.456 UTC 00000 2025-07-08 14:57:09 UTC 1098 686d31c5.44a 2025-07-08 14:57:10.456 UTC LOG: invalid record length at 1FB9/C054DF8: wanted 24, got 0 < 2025-07-08 14:57:10.456 UTC 00000 2025-07-08 14:57:09 UTC 1098 686d31c5.44a 2025-07-08 14:57:10.456 UTC LOG: redo done at 1FB9/C054DC0 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s < 2025-07-08 14:57:10.475 UTC 00000 2025-07-08 14:57:09 UTC 1099 686d31c5.44b 2025-07-08 14:57:10.475 UTC LOG: checkpoint starting: end-of-recovery immediate wait < 2025-07-08 14:57:10.501 UTC 00000 2025-07-08 14:57:09 UTC 1099 686d31c5.44b 2025-07-08 14:57:10.501 UTC LOG: checkpoint complete: wrote 86 buffers (0.0%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.010 s, sync=0.007 s, total=0.028 s; sync files=18, longest=0.003 s, average=0.001 s; distance=339 kB, estimate=339 kB < 2025-07-08 14:57:10.510 UTC 00000 2025-07-08 14:55:38 UTC 923 686d316a.39b 2025-07-08 14:57:10.510 UTC LOG: database system is ready to accept connections

psql -U postgres -h postgres -c "SELECT pg_create_logical_replication_slot('support7561_repslot', 'test_decoding');" pg_create_logical_replication_slot

(support7561_repslot,1FB9/C081668)

postgres(at)support7560_postgres:/var/lib/postgresql/15/main$ psql -U postgres -h postgres -c "SELECT slot_name, plugin, slot_type, database, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;" slot_name | plugin | slot_type | database | active | restart_lsn | confirmed_flush_lsn ---------------------+---------------+-----------+----------+--------+--------------+--------------------- support7561_repslot | test_decoding | logical | postgres | f | 1FB9/C081630 | 1FB9/C081668

SHOW shared_buffers; shared_buffers

1532512kB (1 row)

uname -a Linux support7560_postgres 6.8.0-1030-gcp #32~22.04.1-Ubuntu SMP Tue Apr 29 23:17:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

\dx List of installed extensions Name | Version | Schema | Description --------------------+---------+------------+------------------------------------------------------------------------ btree_gist | 1.7 | public | support for indexing common datatypes in GiST ltree | 1.2 | public | data type for hierarchical tree-like structures pg_stat_statements | 1.10 | public | track planning and execution statistics of all SQL statements executed pg_trgm | 1.6 | public | text similarity measurement and index searching based on trigrams pgcrypto | 1.3 | public | cryptographic functions plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language postgis | 3.5.3 | public | PostGIS geometry and geography spatial types and functions uuid-ossp | 1.1 | public | generate universally unique identifiers (UUIDs) (8 rows)

select * from pg_available_extensions where name like '%citus%'; name | default_version | installed_version | comment ----------------+-----------------+-------------------+---------------------------- citus_columnar | 11.3-1 | | Citus Columnar extension citus | 12.1-1 | | Citus distributed database

show shared_preload_libraries; shared_preload_libraries

pg_stat_statements, pg_repack

----------------------------------------------------------------- Part of docker image related of Citus

BUILD CITUS EXTENSION

FROM baseimage AS citus-builder

COMPILE CITUS

RUN
apt-get update && apt-get upgrade -y &&
apt-get install -y --no-install-recommends
postgresql-server-dev-"$POSTGRES_VERSION"
autoconf flex
git libcurl4-gnutls-dev
libicu-dev libkrb5-dev
liblz4-dev libpam0g-dev
libreadline-dev libselinux1-dev
libssl-dev libxslt1-dev
libzstd-dev make
tzdata-legacy
uuid-dev gcc &&
rm -rf /var/lib/apt/lists/* &&
apt-get clean

RUN
[ "${POSTGRES_VERSION}" -gt 15 ] && CITUS_BRANCH="v${CITUS_VERSION}" || CITUS_BRANCH="release-12.1";
git config --global advice.detachedHead false &&
git clone --quiet --depth 1 --single-branch --branch "${CITUS_BRANCH}"
https://github.com/citusdata/citus.git WORKDIR /citus ENV PG_CONFIG="/usr/lib/postgresql/${POSTGRES_VERSION}/bin/pg_config" RUN ./configure && make clean && make -j "$(nproc)" && make install

END BUILD CITUS EXTENSION

FROM baseimage AS final-build ...

MOVED here so both builds run in parallel

COPY --from=citus-builder
/usr/lib/postgresql/$POSTGRES_VERSION/lib/citus* /usr/lib/postgresql/$POSTGRES_VERSION/lib/ COPY --from=citus-builder
/usr/share/postgresql/$POSTGRES_VERSION/extension/citus* /usr/share/postgresql/$POSTGRES_VERSION/extension/

More details on this link

AbrahimSA avatar Jul 15 '25 14:07 AbrahimSA