cloudberry
cloudberry copied to clipboard
apache
[Bug] Query Causes Segmentation Fault
Apache Cloudberry version
Cloudberry 1.6.0 (this is a pre-Apache release)
What happened
We had a number of segments go down both on 12/16 (multiple waves of segments going down) as well as 12/18 due to a problem query that was run. The log messages show this query along with a segmentation fault error in each segment log that crashed. I've obfuscated the table and column names for the sake of privacy in the below error message as well as the provided DDL and problem query, but everything else is exactly the same as it was when it was executed
Segmentation fault","Failed process was running: SELECT col1
, col17
, LAG(col14) OVER (PARTITION BY col1, col17 ORDER BY col1, col17, col14)
, col14
, col14 - LAG(col14) OVER (PARTITION BY col1, col17 ORDER BY col1, col17, col14) AS t
, col14 - LAG(col14) OVER (PARTITION BY col1, col17 ORDER BY col1, col17, col14) < '5 mins' AS v
FROM myschema.mytable
WHERE col14 >= '2024-11-01'
AND col14 < '2024-12-01'
AND col17 = 'FOOBAR’;",,,,,,0,,"postmaster.c",4315,
2024-12-16 22:08:13.008648 UTC,,,p3534838,th-2126882688,,,,0,,,seg171,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",4023,
2024-12-16 22:08:13.136640 UTC,,,p3534856,th-2126882688,,,,0,,,seg171,,,,,"WARNING","01000","ic-proxy-server: received signal 3",,,,,,,0,,"ic_proxy_main.c",474,
2024-12-16 22:08:13.182293 UTC,,,p3534838,th-2126882688,,,,0,,,seg171,,,,,"LOG","00000","background worker ""ic proxy process"" (PID 3534856) exited with exit code 1",,,,,,,0,,"postmaster.c",4293,
As far as I can tell it doesn't seem like this query was ever successful, however it's hard to tell from the logs since they're pretty verbose. However I did confirm with an engineer that this query was first executed on 12/16 (the day of our segments going down the first time with this query), there's no entry of it being executed at all on 12/17, and then it was executed again on 12/18 causing the same error and crash. This is a very large table with a lot of partitions and just from taking a quick sampling each partition is around 100GB+ in size, however I can get exact sizes if that's helpful for analysis.
I've attached the core dump analysis files for all 31 segments that went down during this time (this is directly related to issue https://github.com/apache/cloudberry/issues/803) provided by @edespino in his comment here https://github.com/apache/cloudberry/issues/803#issuecomment-2555808199. I've also provided the DDL to create the table that I captured using gpbackup and then obfuscated as well as the query itself in files that are attached. Happy to provide any more detail upon request. I've tested this in my local Cloudberry docker environment but haven't been able to recreate the segfault. Mind you though that this test didn't include any data in the table so I imagine that would influence what steps the query execution plan actually takes.
What you think should happen instead
This query shouldn't cause a segmentation fault and should just return the results when it's finished
How to reproduce
I've tested this locally and haven't been able to reproduce the segfault
Operating System
Rocky Linux 8.10 (Green Obsidian)
Anything else
No response
Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct.
CloudBerry Database Interconnect Crash Analysis
Overview
A widespread interconnect crash occurred across a CloudBerry Database 1.6.0 cluster affecting multiple segments. This analysis examines the crash patterns across the affected segments and hosts.
Would it be possible to share a couple of representative core files from this crash? The analysis shows a consistent pattern:
- All crashes are SIGABRT in the interconnect layer (rxThreadFunc)
- The crash propagated across 19 hosts in about 5.2 seconds
- 47 unique segments were affected, with some segments crashing multiple times
Having access to two core files would help with:
- Identifying the specific assertion that's failing
- Understanding the state of the interconnect at crash time
- Examining relevant memory structures
Core Analysis Script Output and Source
Analysis Output
Starting core file analysis...
Core File Analysis Summary
==================================================
Hosts Affected: 19
10.9.1.115: 1 core files
19:22:29 - seg19 (SIGABRT)
10.9.1.14: 5 core files
19:22:28 - seg178 (SIGABRT)
19:22:29 - seg177 (SIGABRT)
19:22:30 - seg183 (SIGABRT)
19:22:31 - seg181 (SIGABRT)
19:22:32 - seg180 (SIGABRT)
10.9.1.140: 3 core files
19:22:29 - seg70 (SIGABRT)
19:22:31 - seg69 (SIGABRT)
19:22:33 - seg71 (SIGABRT)
10.9.1.161: 2 core files
19:22:28 - seg138 (SIGABRT)
19:22:29 - seg143 (SIGABRT)
10.9.1.166: 5 core files
19:22:28 - seg171 (SIGABRT)
19:22:30 - seg174 (SIGABRT)
19:22:31 - seg175 (SIGABRT)
19:22:32 - seg173 (SIGABRT)
19:22:33 - seg171 (SIGABRT)
10.9.1.168: 4 core files
19:22:28 - seg143 (SIGABRT)
19:22:29 - seg147 (SIGABRT)
19:22:30 - seg146 (SIGABRT)
19:22:31 - seg144 (SIGABRT)
10.9.1.175: 4 core files
19:22:29 - seg131 (SIGABRT)
19:22:30 - seg129 (SIGABRT)
19:22:31 - seg128 (SIGABRT)
19:22:32 - seg130 (SIGABRT)
10.9.1.176: 4 core files
19:22:29 - seg44 (SIGABRT)
19:22:30 - seg47 (SIGABRT)
19:22:31 - seg45 (SIGABRT)
19:22:32 - seg46 (SIGABRT)
10.9.1.185: 3 core files
19:22:28 - seg175 (SIGABRT)
19:22:29 - seg179 (SIGABRT)
19:22:31 - seg177 (SIGABRT)
10.9.1.210: 2 core files
19:22:28 - seg45 (SIGABRT)
19:22:30 - seg51 (SIGABRT)
10.9.1.222: 5 core files
19:22:29 - seg63 (SIGABRT)
19:22:30 - seg62 (SIGABRT)
19:22:31 - seg57 (SIGABRT)
19:22:32 - seg61 (SIGABRT)
19:22:33 - seg60 (SIGABRT)
10.9.1.232: 5 core files
19:22:29 - seg51 (SIGABRT)
19:22:30 - seg54 (SIGABRT)
19:22:31 - seg55 (SIGABRT)
19:22:32 - seg53 (SIGABRT)
19:22:33 - seg51 (SIGABRT)
10.9.1.238: 3 core files
19:22:28 - seg106 (SIGABRT)
19:22:29 - seg107 (SIGABRT)
19:22:30 - seg105 (SIGABRT)
10.9.1.240: 2 core files
19:22:29 - seg180 (SIGABRT)
19:22:30 - seg187 (SIGABRT)
10.9.1.242: 2 core files
19:22:28 - seg15 (SIGABRT)
19:22:30 - seg12 (SIGABRT)
10.9.1.246: 1 core files
19:22:30 - seg108 (SIGABRT)
10.9.1.45: 2 core files
19:22:29 - seg54 (SIGABRT)
19:22:30 - seg57 (SIGABRT)
10.9.1.61: 1 core files
19:22:29 - seg138 (SIGABRT)
10.9.1.83: 4 core files
19:22:29 - seg163 (SIGABRT)
19:22:30 - seg162 (SIGABRT)
19:22:31 - seg161 (SIGABRT)
19:22:32 - seg160 (SIGABRT)
Stack Trace Patterns
------------------------------
Pattern occurred 58 times:
1. raise
2. poll
3. rxThreadFunc
4. start_thread
5. clone
Signal Distribution
------------------------------
SIGABRT: 58
Timing Analysis
------------------------------
First core: 19:22:28.073766
Last core: 19:22:33.310271
Duration: 5.237 seconds
Segment Distribution
------------------------------
seg105: 1 cores
seg106: 1 cores
seg107: 1 cores
seg108: 1 cores
seg12: 1 cores
seg128: 1 cores
seg129: 1 cores
seg130: 1 cores
seg131: 1 cores
seg138: 2 cores
seg143: 2 cores
seg144: 1 cores
seg146: 1 cores
seg147: 1 cores
seg15: 1 cores
seg160: 1 cores
seg161: 1 cores
seg162: 1 cores
seg163: 1 cores
seg171: 2 cores
seg173: 1 cores
seg174: 1 cores
seg175: 2 cores
seg177: 2 cores
seg178: 1 cores
seg179: 1 cores
seg180: 2 cores
seg181: 1 cores
seg183: 1 cores
seg187: 1 cores
seg19: 1 cores
seg44: 1 cores
seg45: 2 cores
seg46: 1 cores
seg47: 1 cores
seg51: 3 cores
seg53: 1 cores
seg54: 2 cores
seg55: 1 cores
seg57: 2 cores
seg60: 1 cores
seg61: 1 cores
seg62: 1 cores
seg63: 1 cores
seg69: 1 cores
seg70: 1 cores
seg71: 1 cores
Analysis Script
#!/usr/bin/env python3
import os
import json
import datetime
from collections import defaultdict
from pathlib import Path
def load_core_analysis(json_file):
"""Load and parse a core analysis JSON file."""
with open(json_file) as f:
return json.load(f)
def extract_process_info(file_output):
"""Extract process info from core file output string."""
info = {}
try:
parts = file_output.split("from 'postgres: ")[1].split("'")[0].strip().split()
info['pid'] = parts[0].strip(',')
conn_details = ' '.join(parts[1:])
info['details'] = conn_details
for part in conn_details.split():
if 'seg' in part:
info['segment'] = part
elif 'slice' in part:
info['slice'] = part
elif 'cmd' in part:
info['cmd'] = part
except:
info['details'] = 'Parse failed'
return info
def analyze_stack_trace(stack_trace):
"""Analyze stack trace for key patterns."""
if not stack_trace:
return {}
top_functions = [frame.get('function', 'unknown') for frame in stack_trace[:5]]
return {
'top_functions': top_functions,
'crash_function': top_functions[0] if top_functions else 'unknown'
}
def analyze_cores(base_dir):
results = {
'cores_by_host': defaultdict(list),
'stack_patterns': defaultdict(int),
'segments': defaultdict(list),
'timing': [],
'signals': defaultdict(int)
}
for host_dir in Path(base_dir).iterdir():
if not host_dir.is_dir():
continue
host = host_dir.name
for json_file in host_dir.glob("*.json"):
analysis = load_core_analysis(json_file)
proc_info = extract_process_info(analysis['file_info']['file_output'])
timestamp = datetime.datetime.fromisoformat(analysis['timestamp'])
results['timing'].append({
'host': host,
'timestamp': timestamp,
'segment': proc_info.get('segment', 'unknown'),
'file': json_file.name
})
stack_info = analyze_stack_trace(analysis.get('stack_trace', []))
stack_key = tuple(stack_info.get('top_functions', ['unknown']))
results['stack_patterns'][stack_key] += 1
signal_info = analysis.get('signal_info', {})
signal_name = signal_info.get('signal_name', 'Unknown')
results['signals'][signal_name] += 1
results['cores_by_host'][host].append({
'file': json_file.name,
'timestamp': timestamp,
'process_info': proc_info,
'signal': signal_name,
'crash_function': stack_info.get('crash_function', 'unknown')
})
if 'segment' in proc_info:
results['segments'][proc_info['segment']].append({
'host': host,
'timestamp': timestamp,
'file': json_file.name
})
return results
def print_analysis(results):
print("\nCore File Analysis Summary")
print("=" * 50)
print("\nHosts Affected:", len(results['cores_by_host']))
for host, cores in sorted(results['cores_by_host'].items()):
print(f"\n{host}: {len(cores)} core files")
for core in sorted(cores, key=lambda x: x['timestamp']):
print(f" {core['timestamp'].strftime('%H:%M:%S')} - {core['process_info'].get('segment', 'unknown')} "
f"({core['signal']})")
print("\nStack Trace Patterns")
print("-" * 30)
for stack, count in sorted(results['stack_patterns'].items(), key=lambda x: x[1], reverse=True):
print(f"\nPattern occurred {count} times:")
for i, func in enumerate(stack, 1):
print(f" {i}. {func}")
print("\nSignal Distribution")
print("-" * 30)
for signal, count in sorted(results['signals'].items(), key=lambda x: x[1], reverse=True):
print(f"{signal}: {count}")
timestamps = [t['timestamp'] for t in results['timing']]
if timestamps:
print("\nTiming Analysis")
print("-" * 30)
min_time = min(timestamps)
max_time = max(timestamps)
duration = max_time - min_time
print(f"First core: {min_time.strftime('%H:%M:%S.%f')}")
print(f"Last core: {max_time.strftime('%H:%M:%S.%f')}")
print(f"Duration: {duration.total_seconds():.3f} seconds")
print("\nSegment Distribution")
print("-" * 30)
for segment, occurrences in sorted(results['segments'].items()):
print(f"{segment}: {len(occurrences)} cores")
if __name__ == "__main__":
print("Starting core file analysis...")
results = analyze_cores('.')
print_analysis(results)
can you provide core and postgres binary ? @antoniopetrole
And table ddl of myschema.mytable ?
@yjhjstz Wow I can't believe I forgot to upload the ddl, I've attached it to this comment. Also I shared the coredump with @edespino directly, I don't want to share it publicly since it can contain sensitive data. segfault_table.zip
I have looked through corefiles analysis. It seems that the segmentation fault if caused by another thread with stack:
{
"thread_id": "1",
"lwp_id": "unknown",
"backtrace": [
{
"frame_num": "0",
"location": "0x00007ff0b041cbaf",
"function": "raise",
"args": ""
},
{
"frame_num": "1",
"location": "0x00007ff0b11187bb",
"function": "StandardHandlerForSigillSigsegvSigbus_OnMainThread",
"args": ""
},
{
"frame_num": "3",
"location": "0x00007ff0b116da24",
"function": "comparetup_heap",
"args": ""
},
{
"frame_num": "4",
"location": "0x00007ff0b1169dc2",
"function": "qsort_tuple",
"args": ""
},
{
"frame_num": "5",
"location": "0x00007ff0b116d1f4",
"function": "dumptuples",
"args": ""
},
{
"frame_num": "6",
"location": "0x00007ff0b1170bf6",
"function": "tuplesort_performsort",
"args": ""
},
{
"frame_num": "7",
"location": "0x00007ff0b0ddfbf7",
"function": "ExecSort",
"args": ""
},
{
"frame_num": "8",
"location": "0x00007ff0b0dac77b",
"function": "ExecProcNodeFirst",
"args": ""
},
{
"frame_num": "9",
"location": "0x00007ff0b0de5098",
"function": "begin_partition",
"args": ""
},
{
"frame_num": "10",
"location": "0x00007ff0b0de9278",
"function": "ExecWindowAgg",
"args": ""
},
{
"frame_num": "11",
"location": "0x00007ff0b0dac77b",
"function": "ExecProcNodeFirst",
"args": ""
},
{
"frame_num": "12",
"location": "0x00007ff0b0df1b1c",
"function": "ExecMotion",
"args": ""
},
{
"frame_num": "13",
"location": "0x00007ff0b0dac77b",
"function": "ExecProcNodeFirst",
"args": ""
},
{
"frame_num": "14",
"location": "0x00007ff0b0da347a",
"function": "ExecutePlan.part.1",
"args": ""
},
{
"frame_num": "15",
"location": "0x00007ff0b0da3f89",
"function": "standard_ExecutorRun",
"args": ""
},
{
"frame_num": "16",
"location": "0x00007ff0b0da4135",
"function": "ExecutorRun",
"args": ""
},
{
"frame_num": "17",
"location": "0x00007ff0b0fbe95e",
"function": "PortalRunSelect",
"args": ""
},
{
"frame_num": "18",
"location": "0x00007ff0b0fc022c",
"function": "PortalRun",
"args": ""
},
{
"frame_num": "19",
"location": "0x00007ff0b0fba5df",
"function": "exec_mpp_query",
"args": ""
},
{
"frame_num": "20",
"location": "0x00007ff0b0fbdebd",
"function": "PostgresMain",
"args": ""
},
{
"frame_num": "21",
"location": "0x00007ff0b0f19ddf",
"function": "ServerLoop",
"args": ""
},
{
"frame_num": "22",
"location": "0x00007ff0b0f1af1f",
"function": "PostmasterMain",
"args": ""
},
{
"frame_num": "23",
"location": "0x00000000004017ae",
"function": "main",
"args": ""
}
]
}
],
I will continue to analyze this issue anyway, and it will be helpful if you can provide any one of these corefile and your postgres binary. @antoniopetrole
@foreyes You can reach out to @edespino for the subset of core files, he will be distributing it amongst the core maintainers for analysis. Thanks for looking into this!
@foreyes & @my-ship-it - Here is what I have been able to retrieve from the core files. Given the schema, I was not able to reproduce the issue on a dev environment (single-node).
09:47:33 gpadmin@mdw ~ → ~/cbtoolbox/cbtoolbox core /tmp/cloudberry-cores --output-dir . --gdb-style
CloudBerry Database Core Analysis
================================
Process: Coordinator Write (Read-Only Mode), Database 4001, Segment 177, Connection 349603, Client 10.9.1.160 (PID 64286)
Core: /tmp/cloudberry-cores/core-postgres-11-500-500-3734524-1734389746
Time: 2024-12-26T21:48:02Z
PostgreSQL: postgres (Cloudberry Database) 14.4
CloudBerry: postgres (Cloudberry Database) 1.6.0 build 1
Signal Configuration:
Signal Stop Print Pass Description
SIGSEGV Yes Yes Yes Segmentation fault in rxThreadFunc (thread: Interconnect RX)
Program received signal SIGSEGV (11), Segmentation fault in rxThreadFunc (thread: Interconnect RX)
Fault address: 0x1f40038fbfc
Thread Information:
Thread 2 [LWP 3734528] (Interconnect RX):
#0 0x00007f179ddffac1 in poll from libc.so.6
#1 0x00007f178ad74049 in rxThreadFunc from interconnect.so
#2 0x00007f179e0ab1ca in start_thread from libpthread.so.0
#3 0x00007f179dd068d3 in clone from libc.so.6
Thread 1 [LWP 3734524] (Signal Handler):
#0 0x00007f179e0b5baf in raise from libpthread.so.0
#1 0x00007f179edb17bb in StandardHandlerForSigillSigsegvSigbus_OnMainThread from libpostgres.so
#2 0x00007f179e0b5d10 in ?? from libpthread.so.0
#3 0x0000000000000007 in ??
#4 0x0000000000000000 in ??
Registers:
rax: 0x0 0
rbx: 0xb 11
rcx: 0x7f179e0b5baf 139739412519855
rdx: 0x0 0
rsi: 0x7ffed885b180 140732531061120
rdi: 0x2 2
rbp: 0x118 0x118
rsp: 0x7ffed885b180 0x7ffed885b180
r8: 0x0 0
r9: 0x7ffed885b180 140732531061120
r10: 0x8 8
r11: 0x246 582
r12: 0x7ffed885b2c0 140732531061440
r13: 0x3e8 1000
r14: 0xb 11
r15: 0x7ffed885b340 140732531061568
rip: 0x7f179e0b5baf 0x7f179e0b5baf <raise+255>
eflags: 0x246 [ PF ZF IF ]
es: 0x0 0
Key Shared Libraries:
libpostgres.so [0x00007f179e741fb0-0x00007f179f1742a2]
interconnect.so [0x00007f178ad656c0-0x00007f178ad80bd1]
gp_zstd_compression.so [0x00007f1777016b10-0x00007f1777016fb3]
gdb /usr/local/cloudberry-db-1.6.0/bin/postgres /tmp/cloudberry-cores/core-postgres-11-500-500-3734524-1734389746
GNU gdb (GDB) Rocky Linux 8.2-20.el8.0.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/cloudberry-db-1.6.0/bin/postgres...(no debugging symbols found)...done.
warning: Can't open file (null) during file-backed mapping note processing
warning: Can't open file (null) during file-backed mapping note processing
warning: Can't open file (null) during file-backed mapping note processing
[New LWP 3734524]
[New LWP 3734528]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `postgres: 4001, read_only coredw 10.9.1.160(64286) con349603 seg177 cmd2 slice'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f179e0b5baf in raise () from /lib64/libpthread.so.0
[Current thread is 1 (Thread 0x7f17a003d880 (LWP 3734524))]
Missing separate debuginfos, use: yum debuginfo-install cloudberry-db-1.6.0-1.el8.x86_64
(gdb) w set pagination off
Ambiguous command "w set pagination off": watch, wh, whatis, where, while, while-stepping, winheight, ws.
(gdb) set pagination off
(gdb) set print pretty on
(gdb) set print object on
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f17a003d880 (LWP 3734524) 0x00007f179e0b5baf in raise () from /lib64/libpthread.so.0
2 Thread 0x7f179ff0d700 (LWP 3734528) 0x00007f179ddffac1 in poll () from /lib64/libc.so.6
(gdb) thread apply all bt full
Thread 2 (Thread 0x7f179ff0d700 (LWP 3734528)):
#0 0x00007f179ddffac1 in poll () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f178ad74049 in rxThreadFunc () from /usr/local/cloudberry-db-1.6.0/lib/postgresql/interconnect.so
No symbol table info available.
#2 0x00007f179e0ab1ca in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00007f179dd068d3 in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7f17a003d880 (LWP 3734524)):
#0 0x00007f179e0b5baf in raise () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007f179edb17bb in StandardHandlerForSigillSigsegvSigbus_OnMainThread () from /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
No symbol table info available.
#2 0x00007f179e0b5d10 in ?? () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x0000000000000007 in ?? ()
No symbol table info available.
#4 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) info registers all
rax 0x0 0
rbx 0xb 11
rcx 0x7f179e0b5baf 139739412519855
rdx 0x0 0
rsi 0x7ffed885b180 140732531061120
rdi 0x2 2
rbp 0x118 0x118
rsp 0x7ffed885b180 0x7ffed885b180
r8 0x0 0
r9 0x7ffed885b180 140732531061120
r10 0x8 8
r11 0x246 582
r12 0x7ffed885b2c0 140732531061440
r13 0x3e8 1000
r14 0xb 11
r15 0x7ffed885b340 140732531061568
rip 0x7f179e0b5baf 0x7f179e0b5baf <raise+255>
eflags 0x246 [ PF ZF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
st0 0 (raw 0x00000000000000000000)
st1 0 (raw 0x00000000000000000000)
st2 0 (raw 0x00000000000000000000)
st3 0 (raw 0x00000000000000000000)
st4 0 (raw 0x00000000000000000000)
st5 0 (raw 0x00000000000000000000)
st6 0 (raw 0x00000000000000000000)
st7 0 (raw 0x00000000000000000000)
fctrl 0x37f 895
fstat 0x0 0
ftag 0xffff 65535
fiseg 0x0 0
fioff 0x0 0
foseg 0x0 0
fooff 0x0 0
fop 0x0 0
mxcsr 0x1f80 [ IM DM ZM OM UM PM ]
k0 0xf0776004 4034355204
k1 0x0 0
k2 0x0 0
k3 0x0 0
k4 0x0 0
k5 0x0 0
k6 0x0 0
k7 0x0 0
pkru 0x55555554 1431655764
zmm0 {
v16_float = {0x40000000, 0x0, 0x0 <repeats 14 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0xb8, 0xe3, 0x85, 0xd8, 0xfe, 0x7f, 0x0 <repeats 58 times>},
v32_int16 = {0xe3b8, 0xd885, 0x7ffe, 0x0 <repeats 29 times>},
v16_int32 = {0xd885e3b8, 0x7ffe, 0x0 <repeats 14 times>},
v8_int64 = {0x7ffed885e3b8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x7ffed885e3b8, 0x0, 0x0, 0x0}
}
zmm1 {
v16_float = {0x0, 0x0, 0x0, 0x0 <repeats 13 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0xc0, 0xe3, 0x85, 0xd8, 0xfe, 0x7f, 0x0, 0x0, 0x4e, 0x19, 0x40, 0x0 <repeats 53 times>},
v32_int16 = {0xe3c0, 0xd885, 0x7ffe, 0x0, 0x194e, 0x40, 0x0 <repeats 26 times>},
v16_int32 = {0xd885e3c0, 0x7ffe, 0x40194e, 0x0 <repeats 13 times>},
v8_int64 = {0x7ffed885e3c0, 0x40194e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x40194e00007ffed885e3c0, 0x0, 0x0, 0x0}
}
zmm2 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm3 {
v16_float = {0x0, 0x0, 0x0, 0x0 <repeats 13 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x19, 0x40, 0x0 <repeats 53 times>},
v32_int16 = {0x0, 0x0, 0x0, 0x0, 0x1920, 0x40, 0x0 <repeats 26 times>},
v16_int32 = {0x0, 0x0, 0x401920, 0x0 <repeats 13 times>},
v8_int64 = {0x0, 0x401920, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x4019200000000000000000, 0x0, 0x0, 0x0}
}
zmm4 {
v16_float = {0x0, 0x0 <repeats 15 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x1, 0x0 <repeats 63 times>},
v32_int16 = {0x1, 0x0 <repeats 31 times>},
v16_int32 = {0x1, 0x0 <repeats 15 times>},
v8_int64 = {0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x1, 0x0, 0x0, 0x0}
}
zmm5 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm6 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm7 {
v16_float = {0x0, 0x0, 0x80000000, 0x0, 0x0 <repeats 12 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0, 0xe3, 0x85, 0xd8, 0xfe, 0x7f, 0x0, 0x0, 0xd0, 0xe2, 0x85, 0xd8, 0xfe, 0x7f, 0x0 <repeats 50 times>},
v32_int16 = {0xe300, 0xd885, 0x7ffe, 0x0, 0xe2d0, 0xd885, 0x7ffe, 0x0 <repeats 25 times>},
v16_int32 = {0xd885e300, 0x7ffe, 0xd885e2d0, 0x7ffe, 0x0 <repeats 12 times>},
v8_int64 = {0x7ffed885e300, 0x7ffed885e2d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x7ffed885e2d000007ffed885e300, 0x0, 0x0, 0x0}
}
zmm8 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm9 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm10 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm11 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm12 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm13 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm14 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm15 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm16 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm17 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm18 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm19 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm20 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm21 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm22 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm23 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm24 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm25 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm26 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm27 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm28 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm29 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm30 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
zmm31 {
v16_float = {0x0 <repeats 16 times>},
v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v64_int8 = {0x0 <repeats 64 times>},
v32_int16 = {0x0 <repeats 32 times>},
v16_int32 = {0x0 <repeats 16 times>},
v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
v4_int128 = {0x0, 0x0, 0x0, 0x0}
}
(gdb) print $_siginfo
$1 = {
si_signo = 11,
si_errno = 0,
si_code = -6,
_sifields = {
_pad = {3734524, 500, 0 <repeats 26 times>},
_kill = {
si_pid = 3734524,
si_uid = 500
},
_timer = {
si_tid = 3734524,
si_overrun = 500,
si_sigval = {
sival_int = 0,
sival_ptr = 0x0
}
},
_rt = {
si_pid = 3734524,
si_uid = 500,
si_sigval = {
sival_int = 0,
sival_ptr = 0x0
}
},
_sigchld = {
si_pid = 3734524,
si_uid = 500,
si_status = 0,
si_utime = 0,
si_stime = 0
},
_sigfault = {
si_addr = 0x1f40038fbfc,
_addr_lsb = 0,
_addr_bnd = {
_lower = 0x0,
_upper = 0x0
}
},
_sigpoll = {
si_band = 2147487382524,
si_fd = 0
}
}
}
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f179fc15740 0x00007f179fc22b27 Yes (*) /lib64/libz.so.1
0x00007f179e741fb0 0x00007f179f1742a2 Yes (*) /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
0x00007f179e0a9840 0x00007f179e0b7d15 Yes (*) /lib64/libpthread.so.0
0x00007f179dceec40 0x00007f179de4b4bd Yes (*) /lib64/libc.so.6
0x00007f179d8456c0 0x00007f179d9b7583 Yes (*) /usr/local/cloudberry-db-1.6.0/lib/libxerces-c-3.2.so
0x00007f179d4fb440 0x00007f179d5125db Yes (*) /lib64/liblz4.so.1
0x00007f179d2e9980 0x00007f179d2f6626 Yes (*) /lib64/libbz2.so.1
0x00007f179cfaf490 0x00007f179d093ede Yes (*) /lib64/libxml2.so.2
0x00007f179cd72f30 0x00007f179cd7a9da Yes (*) /lib64/libpam.so.0
0x00007f179cb6a430 0x00007f179cb6d770 Yes (*) /lib64/librt.so.1
0x00007f179c8f3db0 0x00007f179c9409ea Yes (*) /lib64/libssl.so.1.1
0x00007f179c464000 0x00007f179c609e20 Yes (*) /lib64/libcrypto.so.1.1
0x00007f179c1a1ac0 0x00007f179c1d83e4 Yes (*) /lib64/libgssapi_krb5.so.2
0x00007f179bf8fe70 0x00007f179bf90a82 Yes (*) /lib64/libdl.so.2
0x00007f179bd0ff70 0x00007f179bd6fb6b Yes (*) /lib64/libcurl.so.4
0x00007f179ba60e70 0x00007f179baed67a Yes (*) /lib64/libzstd.so.1
0x00007f179b81d400 0x00007f179b84b960 Yes (*) /lib64/libldap-2.4.so.2
0x00007f179b507b90 0x00007f179b5bdae2 Yes (*) /lib64/libstdc++.so.6
0x00007f179b102520 0x00007f179b1a180a Yes (*) /lib64/libm.so.6
0x00007f179aee0e00 0x00007f179aef1a55 Yes (*) /lib64/libgcc_s.so.1
0x00007f179fe2c080 0x00007f179fe50407 Yes /lib64/ld-linux-x86-64.so.2
0x00007f179acba5d0 0x00007f179acd1516 Yes (*) /lib64/liblzma.so.5
0x00007f179aa89a20 0x00007f179aa90ca3 Yes (*) /lib64/libaudit.so.1
0x00007f179a7c29b0 0x00007f179a82bdd1 Yes (*) /lib64/libkrb5.so.3
0x00007f179a5893a0 0x00007f179a594f54 Yes (*) /lib64/libk5crypto.so.3
0x00007f179a3817c0 0x00007f179a3823e9 Yes (*) /lib64/libcom_err.so.2
0x00007f179a173560 0x00007f179a17b30b Yes (*) /lib64/libkrb5support.so.0
0x00007f1799f6c7c0 0x00007f1799f6d538 Yes (*) /lib64/libkeyutils.so.1
0x00007f1799d56d60 0x00007f1799d6316e Yes (*) /lib64/libresolv.so.2
0x00007f1799b31160 0x00007f1799b44135 Yes (*) /lib64/libnghttp2.so.14
0x00007f179990f9f0 0x00007f1799913862 Yes (*) /lib64/libidn2.so.0
0x00007f17996aedd0 0x00007f17996ee935 Yes (*) /lib64/libssh.so.4
0x00007f179948e580 0x00007f179948ff86 Yes (*) /lib64/libpsl.so.5
0x00007f1799280ce0 0x00007f179928801a Yes (*) /lib64/liblber-2.4.so.2
0x00007f1799070c00 0x00007f1799078039 Yes (*) /lib64/libbrotlidec.so.1
0x00007f1798e573f0 0x00007f1798e68817 Yes (*) /lib64/libsasl2.so.3
0x00007f1798c4d8c0 0x00007f1798c4f870 Yes (*) /lib64/libcap-ng.so.0
0x00007f1798a28a80 0x00007f1798a407bf Yes (*) /lib64/libselinux.so.1
0x00007f17986b2770 0x00007f17986e64fe Yes (*) /lib64/libunistring.so.2
0x00007f179847f6b0 0x00007f179847f997 Yes (*) /lib64/libbrotlicommon.so.1
0x00007f17982573d0 0x00007f1798261040 Yes (*) /lib64/libcrypt.so.1
0x00007f1797fd4380 0x00007f179802f973 Yes (*) /lib64/libpcre2-8.so.0
0x00007f178ad656c0 0x00007f178ad80bd1 Yes (*) /usr/local/cloudberry-db-1.6.0/lib/postgresql/interconnect.so
0x00007f178ab37890 0x00007f178ab52220 Yes (*) /lib64/libuv.so.1
0x00007f177729c7b0 0x00007f17772a2d32 Yes (*) /lib64/libnss_files.so.2
0x00007f1777016b10 0x00007f1777016fb3 Yes (*) /usr/local/cloudberry-db-1.6.0/lib/postgresql/gp_zstd_compression.so
(*): Shared library is missing debugging information.
x/1i $pc
=> 0x7f179e0b5baf <raise+255>: rorb %cl,-0x32(%rcx,%rcx,4)
(gdb) info proc mappings
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0x400000 0x403000 0x3000 0x0 /usr/local/cloudberry-db-1.6.0/bin/postgres
0x602000 0x603000 0x1000 0x2000 /usr/local/cloudberry-db-1.6.0/bin/postgres
0x603000 0x604000 0x1000 0x3000 /usr/local/cloudberry-db-1.6.0/bin/postgres
0x7f1777016000 0x7f1777018000 0x2000 0x0 /usr/local/cloudberry-db-1.6.0/lib/postgresql/gp_zstd_compression.so
0x7f1777018000 0x7f1777217000 0x1ff000 0x2000 /usr/local/cloudberry-db-1.6.0/lib/postgresql/gp_zstd_compression.so
0x7f1777217000 0x7f1777218000 0x1000 0x1000 /usr/local/cloudberry-db-1.6.0/lib/postgresql/gp_zstd_compression.so
0x7f1777218000 0x7f1777219000 0x1000 0x2000 /usr/local/cloudberry-db-1.6.0/lib/postgresql/gp_zstd_compression.so
0x7f177729a000 0x7f17772a5000 0xb000 0x0 /usr/lib64/libnss_files-2.28.so
0x7f17772a5000 0x7f17774a5000 0x200000 0xb000 /usr/lib64/libnss_files-2.28.so
0x7f17774a5000 0x7f17774a6000 0x1000 0xb000 /usr/lib64/libnss_files-2.28.so
0x7f17774a6000 0x7f17774a7000 0x1000 0xc000 /usr/lib64/libnss_files-2.28.so
0x7f17774ad000 0x7f178ab2b000 0x1367e000 0x0 /dev/zero (deleted)
0x7f178ab2b000 0x7f178ab5c000 0x31000 0x0 /usr/lib64/libuv.so.1.0.0
0x7f178ab5c000 0x7f178ad5c000 0x200000 0x31000 /usr/lib64/libuv.so.1.0.0
0x7f178ad5c000 0x7f178ad5d000 0x1000 0x31000 /usr/lib64/libuv.so.1.0.0
0x7f178ad5d000 0x7f178ad5e000 0x1000 0x32000 /usr/lib64/libuv.so.1.0.0
0x7f178ad5e000 0x7f178ad8b000 0x2d000 0x0 /usr/local/cloudberry-db-1.6.0/lib/postgresql/interconnect.so
0x7f178ad8b000 0x7f178af8a000 0x1ff000 0x2d000 /usr/local/cloudberry-db-1.6.0/lib/postgresql/interconnect.so
0x7f178af8a000 0x7f178af8b000 0x1000 0x2c000 /usr/local/cloudberry-db-1.6.0/lib/postgresql/interconnect.so
0x7f178af8b000 0x7f178af8c000 0x1000 0x2d000 /usr/local/cloudberry-db-1.6.0/lib/postgresql/interconnect.so
0x7f178b01b000 0x7f1797fd2000 0xcfb7000 0x0 /usr/lib/locale/locale-archive
0x7f1797fd2000 0x7f1798055000 0x83000 0x0 /usr/lib64/libpcre2-8.so.0.7.1
0x7f1798055000 0x7f1798254000 0x1ff000 0x83000 /usr/lib64/libpcre2-8.so.0.7.1
0x7f1798254000 0x7f1798255000 0x1000 0x82000 /usr/lib64/libpcre2-8.so.0.7.1
0x7f1798255000 0x7f1798256000 0x1000 0x83000 /usr/lib64/libpcre2-8.so.0.7.1
0x7f1798256000 0x7f1798276000 0x20000 0x0 /usr/lib64/libcrypt.so.1.1.0
0x7f1798276000 0x7f1798475000 0x1ff000 0x20000 /usr/lib64/libcrypt.so.1.1.0
0x7f1798475000 0x7f1798476000 0x1000 0x1f000 /usr/lib64/libcrypt.so.1.1.0
0x7f179847f000 0x7f179849e000 0x1f000 0x0 /usr/lib64/libbrotlicommon.so.1.0.6
0x7f179849e000 0x7f179869e000 0x200000 0x1f000 /usr/lib64/libbrotlicommon.so.1.0.6
0x7f179869e000 0x7f179869f000 0x1000 0x1f000 /usr/lib64/libbrotlicommon.so.1.0.6
0x7f179869f000 0x7f17986a0000 0x1000 0x20000 /usr/lib64/libbrotlicommon.so.1.0.6
0x7f17986a0000 0x7f179881d000 0x17d000 0x0 /usr/lib64/libunistring.so.2.1.0
0x7f179881d000 0x7f1798a1c000 0x1ff000 0x17d000 /usr/lib64/libunistring.so.2.1.0
0x7f1798a1c000 0x7f1798a20000 0x4000 0x17c000 /usr/lib64/libunistring.so.2.1.0
0x7f1798a20000 0x7f1798a21000 0x1000 0x180000 /usr/lib64/libunistring.so.2.1.0
0x7f1798a21000 0x7f1798a48000 0x27000 0x0 /usr/lib64/libselinux.so.1
0x7f1798a48000 0x7f1798c48000 0x200000 0x27000 /usr/lib64/libselinux.so.1
0x7f1798c48000 0x7f1798c49000 0x1000 0x27000 /usr/lib64/libselinux.so.1
0x7f1798c49000 0x7f1798c4a000 0x1000 0x28000 /usr/lib64/libselinux.so.1
0x7f1798c4c000 0x7f1798c51000 0x5000 0x0 /usr/lib64/libcap-ng.so.0.0.0
0x7f1798c51000 0x7f1798e50000 0x1ff000 0x5000 /usr/lib64/libcap-ng.so.0.0.0
0x7f1798e50000 0x7f1798e51000 0x1000 0x4000 /usr/lib64/libcap-ng.so.0.0.0
0x7f1798e52000 0x7f1798e6e000 0x1c000 0x0 /usr/lib64/libsasl2.so.3.0.0
0x7f1798e6e000 0x7f179906e000 0x200000 0x1c000 /usr/lib64/libsasl2.so.3.0.0
0x7f179906e000 0x7f179906f000 0x1000 0x1c000 /usr/lib64/libsasl2.so.3.0.0
0x7f179906f000 0x7f1799070000 0x1000 0x1d000 /usr/lib64/libsasl2.so.3.0.0
0x7f1799070000 0x7f179907b000 0xb000 0x0 /usr/lib64/libbrotlidec.so.1.0.6
0x7f179907b000 0x7f179927b000 0x200000 0xb000 /usr/lib64/libbrotlidec.so.1.0.6
0x7f179927b000 0x7f179927c000 0x1000 0xb000 /usr/lib64/libbrotlidec.so.1.0.6
0x7f179927d000 0x7f179928b000 0xe000 0x0 /usr/lib64/liblber-2.4.so.2.10.9
0x7f179928b000 0x7f179948b000 0x200000 0xe000 /usr/lib64/liblber-2.4.so.2.10.9
0x7f179948b000 0x7f179948c000 0x1000 0xe000 /usr/lib64/liblber-2.4.so.2.10.9
0x7f179948c000 0x7f179948d000 0x1000 0xf000 /usr/lib64/liblber-2.4.so.2.10.9
0x7f179948d000 0x7f179949d000 0x10000 0x0 /usr/lib64/libpsl.so.5.3.1
0x7f179949d000 0x7f179969c000 0x1ff000 0x10000 /usr/lib64/libpsl.so.5.3.1
0x7f179969c000 0x7f179969d000 0x1000 0xf000 /usr/lib64/libpsl.so.5.3.1
0x7f179969e000 0x7f179970b000 0x6d000 0x0 /usr/lib64/libssh.so.4.8.7
0x7f179970b000 0x7f179990a000 0x1ff000 0x6d000 /usr/lib64/libssh.so.4.8.7
0x7f179990a000 0x7f179990c000 0x2000 0x6c000 /usr/lib64/libssh.so.4.8.7
0x7f179990c000 0x7f179990e000 0x2000 0x6e000 /usr/lib64/libssh.so.4.8.7
0x7f179990e000 0x7f179992a000 0x1c000 0x0 /usr/lib64/libidn2.so.0.3.6
0x7f179992a000 0x7f1799b2a000 0x200000 0x1c000 /usr/lib64/libidn2.so.0.3.6
0x7f1799b2a000 0x7f1799b2b000 0x1000 0x1c000 /usr/lib64/libidn2.so.0.3.6
0x7f1799b2c000 0x7f1799b51000 0x25000 0x0 /usr/lib64/libnghttp2.so.14.17.0
0x7f1799b51000 0x7f1799d50000 0x1ff000 0x25000 /usr/lib64/libnghttp2.so.14.17.0
0x7f1799d50000 0x7f1799d51000 0x1000 0x24000 /usr/lib64/libnghttp2.so.14.17.0
0x7f1799d51000 0x7f1799d53000 0x2000 0x25000 /usr/lib64/libnghttp2.so.14.17.0
0x7f1799d53000 0x7f1799d67000 0x14000 0x0 /usr/lib64/libresolv-2.28.so
0x7f1799d67000 0x7f1799f67000 0x200000 0x14000 /usr/lib64/libresolv-2.28.so
0x7f1799f67000 0x7f1799f68000 0x1000 0x14000 /usr/lib64/libresolv-2.28.so
0x7f1799f68000 0x7f1799f69000 0x1000 0x15000 /usr/lib64/libresolv-2.28.so
0x7f1799f6b000 0x7f1799f6e000 0x3000 0x0 /usr/lib64/libkeyutils.so.1.6
0x7f1799f6e000 0x7f179a16d000 0x1ff000 0x3000 /usr/lib64/libkeyutils.so.1.6
0x7f179a16d000 0x7f179a16e000 0x1000 0x2000 /usr/lib64/libkeyutils.so.1.6
0x7f179a16f000 0x7f179a17e000 0xf000 0x0 /usr/lib64/libkrb5support.so.0.1
0x7f179a17e000 0x7f179a37e000 0x200000 0xf000 /usr/lib64/libkrb5support.so.0.1
0x7f179a37e000 0x7f179a37f000 0x1000 0xf000 /usr/lib64/libkrb5support.so.0.1
0x7f179a37f000 0x7f179a380000 0x1000 0x10000 /usr/lib64/libkrb5support.so.0.1
0x7f179a380000 0x7f179a383000 0x3000 0x0 /usr/lib64/libcom_err.so.2.1
0x7f179a383000 0x7f179a582000 0x1ff000 0x3000 /usr/lib64/libcom_err.so.2.1
0x7f179a582000 0x7f179a583000 0x1000 0x2000 /usr/lib64/libcom_err.so.2.1
0x7f179a583000 0x7f179a584000 0x1000 0x3000 /usr/lib64/libcom_err.so.2.1
0x7f179a584000 0x7f179a599000 0x15000 0x0 /usr/lib64/libk5crypto.so.3.1
0x7f179a599000 0x7f179a798000 0x1ff000 0x15000 /usr/lib64/libk5crypto.so.3.1
0x7f179a798000 0x7f179a79a000 0x2000 0x14000 /usr/lib64/libk5crypto.so.3.1
0x7f179a79a000 0x7f179a79b000 0x1000 0x16000 /usr/lib64/libk5crypto.so.3.1
0x7f179a79b000 0x7f179a875000 0xda000 0x0 /usr/lib64/libkrb5.so.3.3
0x7f179a875000 0x7f179aa75000 0x200000 0xda000 /usr/lib64/libkrb5.so.3.3
0x7f179aa75000 0x7f179aa84000 0xf000 0xda000 /usr/lib64/libkrb5.so.3.3
0x7f179aa84000 0x7f179aa86000 0x2000 0xe9000 /usr/lib64/libkrb5.so.3.3
0x7f179aa86000 0x7f179aaa6000 0x20000 0x0 /usr/lib64/libaudit.so.1.0.0
0x7f179aaa6000 0x7f179aca5000 0x1ff000 0x20000 /usr/lib64/libaudit.so.1.0.0
0x7f179aca5000 0x7f179aca6000 0x1000 0x1f000 /usr/lib64/libaudit.so.1.0.0
0x7f179aca6000 0x7f179aca7000 0x1000 0x20000 /usr/lib64/libaudit.so.1.0.0
0x7f179acb7000 0x7f179acdc000 0x25000 0x0 /usr/lib64/liblzma.so.5.2.4
0x7f179acdc000 0x7f179aedc000 0x200000 0x25000 /usr/lib64/liblzma.so.5.2.4
0x7f179aedc000 0x7f179aedd000 0x1000 0x25000 /usr/lib64/liblzma.so.5.2.4
0x7f179aede000 0x7f179aef5000 0x17000 0x0 /usr/lib64/libgcc_s-8-20210514.so.1
0x7f179aef5000 0x7f179b0f4000 0x1ff000 0x17000 /usr/lib64/libgcc_s-8-20210514.so.1
0x7f179b0f4000 0x7f179b0f5000 0x1000 0x16000 /usr/lib64/libgcc_s-8-20210514.so.1
0x7f179b0f5000 0x7f179b0f6000 0x1000 0x17000 /usr/lib64/libgcc_s-8-20210514.so.1
0x7f179b0f6000 0x7f179b277000 0x181000 0x0 /usr/lib64/libm-2.28.so
0x7f179b277000 0x7f179b476000 0x1ff000 0x181000 /usr/lib64/libm-2.28.so
0x7f179b476000 0x7f179b477000 0x1000 0x180000 /usr/lib64/libm-2.28.so
0x7f179b477000 0x7f179b478000 0x1000 0x181000 /usr/lib64/libm-2.28.so
0x7f179b478000 0x7f179b5fe000 0x186000 0x0 /usr/lib64/libstdc++.so.6.0.25
0x7f179b5fe000 0x7f179b7fd000 0x1ff000 0x186000 /usr/lib64/libstdc++.so.6.0.25
0x7f179b7fd000 0x7f179b809000 0xc000 0x185000 /usr/lib64/libstdc++.so.6.0.25
0x7f179b809000 0x7f179b80a000 0x1000 0x191000 /usr/lib64/libstdc++.so.6.0.25
0x7f179b80d000 0x7f179b85a000 0x4d000 0x0 /usr/lib64/libldap-2.4.so.2.10.9
0x7f179b85a000 0x7f179ba59000 0x1ff000 0x4d000 /usr/lib64/libldap-2.4.so.2.10.9
0x7f179ba59000 0x7f179ba5b000 0x2000 0x4c000 /usr/lib64/libldap-2.4.so.2.10.9
0x7f179ba5b000 0x7f179ba5c000 0x1000 0x4e000 /usr/lib64/libldap-2.4.so.2.10.9
0x7f179ba5c000 0x7f179bafe000 0xa2000 0x0 /usr/lib64/libzstd.so.1.4.4
0x7f179bafe000 0x7f179bcfe000 0x200000 0xa2000 /usr/lib64/libzstd.so.1.4.4
0x7f179bcfe000 0x7f179bcff000 0x1000 0xa2000 /usr/lib64/libzstd.so.1.4.4
0x7f179bcff000 0x7f179bd00000 0x1000 0xa3000 /usr/lib64/libzstd.so.1.4.4
0x7f179bd00000 0x7f179bd8b000 0x8b000 0x0 /usr/lib64/libcurl.so.4.5.0
0x7f179bd8b000 0x7f179bf8b000 0x200000 0x8b000 /usr/lib64/libcurl.so.4.5.0
0x7f179bf8b000 0x7f179bf8e000 0x3000 0x8b000 /usr/lib64/libcurl.so.4.5.0
0x7f179bf8e000 0x7f179bf8f000 0x1000 0x8e000 /usr/lib64/libcurl.so.4.5.0
0x7f179bf8f000 0x7f179bf92000 0x3000 0x0 /usr/lib64/libdl-2.28.so
0x7f179bf92000 0x7f179c191000 0x1ff000 0x3000 /usr/lib64/libdl-2.28.so
0x7f179c191000 0x7f179c192000 0x1000 0x2000 /usr/lib64/libdl-2.28.so
0x7f179c192000 0x7f179c193000 0x1000 0x3000 /usr/lib64/libdl-2.28.so
0x7f179c193000 0x7f179c1e5000 0x52000 0x0 /usr/lib64/libgssapi_krb5.so.2.2
0x7f179c1e5000 0x7f179c3e5000 0x200000 0x52000 /usr/lib64/libgssapi_krb5.so.2.2
0x7f179c3e5000 0x7f179c3e7000 0x2000 0x52000 /usr/lib64/libgssapi_krb5.so.2.2
0x7f179c3e7000 0x7f179c3e8000 0x1000 0x54000 /usr/lib64/libgssapi_krb5.so.2.2
0x7f179c3e8000 0x7f179c69e000 0x2b6000 0x0 /usr/lib64/libcrypto.so.1.1.1k
0x7f179c69e000 0x7f179c89e000 0x200000 0x2b6000 /usr/lib64/libcrypto.so.1.1.1k
0x7f179c89e000 0x7f179c8ca000 0x2c000 0x2b6000 /usr/lib64/libcrypto.so.1.1.1k
0x7f179c8ca000 0x7f179c8ce000 0x4000 0x2e2000 /usr/lib64/libcrypto.so.1.1.1k
0x7f179c8d3000 0x7f179c95a000 0x87000 0x0 /usr/lib64/libssl.so.1.1.1k
0x7f179c95a000 0x7f179cb5a000 0x200000 0x87000 /usr/lib64/libssl.so.1.1.1k
0x7f179cb5a000 0x7f179cb63000 0x9000 0x87000 /usr/lib64/libssl.so.1.1.1k
0x7f179cb63000 0x7f179cb67000 0x4000 0x90000 /usr/lib64/libssl.so.1.1.1k
0x7f179cb68000 0x7f179cb6f000 0x7000 0x0 /usr/lib64/librt-2.28.so
0x7f179cb6f000 0x7f179cd6e000 0x1ff000 0x7000 /usr/lib64/librt-2.28.so
0x7f179cd6e000 0x7f179cd6f000 0x1000 0x6000 /usr/lib64/librt-2.28.so
0x7f179cd6f000 0x7f179cd70000 0x1000 0x7000 /usr/lib64/librt-2.28.so
0x7f179cd70000 0x7f179cd7e000 0xe000 0x0 /usr/lib64/libpam.so.0.84.2
0x7f179cd7e000 0x7f179cf7e000 0x200000 0xe000 /usr/lib64/libpam.so.0.84.2
0x7f179cf7e000 0x7f179cf7f000 0x1000 0xe000 /usr/lib64/libpam.so.0.84.2
0x7f179cf7f000 0x7f179cf80000 0x1000 0xf000 /usr/lib64/libpam.so.0.84.2
0x7f179cf80000 0x7f179d0dd000 0x15d000 0x0 /usr/lib64/libxml2.so.2.9.7
0x7f179d0dd000 0x7f179d2dc000 0x1ff000 0x15d000 /usr/lib64/libxml2.so.2.9.7
0x7f179d2dc000 0x7f179d2e5000 0x9000 0x15c000 /usr/lib64/libxml2.so.2.9.7
0x7f179d2e5000 0x7f179d2e6000 0x1000 0x165000 /usr/lib64/libxml2.so.2.9.7
0x7f179d2e8000 0x7f179d2f8000 0x10000 0x0 /usr/lib64/libbz2.so.1.0.6
0x7f179d2f8000 0x7f179d4f7000 0x1ff000 0x10000 /usr/lib64/libbz2.so.1.0.6
0x7f179d4f7000 0x7f179d4f8000 0x1000 0xf000 /usr/lib64/libbz2.so.1.0.6
0x7f179d4f8000 0x7f179d4f9000 0x1000 0x10000 /usr/lib64/libbz2.so.1.0.6
0x7f179d4f9000 0x7f179d515000 0x1c000 0x0 /usr/lib64/liblz4.so.1.8.3
0x7f179d515000 0x7f179d714000 0x1ff000 0x1c000 /usr/lib64/liblz4.so.1.8.3
0x7f179d714000 0x7f179d715000 0x1000 0x1b000 /usr/lib64/liblz4.so.1.8.3
0x7f179d716000 0x7f179da92000 0x37c000 0x0 /usr/local/cloudberry-db-1.6.0/lib/libxerces-c-3.2.so
0x7f179da92000 0x7f179dc92000 0x200000 0x37c000 /usr/local/cloudberry-db-1.6.0/lib/libxerces-c-3.2.so
0x7f179dc92000 0x7f179dca5000 0x13000 0x37c000 /usr/local/cloudberry-db-1.6.0/lib/libxerces-c-3.2.so
0x7f179dca5000 0x7f179dccc000 0x27000 0x38f000 /usr/local/cloudberry-db-1.6.0/lib/libxerces-c-3.2.so
0x7f179dccd000 0x7f179de9a000 0x1cd000 0x0 /usr/lib64/libc-2.28.so
0x7f179de9a000 0x7f179e099000 0x1ff000 0x1cd000 /usr/lib64/libc-2.28.so
0x7f179e099000 0x7f179e09d000 0x4000 0x1cc000 /usr/lib64/libc-2.28.so
0x7f179e09d000 0x7f179e09f000 0x2000 0x1d0000 /usr/lib64/libc-2.28.so
0x7f179e0a3000 0x7f179e0be000 0x1b000 0x0 /usr/lib64/libpthread-2.28.so
0x7f179e0be000 0x7f179e2bd000 0x1ff000 0x1b000 /usr/lib64/libpthread-2.28.so
0x7f179e2bd000 0x7f179e2be000 0x1000 0x1a000 /usr/lib64/libpthread-2.28.so
0x7f179e2be000 0x7f179e2bf000 0x1000 0x1b000 /usr/lib64/libpthread-2.28.so
0x7f179e2c3000 0x7f179f5bd000 0x12fa000 0x0 /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
0x7f179f5bd000 0x7f179f7bd000 0x200000 0x12fa000 /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
0x7f179f7bd000 0x7f179f82e000 0x71000 0x12fa000 /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
0x7f179f82e000 0x7f179f874000 0x46000 0x136b000 /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
0x7f179fc13000 0x7f179fc29000 0x16000 0x0 /usr/lib64/libz.so.1.2.11
0x7f179fc29000 0x7f179fe29000 0x200000 0x16000 /usr/lib64/libz.so.1.2.11
0x7f179fe29000 0x7f179fe2a000 0x1000 0x16000 /usr/lib64/libz.so.1.2.11
0x7f179fe2b000 0x7f179fe5a000 0x2f000 0x0 /usr/lib64/ld-2.28.so
0x7f17a000e000 0x7f17a003d000 0x2f000 0x0 /dev/shm/PostgreSQL.3975855616
0x7f17a0057000 0x7f17a0058000 0x1000 0x0 /SYSV80000080 (deleted)
0x7f17a005a000 0x7f17a005b000 0x1000 0x2f000 /usr/lib64/ld-2.28.so
0x7f17a005b000 0x7f17a005d000 0x2000 0x30000 /usr/lib64/ld-2.28.so
(gdb) thread apply all print $_thread
Thread 2 (Thread 0x7f179ff0d700 (LWP 3734528)):
$2 = 2
Thread 1 (Thread 0x7f17a003d880 (LWP 3734524)):
$3 = 1
(gdb) print $_siginfo._sifields._sigfault
$4 = {
si_addr = 0x1f40038fbfc,
_addr_lsb = 0,
_addr_bnd = {
_lower = 0x0,
_upper = 0x0
}
}
(gdb) info frame
Stack level 0, frame at 0x7ffed885b2a0:
rip = 0x7f179e0b5baf in raise; saved rip = 0x7f179edb17bb
called by frame at 0x7ffed885c380
Arglist at 0x7ffed885b178, args:
Locals at 0x7ffed885b178, Previous frame's sp is 0x7ffed885b2a0
Saved registers:
rbx at 0x7ffed885b290, rip at 0x7ffed885b298
(gdb) info locals
No symbol table info available.
(gdb) bt full
#0 0x00007f179e0b5baf in raise () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007f179edb17bb in StandardHandlerForSigillSigsegvSigbus_OnMainThread () from /usr/local/cloudberry-db-1.6.0/lib/libpostgres.so
No symbol table info available.
#2 0x00007f179e0b5d10 in ?? () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x0000000000000007 in ?? ()
No symbol table info available.
#4 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) quit
@antoniopetrole - I can provide a debug version (with symbols) of the pre-Apache Cloudberry 1.6.0 release in rpm format. If you have a dev environment that can reproduce this issue, this should be a drop-in-place binary swap.
Okay, I am analyzing the location of the error and trying to reproduce the problem with these informations.
So I've gotten the green light to go ahead and try to recreate this issue in our dev environment. If I manage to do this then the next step will be swapping out the binaries in this environment with a build that has debug flags enabled so we have more information for troubleshooting. I'll keep everyone in the loop as I go about recreating this
Excellent ... I'll be ready with an pre-Apache Cloudberry v1.6.0 debug rpm (el8).
@antoniopetrole - Here is a pre-Apache Cloudberry v1.6.0 debug rpm (el8):
RPM: https://cloudberry-rpm-dev-bucket.s3.us-east-1.amazonaws.com/repo/el8/x86_64/cloudberry-db-1.6.0-1.debug.el8.x86_64.rpm
SHA512: d65f2cefce929414dcd90238eadac9961120c80a6430b3c4e82a6b412da04bb907d05112cca67fd479acc757206b416a275171cb07b3829ff9135b900fb7b02f
Let us know if you reproduce the issue in your dev environment.
Okay, I am analyzing the location of the error and trying to reproduce the problem with these informations.
@foreyes Here is some additional information from the cores.
Just want to give everyone an update, I've yet to be able to recreate this in our development environment. I still have a few more things to try and validate though so I'll keep everyone in the loop
I'm still analyzing this issue. It seems that sufficient information cannot be obtained from the current corefile. Still trying to reproduce this issue, and we have a similar internal issue that may be related to this.
The original coredump file may be helpful, could you please share it to me via email? @edespino
@foreyes I am not conformable sharing the core file directly with you. We could get on a session together and I can share the contents in a live session with you.
@foreyes I am not conformable sharing the core file directly with you. We could get on a session together and I can share the contents in a live session with you.
Hey @edespino, let's schedule a live session, please.
After a thorough investigation of the coredump file, we found that some memory may have been released prematurely, so we have a potential fix: #851. And we are still finding the root cause. @antoniopetrole
Could you help to check if it’s possible to determine whether memory usage was high when the issue occurred, and if there were any related OOM logs or changes to memory-related GUC parameters? Thanks! @antoniopetrole
@foreyes I'll take a look at this either later today or tomorrow. We collect and store metrics around memory, disk, and CPU utilization so we should have data on this. Just an FYI I'm in training all of this week so I might not be responding as quickly as normal, thank you again for spending time looking into this!
Just realized I never replied to this, apologies for that things have been super busy. Just wanted to add some context that Greg Spiegelberg found. He went through all of our coredumps we've had since deploying the cluster and put together a count of the functions that cause coredumps the most. Here are those counts
count | function 1 | detoast_attr 20 | pg_detoast_datum_packed 21 | __memmove_evex_unaligned_erms 138 | comparetup_heap