firebird icon indicating copy to clipboard operation
firebird copied to clipboard

Random FB2.5 crashed after compiling with newer GCC/glib

Open mrgleba opened this issue 1 year ago • 14 comments

As a start I am aware FB2.5 is no longer supported (we are still blocked with the migration because of the cursor stability change). I am filing this issue in case somebody knows a solution and as a reference to others with a similar problem.

We compile FB from source (with -O3) and everything went fine till Debian bullseye (gcc 10.2.1, glibc 2.31).

When repeating the same procedure on Debian bookworm (gcc 12.2, glibc 2.36) and Ubuntu jammy (gcc 11.2.0, glibc 2.35) we get random crashes /usr/local/firebird/bin/fbguard: /usr/local/firebird/bin/fbserver terminated abnormally (-1) or hangs.

We've tried -fno-tree-vectorize as suggested by a known bug in Debian bookworm but this didn't solve the issue.

We weren't able to establish if the problem lies within the compiler or the libraries as compiling with glibc 2.31 and running on glibc 2.34 and later is immpossible due to libpthread being removed/integrated directly into glibc (missing linked lib).

I'd appreciate any help looking for a cause/solution.

Have later versions of FB been successfuly tried running on those newer systems compiled with -O3?

mrgleba avatar Nov 20 '23 16:11 mrgleba

Check https://github.com/FirebirdSQL/firebird/issues/5384. Most likely it is a lifetime or destructor optimization issue.

aafemt avatar Nov 20 '23 16:11 aafemt

After some trial & error I've found the gcc switch that solves the problem: -fabi-version=15

As the problems begin with gcc-11 it seems from https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Dialect-Options.html#index-fabi-version that the problem is: Version 16, which first appeared in G++ 11, changes the mangling of alignof to be distinct from that of alignof, and dependent operator names.

mrgleba avatar Nov 23 '23 09:11 mrgleba

Regretfuly it still crashes with the above flag 😢

mrgleba avatar Nov 23 '23 12:11 mrgleba

Instead of trying to guess what happens I suggest you to use core dumps + gdb and send here stack traces for 2 or 3 crashes. Usually it's better to fix crash :)

AlexPeshkoff avatar Nov 23 '23 13:11 AlexPeshkoff

That's probably where I'll start next :) Are there any tutorials how to do it on linux?

mrgleba avatar Nov 23 '23 13:11 mrgleba

On 11/23/23 16:27, Michał Ziemski wrote:

That's probably where I'll start next :) Are there any tutorials how to do it on linux?

Try http://www.ibphoenix.com/resources/documents/search/doc_36 It's very old but if you carefully skip all about old versions it works.

AlexPeshkoff avatar Nov 23 '23 14:11 AlexPeshkoff

Might https://ib-aid.com/articles/problem-with-libpthread-library-in-linux-installation-of-2.5 be the reason for the crashes?

mrgleba avatar Apr 12 '24 08:04 mrgleba

On 4/12/24 11:59, Michał Ziemski wrote:

Might https://ib-aid.com/articles/problem-with-libpthread-library-in-linux-installation-of-2.5 be the reason for the crashes?

Not realted with gcc, what about glibc - definitely may be the reason. Try suggested test first of all.

AlexPeshkoff avatar Apr 12 '24 09:04 AlexPeshkoff

This seems to be the problem. In our current environment (old debian bullseye):

Set signals
Sigenv works

On the newer system (debian bookworm):

Set signals
Segmentation fault (core dumped)

Any chance for a backport to 2.5 (even as just a commit) (a humble plea from "WELES" a years-long silver sponsor of Firebird)? We're still some work away before we can migrate to ver 5. Cursor stability introduced in 3.0 is a blocker for us as we'd need to upgrade our code to factor that in.

mrgleba avatar Apr 12 '24 09:04 mrgleba

The main problem here is that I'm missing build/test infrastructure for FB25 (almost 15 years old compiler/etc is needed). What do you think about HQ patch? But please be aware that it's also not well tested yet.

AlexPeshkoff avatar Apr 12 '24 10:04 AlexPeshkoff

Is the HQ patch available somwhere publicly? We could give it a try in our dev environment.

mrgleba avatar Apr 12 '24 13:04 mrgleba

segfault.zip Please try it

AlexPeshkoff avatar Apr 15 '24 09:04 AlexPeshkoff

Thank you. I'll give it a spin in our dev environment and report back.

mrgleba avatar Apr 15 '24 10:04 mrgleba

After ~1 month of testing I can confirm the patch solves the issue.

In case anyone should look for the patched code, it's available at: https://github.com/mrgleba/firebird/commits/B2_5_Release

mrgleba avatar May 20 '24 13:05 mrgleba