firebird icon indicating copy to clipboard operation
firebird copied to clipboard
verified publisher icon FirebirdSQL

Firebird 4.04 - Superserver (64 Bit) - Crash during backup & Crash normal use

Open NilsBoedeker opened this issue 1 year ago • 17 comments

Hi,

We use FB 4.04 for two month and we observe a lot of FB crashs during normal use and especial during backup prozess.

a) The crashes happen often when the FB server is under load, 24/7, between 125 and 200 connections and 300 and 400 active statement if we start an backup . We use the parameter -SE but the crash also happend without this parmeter

b) The crash also happend under normal use (without backup. Every 2 days (+- 1 day) we have also an crash of fbserver. the IBGuard restarts the fbserver.exe process.

We use FB with very much RAM (48 GB) and 16 CPUs, FB Settings via. https://cc.ib-aid.com/democalc.html

Additional: We use an own written UDR and der FreeAdhocUDF.dll, we know that UDFs and are possible problem for stability but the most queries are without using this UDFs & UDRs. But the crash dump don't point in this direction

Have you any idea how we can solve the issue wirt the crash during the backup?

Du you need any additional information & data

With best regards

Nils Bödeker

Crash Dump Analyse attached:

Verlag Eugen Ulmer

Web- und Softwareentwicklung

Nils Bödeker

Bürgerwohlsweg 7

28215 Bremen

Tel: [+49 172 7468066]

[email protected]

Details

************* Preparing the environment for Debugger Extensions Gallery repositories ************** ExtensionRepository : Implicit UseExperimentalFeatureForNugetShare : true AllowNugetExeUpdate : true NonInteractiveNuget : true AllowNugetMSCredentialProviderInstall : true AllowParallelInitializationOfLocalRepositories : true

EnableRedirectToV8JsProvider : false

-- Configuring repositories ----> Repository : LocalInstalled, Enabled: true ----> Repository : UserExtensions, Enabled: true

Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.015 seconds ----> Repository : UserExtensions, Enabled: true, Packages count: 0 ----> Repository : LocalInstalled, Enabled: true, Packages count: 41

Microsoft (R) Windows Debugger Version 10.0.27553.1004 AMD64 Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [D:\tmp\firebird.exe.5968.dmp] User Mini Dump File with Full Memory: Only application data is available

************* Path validation summary ************** Response Time (ms) Location Deferred srv* Symbol search path is: srv* Executable search path is: Windows 10 Version 17763 MP (16 procs) Free x64 Product: Server, suite: TerminalServer SingleUserTS Edition build lab: 17763.1.amd64fre.rs5_release.180914-1434 Debug session time: Tue Apr 2 16:34:52.000 2024 (UTC + 2:00) System Uptime: 41 days 17:39:48.307 Process Uptime: 0 days 5:32:06.000 ................................................................ ........ Loading unloaded module list ..... This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (1750.1c08): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available) Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT For analysis of this file, run !analyze -v ucrtbase!abort+0x4e: 00007ffe`cd6ce77e cd29 int 29h 0:014> !analyze -v

  •                                                                         *

  •                    Exception Analysis                                   *

  •                                                                         *

*** WARNING: Check Image - Checksum mismatch - Dump: 0x10252d, File: 0x10606e - C:\ProgramData\Dbg\sym\ucrtbase.dll\48AC8393fa000\ucrtbase.dll *** WARNING: Unable to verify checksum for fbclient.dll *** WARNING: Unable to verify checksum for engine13.dll

KEY_VALUES_STRING: 1

Key  : Analysis.CPU.mSec
Value: 358

Key  : Analysis.Elapsed.mSec
Value: 3372

Key  : Analysis.IO.Other.Mb
Value: 1

Key  : Analysis.IO.Read.Mb
Value: 0

Key  : Analysis.IO.Write.Mb
Value: 2

Key  : Analysis.Init.CPU.mSec
Value: 61

Key  : Analysis.Init.Elapsed.mSec
Value: 11831

Key  : Analysis.Memory.CommitPeak.Mb
Value: 98

Key  : FailFast.Name
Value: FATAL_APP_EXIT

Key  : FailFast.Type
Value: 7

Key  : Failure.Bucket
Value: FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort

Key  : Failure.Hash
Value: {e31753ac-c98a-8055-3663-47e707543d20}

Key  : Timeline.OS.Boot.DeltaSec
Value: 3605988

Key  : Timeline.Process.Start.DeltaSec
Value: 19926

Key  : WER.OS.Branch
Value: rs5_release

Key  : WER.OS.Version
Value: 10.0.17763.1

Key  : WER.Process.Version
Value: 4.0.4.3010

FILE_IN_CAB: firebird.exe.5968.dmp

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

CONTEXT: 00007ffed0b388d1 -- (.cxr 0x7ffed0b388d1) rax=480000021e850f0f rbx=8341d23300000086 rcx=894800000098838b rdx=850f00007d830845 rsi=0327840ff8b54148 rdi=483cfe24068a0000 rip=0ffda817418d0000 rsp=000003e6830f02fd rbp=3e80d68b48ff3345 r8=0a8a00000341840f r9=0f583cc52241c18a r10=4cc18a0000052884 r11=0f403cf02401428d r12=f2f9800000022484 r13=418d000a2c5a840f r14=000282860f013c3e r15=0565840ff3f98000 iopl=2 nv dn di ng nz na pe cy cs=2e79 ss=1065 ds=000a es=8b44 fs=0870 gs=8b4c efl=00006583 0ffda817`418d0000 ?? ??? Resetting default scope

EXCEPTION_RECORD: 00000000703eccc0 -- (.exr 0x703eccc0) ExceptionAddress: 00001fa00010004f ExceptionCode: 69727720 ExceptionFlags: 6e657474 NumberParameters: 2818099 Parameter[0]: 00000202002b002b Parameter[1]: 36313a6b6162670a Parameter[2]: 00001fa00010004f Parameter[3]: 0053002b002b0033 Parameter[4]: 00000206002b002b Parameter[5]: 3536313a6b616267 Parameter[6]: 6572203030303031 Parameter[7]: 7277207364726f63 Parameter[8]: 670a0d6e65747469 Parameter[9]: 323536313a6b6162 Parameter[10]: 00007ffec1a74528 Parameter[11]: 00000000703edaa0 Parameter[12]: 00000000703edbe0 Parameter[13]: 0000000000000001 Parameter[14]: 00007ffebc12af48

PROCESS_NAME: firebird.exe

ERROR_CODE: (NTSTATUS) 0xc0000409 - Das System hat in dieser Anwendung den berlauf eines stapelbasierten Puffers ermittelt. Dieser berlauf k nnte einem b sartigen Benutzer erm glichen, die Steuerung der Anwendung zu bernehmen.

EXCEPTION_CODE_STR: c0000409

EXCEPTION_PARAMETER1: 0000000000000007

IP_ON_HEAP: 0ffda817418d0000 The fault address in not in any loaded module, please check your build's rebase log at \bin\build_logs\timebuild\ntrebase.log for module which may contain the address if it were loaded.

UNALIGNED_STACK_POINTER: 000003e6830f02fd

FAULTING_THREAD: 00001c08

STACK_TEXT:
00000000703eb150 00007ffec188512e : 0000000000000003 00009a8100000003 fffffffffffffffe 00007ffec18f3cd1 : ucrtbase!abort+0x4e 00000000703eb180 00007ffec18f0b8c : 00000000703eb230 0000000000000000 00000000001de280 0000000000000000 : VCRUNTIME140!_purecall+0x2e 00000000703eb1b0 00007ffec1909259 : 00000000463f2b30 00000000463f2b38 0000000000000000 00000000703eb2a8 : fbclient!isc_version+0x118c 00000000703eb200 00007ffec1905d14 : 00000000463f2b38 00000000703eb328 00000000703edfb8 0000000000000000 : fbclient!isc_version+0x19859 00000000703eb300 00007ffebbeefcf2 : 00006d5d94d032b0 00000000ffffffff 00007ffebbb30000 00007ffec1884be6 : fbclient!isc_version+0x16314 00000000703eb360 00007ffebbde10c9 : 0000000000000000 00000000703edef8 00007ffebc03bfa8 00007ffebc03bfa8 : engine13!firebird_plugin+0x180b92 00000000703eb3a0 00007ffec1881030 : 00007ffebbfb2a40 00007ffe00000103 00000000703edeb0 00007ffed0b7e9c0 : engine13!firebird_plugin+0x71f69 00000000703eb3e0 00007ffec188358f : 00007ffebbfb2a40 00000000703eb4c8 00007ffe00000103 00007ffec188bcd2 : VCRUNTIME140!_CallSettingFrame+0x20 00000000703eb410 00007ffec188bc94 : 00000000703eb4c8 00000000703eb4c8 00007ffebc03bfa8 00000000ffffffff : VCRUNTIME140!__FrameHandler3::FrameUnwindToState+0x11b 00000000703eb480 00007ffec1882fd9 : 00007ffebc03bfa8 00000000703ebb80 00006d5d94d04040 00000000703edeb0 : VCRUNTIME140!__FrameHandler3::FrameUnwindToEmptyState+0x54 00000000703eb4b0 00007ffec188c15d : 00000000703ebb80 00000000703ebd00 00000000703ebd00 00000000703eb600 : VCRUNTIME140!__InternalCxxFrameHandler<__FrameHandler3>+0x111 00000000703eb510 00007ffebbf6c6fc : 00000000703edeb0 00007ffebc10b604 00000000703edeb0 00007ffe00000000 : VCRUNTIME140!__CxxFrameHandler3+0x6d 00000000703eb560 00007ffed0bd4caf : 00007ffebc2cec00 00000000703ebb00 00000000703eeb20 0000000000000000 : engine13!firebird_plugin+0x1fd59c 00000000703eb590 00007ffed0b3610c : 00000000703ebb00 00007ffebc2cec40 00007ffebc2ce5d4 00000000703f0000 : ntdll!RtlpExecuteHandlerForUnwind+0xf 00000000703eb5c0 00007ffec188bfd5 : 00007ffebc10a060 00007ffec188bfd5 00007ffec1a505f8 fffffffffffffffe : ntdll!RtlUnwindEx+0x51c 00000000703ebcd0 00007ffec18826d4 : 00007ffebc10a060 00000000703ec328 00000000703edae0 000000000000000d : VCRUNTIME140!__FrameHandler3::UnwindNestedFrames+0x109 00000000703ec290 00007ffec18829cb : 00000000703edae0 00000000703eeb20 00000000703ec3f9 00007ffebc039150 : VCRUNTIME140!CatchIt<__FrameHandler3>+0xb4 00000000703ec320 00007ffec18830df : 00007ffebc039150 00000000703ec4f0 00000000703eccc0 00000000703ec500 : VCRUNTIME140!FindHandler<__FrameHandler3>+0x2db 00000000703ec440 00007ffec188c15d : 00000000703ecb00 00000000703edae0 00000000703edae0 00000000703eccc0 : VCRUNTIME140!__InternalCxxFrameHandler<__FrameHandler3>+0x217 00000000703ec4a0 00007ffebbf6c6fc : 00000000703eeb20 00007ffebc109fac 00000000703eeb20 0000000000000000 : VCRUNTIME140!__CxxFrameHandler3+0x6d 00000000703ec4f0 00007ffed0bd4c2f : 0000000000000000 00000000703eca90 0000000000000001 00007ffebbb30000 : engine13!firebird_plugin+0x1fd59c 00000000703ec520 00007ffed0b34cef : 00000000703eca90 0000000000000001 0000000000000000 00007ffebbb30000 : ntdll!RtlpExecuteHandlerForException+0xf 00000000703ec550 00007ffed0b38ae6 : 00000000703eccc0 00007ffed0b388d1 00000000703eccc0 0000000000000000 : ntdll!RtlDispatchException+0x40f 00000000703ecc80 00007ffecd8149b9 : 0000000022163cd0 00007ffebc12af48 0000000000000000 00000000703edc48 : ntdll!RtlRaiseException+0x316 00000000703edac0 00007ffec188485d : 000000000000000f 0000000000000001 0000000000000001 00007ffebbea57c9 : KERNELBASE!RaiseException+0x69 00000000703edba0 00007ffebbe64853 : 00007ffebbb30000 00000000703eef90 0000000000000053 0000000000000001 : VCRUNTIME140!_CxxThrowException+0xad 00000000703edc10 00007ffebbe64674 : 00000000703eef90 00007ffebc039258 0000000000000000 00007ffebbea0b61 : engine13!firebird_plugin+0xf56f3 00000000703edc40 00007ffebbe6473f : 00000000703eef90 0000000000000016 0000000000000016 00007ffebc039964 : engine13!firebird_plugin+0xf5514 00000000703edd70 00007ffebbe72439 : 00000000703eef90 00000000703eef90 00000000703eef90 00000000703edfb0 : engine13!firebird_plugin+0xf55df 00000000703edeb0 00007ffebbe7309d : 00000000703ee750 000000002b720e50 000000002b72ec80 000000002b89d670 : engine13!firebird_plugin+0x1032d9 00000000703ee650 00007ffebbe70065 : 00000000703eef90 00000000703eef90 000000003488c3b8 000000002b720e50 : engine13!firebird_plugin+0x103f3d 00000000703ee8d0 00007ffebbe68065 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : engine13!firebird_plugin+0x100f05 00000000703eeb20 00007ffebbe6488c : 0000000000000000 0000000035fb1830 0000000002098fa0 0000000000000000 : engine13!firebird_plugin+0xf8f05 00000000703efd50 00007ffebbdf8dd1 : 0000000000000001 00007ffebbe9e99d 0000000000000001 00000000000695f0 : engine13!firebird_plugin+0xf572c 00000000703efe60 00007ffebbeb3da5 : 0000000022163b40 0000000035fb1830 0000000022163b40 0000000000000000 : engine13!firebird_plugin+0x89c71 00000000703efec0 00007ffecd68268a : 000000004ef86ef0 000000002d564f20 0000000000000000 0000000000000000 : engine13!firebird_plugin+0x144c45 00000000703eff00 00007ffed04d7ac4 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x3a 00000000703eff30 00007ffed0b8a4e1 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0x14 00000000703eff60 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x21

SYMBOL_NAME: ucrtbase!abort+4e

MODULE_NAME: ucrtbase

IMAGE_NAME: ucrtbase.dll

STACK_COMMAND: ~14s ; .cxr ; kb

FAILURE_BUCKET_ID: FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort

OS_VERSION: 10.0.17763.1

BUILDLAB_STR: rs5_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

IMAGE_VERSION: 10.0.17763.1490

FAILURE_ID_HASH: {e31753ac-c98a-8055-3663-47e707543d20}

Followup: MachineOwner

NilsBoedeker avatar Apr 02 '24 15:04 NilsBoedeker

Moved crash analysis under the spoiler.

hvlad avatar Apr 03 '24 07:04 hvlad

Coud you provide a crash dump (D:\tmp\firebird.exe.5968.dmp), please ?

hvlad avatar Apr 03 '24 07:04 hvlad

Hi

as RAR File it have a size of 1.6 GB (uncompressed 20 GB). I can't upload it hier

I send it bei WE Transfer to [email protected]. Please confirm if you can load it

Nils

NilsBoedeker avatar Apr 03 '24 10:04 NilsBoedeker

Sure, downloading. Thanks

hvlad avatar Apr 03 '24 11:04 hvlad

If you need the database or the installation copy of firebird, please ask

NilsBoedeker avatar Apr 03 '24 12:04 NilsBoedeker

I an second crash dump from today (3.5 GB), same Error-Message. If you need this crash dump also please give me a message

NilsBoedeker avatar Apr 03 '24 15:04 NilsBoedeker

Yes, let me look at it too.

hvlad avatar Apr 04 '24 10:04 hvlad

I send:

a) the second dump b) the installation dir of firebird

in 2 splitted rar files via WE transfer.

Do you also need the database?

Nils

NilsBoedeker avatar Apr 04 '24 14:04 NilsBoedeker

Both dumps shows almost identical problem with blobs in fbclient.dll. It is not clear what exactly happens, unfortunately.

Do you also need the database?

Yes, please, if it helps to reproduse the issue.

hvlad avatar Apr 04 '24 16:04 hvlad

On the way... 2 splitted files via wetransfer.

Sometimes I got on client site the error:

invalid BLOB handle

before crash.

I check the database for errors, but there a no errors:

NilsBoedeker avatar Apr 08 '24 10:04 NilsBoedeker

Downloading, thanks. Is it enough to backup database to get the crash ? As for "invalid BLOB handle" - it is definitely related but should not crash the Firebird.

hvlad avatar Apr 08 '24 10:04 hvlad

Is it enough to backup database to get the crash ?

No, the error don‘t happed everytime on the live system and never on my test system.

It only happens in multi session use and more often when heavy losd is on the server.

It also happend without running backup, but more often during backup because this means more load.

NilsBoedeker avatar Apr 08 '24 12:04 NilsBoedeker

Did backup many times in a different ways, no luck - it doesn't crashes.

hvlad avatar Apr 08 '24 15:04 hvlad

This error only occurs under load on our live system.

The database is 24/7 in use and has permanent 125 and 200 connections and 300 and 400 active statements.

On average, fbserver.exe crashes once every 24 hours. often during backup process (i assume more load).

This is very difficult to simulate. Have you any idea how we can log this crash better?

NilsBoedeker avatar Apr 09 '24 06:04 NilsBoedeker

Perhaps this link can help: https://ib-aid.com/articles/how-to-configure-firebird-dump-collection-at-windows/

aafemt avatar Apr 09 '24 09:04 aafemt

@aafemt: @NilsBoedeker already provides me with dumps, why do you post this ?

hvlad avatar Apr 09 '24 09:04 hvlad

I'm sorry, I missed that you already have the dump when read that you are unable to reproduce the issue using backup.

aafemt avatar Apr 09 '24 09:04 aafemt