firebird icon indicating copy to clipboard operation
firebird copied to clipboard
verified publisher icon FirebirdSQL

Quering MON$ table failes and causes 100% CPU

Open 1Hace opened this issue 3 years ago • 10 comments

Hi All,

The error seems to start when a MON$ table was queried. Several times there weren’t any issues when opening a MON$ table. After the first time the failure occured, everytime a MON$ table was queried it will 'hang'.

  • Connected users will increase when a client connects, and the user stays connected even when the application (client) was terminated (killed)
  • There is no commit anymore, transactions increasing (difference between oldest transaction and Next transaction increases) (Our application executes about 300 select queries per second.)
  • Firebird uses 8% cpu continuously (12 core system, so 1 core to the max?), 45mb memory
  • Our tables and RDB$ can be queried, when a MON$ table is queried, the application will hang (isql, IBOConsole, FlameRobin)
  • Last modified date/time of the FDB file doesn’t change (anymore?), but data is saved to the file. (a copy of the fdb will have some changed data inside)
  • The generation number also increases.
  • It seems that there is no sweep executed. (didn’t try to start it manually)
  • When a query was executed, a new thread was created. And this thread keeps running for while, even when the application which exectued the query was killed (the application hangs because a query on a MON$ table was executed)
  • A customer had the same sort of issue, but even normal Select queries didn't execute anymore.
  • I have added some debug information below.

Server details: Firebird 4.0.2 SuperServer

Database header page information: Flags 0 Generation 129095599 System Change Number 0 Page size 16384 ODS version 13.0 Oldest transaction 58157208 Oldest active 58157209 Oldest snapshot 58157208 Next transaction 128834245 Sequence number 0 Next attachment ID 258428 Implementation HW=AMD/Intel/x64 little-endian OS=Windows CC=MSVC Shadow count 0 Page buffers 0 Next header page 0 Database dialect 3 Creation date Apr 12, 2022 5:55:38 Attributes force write

It is running for more than 3 days in this error situation. If there is anything I can test, please let me know.

Regards & thanks, Hans

Attatched WinDBG: User Mode Time Thread Time 12:209c 3 days 17:10:51.718 2:1f48 0 days 3:26:44.625 33:17ac 0 days 0:03:06.906 32:1170 0 days 0:02:17.218 20:1f20 0 days 0:01:43.390

Thread 12:

Child-SP RetAddr Call Site

00 000000003a9ae5c0 00007ffd3e93f045 Engine13!Jrd::Attachment::deletePool+0xb4 [z:\fb40\v4.0.2\src\jrd\attachment.cpp @ 173] 01 (Inline Function) ---------------- Engine13!Jrd::dsql_dbb::deletePool+0x19 [z:\fb40\v4.0.2\src\dsql\dsql.h @ 165] 02 000000003a9ae5f0 00007ffd3e9382c3 Engine13!prepareStatement+0xbb5 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 1644] 03 000000003a9aeb00 00007ffd3ea93968 Engine13!DSQL_prepare+0x93 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 413] 04 000000003a9aebb0 00007ffd3ea815c0 Engine13!Jrd::JAttachment::prepare+0x248 [z:\fb40\v4.0.2\src\jrd\jrd.cpp @ 5609] 05 000000003a9aeee0 00007ffd5aa33b41 Engine13!Firebird::IAttachmentBaseImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IAttachment> > > > >::cloopprepareDispatcher+0x70 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 11176] 06 (Inline Function) ---------------- fbclient!Firebird::IAttachment::prepare+0x4d [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 2532] 07 000000003a9aef60 00007ffd5aa27770 fbclient!Why::YAttachment::prepare+0xc1 [z:\fb40\v4.0.2\src\yvalve\why.cpp @ 5519] 08 000000003a9af000 000000014002078f fbclient!Firebird::IAttachmentBaseImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IAttachment > > > >::cloopprepareDispatcher+0x70 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 11176] 09 (Inline Function) ---------------- firebird!Firebird::IAttachment::prepare+0x43 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 2532] 0a 000000003a9af080 0000000140020ffc firebird!rem_port::prepare_statement+0x36f [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 4682] 0b 000000003a9afb60 000000014001f66b firebird!process_packet+0x4fc [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 4988] 0c 000000003a9afe00 000000014003dd35 firebird!loopThread+0x1bb [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 6534] 0d (Inline Function) ---------------- firebird!anonymous-namespace'::ThreadArgs::run+0x5 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 78] 0e 000000003a9afef0 00007ffd7f12cab0 firebird!threadStart+0x65 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 97] 0f 000000003a9aff30 00007ffd80f68364 ucrtbase!o__realloc_base+0x60 10 000000003a9aff60 00007ffd81d05e91 KERNEL32!BaseThreadInitThunk+0x14 11 000000003a9aff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Thread 2:

Child-SP RetAddr Call Site

00 (Inline Function) ---------------- firebird!Firebird::LinkedList::getElement+0xb [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 1566] 01 (Inline Function) ---------------- firebird!Firebird::FreeObjectsFirebird::LinkedList,Firebird::LowLimits::allocateBlock+0x5c [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 1659] 02 00000000025df570 000000014002ced2 firebird!Firebird::MemPool::alloc+0xb6 [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 2245] 03 00000000025df5e0 000000014002cf19 firebird!Firebird::MemPool::allocate2+0x32 [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 2301] 04 00000000025df610 00000001400121db firebird!Firebird::MemPool::allocate+0x19 [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 2336] 05 (Inline Function) ---------------- firebird!operator new+0xa [z:\fb40\v4.0.2\src\common\classes\alloc.h @ 366] 06 (Inline Function) ---------------- firebird!Firebird::ObjectsArray<Firebird::Array<char,Firebird::EmptyStorage >,Firebird::Array<Firebird::Array<char,Firebird::EmptyStorage > *,Firebird::InlineStorage<Firebird::Array<char,Firebird::EmptyStorage > *,8,Firebird::Array<char,Firebird::EmptyStorage > *> > >::add+0x11 [z:\fb40\v4.0.2\src\common\classes\objects_array.h @ 205] 07 00000000025df640 000000014000bb37 firebird!SRVR_multi_thread+0x35b [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 1656] 08 00000000025dfe90 000000014003dd35 firebird!inet_connect_wait_thread+0x77 [z:\fb40\v4.0.2\src\remote\server\os\win32\srvr_w32.cpp @ 413] 09 (Inline Function) ---------------- firebird!anonymous-namespace'::ThreadArgs::run+0x5 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 78] 0a 00000000025dfef0 00007ffd7f12cab0 firebird!threadStart+0x65 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 97] 0b 00000000025dff30 00007ffd80f68364 ucrtbase!o__realloc_base+0x60 0c 00000000025dff60 00007ffd81d05e91 KERNEL32!BaseThreadInitThunk+0x14 0d 00000000025dff90 0000000000000000 ntdll!RtlUserThreadStart+0x21

A lot of threads have the same call stack as Thread 9 Thread 9:

Child-SP RetAddr Call Site

00 000000005550d9e8 00007ffd7e3d75ff ntdll!ZwWaitForSingleObject+0x14 01 000000005550d9f0 00007ffd3ebe851c KERNELBASE!WaitForSingleObjectEx+0x8f 02 000000005550da90 00007ffd3eb7f747 Engine13!Firebird::SharedMemoryBase::eventWait+0x5c [z:\fb40\v4.0.2\src\common\isc_sync.cpp @ 801] 03 000000005550dac0 00007ffd3eb7c91f Engine13!Jrd::LockManager::wait_for_request+0x417 [z:\fb40\v4.0.2\src\lock\lock.cpp @ 3833] 04 000000005550dbb0 00007ffd3eb7c1df Engine13!Jrd::LockManager::grant_or_que+0xaf [z:\fb40\v4.0.2\src\lock\lock.cpp @ 2207] 05 000000005550dbf0 00007ffd3eaa1345 Engine13!Jrd::LockManager::enqueue+0x2cf [z:\fb40\v4.0.2\src\lock\lock.cpp @ 532] 06 000000005550dc70 00007ffd3eaa0dab Engine13!enqueue+0xa5 [z:\fb40\v4.0.2\src\jrd\lck.cpp @ 947] 07 (Inline Function) ---------------- Engine13!ENQUEUE+0x34 [z:\fb40\v4.0.2\src\jrd\lck.cpp @ 149] 08 000000005550dcf0 00007ffd3eab1955 Engine13!LCK_lock+0x13b [z:\fb40\v4.0.2\src\jrd\lck.cpp @ 676] 09 000000005550de60 00007ffd3eab3075 Engine13!Jrd::MonitoringSnapshot::MonitoringSnapshot+0x365 [z:\fb40\v4.0.2\src\jrd\monitoring.cpp @ 495] 0a 000000005550ed70 00007ffd3eab39d1 Engine13!Jrd::MonitoringSnapshot::create+0x65 [z:\fb40\v4.0.2\src\jrd\monitoring.cpp @ 423] 0b 000000005550edb0 00007ffd3eaf9e21 Engine13!Jrd::MonitoringTableScan::getFormat+0x11 [z:\fb40\v4.0.2\src\jrd\monitoring.cpp @ 109] 0c 000000005550ede0 00007ffd3e9c6ba5 Engine13!Jrd::VirtualTableScan::open+0x91 [z:\fb40\v4.0.2\src\jrd\recsrc\virtualtablescan.cpp @ 57] 0d 000000005550ee10 00007ffd3ea3093f Engine13!Jrd::ForNode::execute+0x95 [z:\fb40\v4.0.2\src\dsql\stmtnodes.cpp @ 5099] 0e 000000005550ee50 00007ffd3ea32274 Engine13!EXE_looper+0x19f [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 1389] 0f 000000005550ef30 00007ffd3ea31fab Engine13!looper_seh+0x34 [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 1497] 10 000000005550ef90 00007ffd3ea314ee Engine13!execute_looper+0xfb [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 1068] 11 000000005550eff0 00007ffd3ea7c07b Engine13!EXE_start+0x24e [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 934] 12 000000005550f090 00007ffd3e93a6b0 Engine13!JRD_start+0x2b [z:\fb40\v4.0.2\src\jrd\jrd.cpp @ 9238] 13 000000005550f0c0 00007ffd3e93b6a0 Engine13!Jrd::DsqlDmlRequest::doExecute+0x70 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 713] 14 000000005550f2d0 00007ffd3e9381cf Engine13!Jrd::DsqlDmlRequest::execute+0x1d0 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 868] 15 000000005550f390 00007ffd3ea932d5 Engine13!DSQL_open+0x38f [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 231] 16 000000005550f450 00007ffd3ea814d2 Engine13!Jrd::JStatement::openCursor+0x115 [z:\fb40\v4.0.2\src\jrd\jrd.cpp @ 5118] 17 000000005550f740 00007ffd5aa331c6 Engine13!Firebird::IStatementBaseImpl<Jrd::JStatement,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JStatement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JStatement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IStatement> > > > >::cloopopenCursorDispatcher+0x72 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 10029] 18 (Inline Function) ---------------- fbclient!Firebird::IStatement::openCursor+0x4f [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 1898] 19 000000005550f7c0 00007ffd5aa2769d fbclient!Why::YStatement::openCursor+0xc6 [z:\fb40\v4.0.2\src\yvalve\why.cpp @ 4436] 1a 000000005550f880 000000014001b942 fbclient!Firebird::IStatementBaseImpl<Why::YStatement,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YStatement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YStatement,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IStatement > > > >::cloopopenCursorDispatcher+0x6d [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 10029] 1b (Inline Function) ---------------- firebird!Firebird::IStatement::openCursor+0x42 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 1898] 1c 000000005550f900 0000000140020fa0 firebird!rem_port::execute_statement+0x522 [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 3869] 1d 000000005550fb60 000000014001f66b firebird!process_packet+0x4a0 [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 4971] 1e 000000005550fe00 000000014003dd35 firebird!loopThread+0x1bb [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 6534] 1f (Inline Function) ---------------- firebird!anonymous-namespace'::ThreadArgs::run+0x5 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 78] 20 000000005550fef0 00007ffd7f12cab0 firebird!threadStart+0x65 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 97] 21 000000005550ff30 00007ffd80f68364 ucrtbase!o__realloc_base+0x60 22 000000005550ff60 00007ffd81d05e91 KERNEL32!BaseThreadInitThunk+0x14 23 000000005550ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

1Hace avatar Oct 04 '22 12:10 1Hace

A lot of threads have the same call stack as Thread 9

It means that a lot of user sessions query MON$ tables. Why???

dyemanov avatar Oct 04 '22 14:10 dyemanov

Hi!

We have the same issue on FB3.0.10.

A few month ago we started to migrate from 2.5.9 to 3.0.10. Hundreds of databases were converted. There are many more in progress. Since then 5 or 6 times the mon$ table select hangs on different databases. It is a very rare select only run once when the our program starting and counting the number of the concurent users to check the program’s license.

We have tried to reproduce it, but so far we have not succeeded.

3.0.10 classic on linux.

Everything else works except just we can't query from mon$ tables.

As soon as we reach the affected connection by killing (kill -9 pid) the processes one by one, the problem is solved and the query of the mon$ tables works again. (It is not necessary to kill all processes from the given database. If there is another transaction that querys the mon$ table, it will run immediately, waking up from the frozen state.)

According to our experience, it does not depend on the database load, it has already happened with 2-3 database connections and even with 100 connections.

András

From: 1Hace @.> Sent: Tuesday, October 4, 2022 2:39 PM To: FirebirdSQL/firebird @.> Cc: Subscribed @.***> Subject: [FirebirdSQL/firebird] Quering MON$ table failes and causes 100% CPU (Issue #7328)

Hi All,

The error seems to start when a MON$ table was queried. Several times there weren’t any issues when opening a MON$ table. After the first time the failure occured, everytime a MON$ table was queried it will 'hang'.

  • Connected users will increase when a client connects, and the user stays connected even when the application (client) was terminated (killed)
  • There is no commit anymore, transactions increasing (difference between oldest transaction and Next transaction increases) (Our application executes about 300 select queries per second.)
  • Firebird uses 8% cpu continuously (12 core system, so 1 core to the max?), 45mb memory
  • Our tables and RDB$ can be queried, when a MON$ table is queried, the application will hang (isql, IBOConsole, FlameRobin)
  • Last modified date/time of the FDB file doesn’t change (anymore?), but data is saved to the file. (a copy of the fdb will have some changed data inside)
  • The generation number also increases.
  • It seems that there is no sweep executed. (didn’t try to start it manually)
  • When a query was executed, a new thread was created. And this thread keeps running for while, even when the application which exectued the query was killed (the application hangs because a query on a MON$ table was executed)
  • A customer had the same sort of issue, but even normal Select queries didn't execute anymore.
  • I have added some debug information below.

Server details: Firebird 4.0.2 SuperServer

Database header page information: Flags 0 Generation 129095599 System Change Number 0 Page size 16384 ODS version 13.0 Oldest transaction 58157208 Oldest active 58157209 Oldest snapshot 58157208 Next transaction 128834245 Sequence number 0 Next attachment ID 258428 Implementation HW=AMD/Intel/x64 little-endian OS=Windows CC=MSVC Shadow count 0 Page buffers 0 Next header page 0 Database dialect 3 Creation date Apr 12, 2022 5:55:38 Attributes force write

It is running for more than 3 days in this error situation. If there is anything I can test, please let me know.

Regards & thanks, Hans

Attatched WinDBG: User Mode Time Thread Time 12:209c 3 days 17:10:51.718 2:1f48 0 days 3:26:44.625 33:17ac 0 days 0:03:06.906 32:1170 0 days 0:02:17.218 20:1f20 0 days 0:01:43.390

Thread 12:

Child-SP RetAddr Call Site

00 000000003a9ae5c0 00007ffd3e93f045 Engine13!Jrd::Attachment::deletePool+0xb4 [z:\fb40\v4.0.2\src\jrd\attachment.cpp @ 173] 01 (Inline Function) ---------------- Engine13!Jrd::dsql_dbb::deletePool+0x19 [z:\fb40\v4.0.2\src\dsql\dsql.h @ 165] 02 000000003a9ae5f0 00007ffd3e9382c3 Engine13!prepareStatement+0xbb5 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 1644] 03 000000003a9aeb00 00007ffd3ea93968 Engine13!DSQL_prepare+0x93 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 413] 04 000000003a9aebb0 00007ffd3ea815c0 Engine13!Jrd::JAttachment::prepare+0x248 [z:\fb40\v4.0.2\src\jrd\jrd.cpp @ 5609] 05 000000003a9aeee0 00007ffd5aa33b41 Engine13!Firebird::IAttachmentBaseImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IAttachment > > > >::cloopprepareDispatcher+0x70 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 11176] 06 (Inline Function) ---------------- fbclient!Firebird::IAttachment::prepare+0x4d [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 2532] 07 000000003a9aef60 00007ffd5aa27770 fbclient!Why::YAttachment::prepare+0xc1 [z:\fb40\v4.0.2\src\yvalve\why.cpp @ 5519] 08 000000003a9af000 000000014002078f fbclient!Firebird::IAttachmentBaseImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IAttachment > > > >::cloopprepareDispatcher+0x70 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 11176] 09 (Inline Function) ---------------- firebird!Firebird::IAttachment::prepare+0x43 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 2532] 0a 000000003a9af080 0000000140020ffc firebird!rem_port::prepare_statement+0x36f [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 4682] 0b 000000003a9afb60 000000014001f66b firebird!process_packet+0x4fc [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 4988] 0c 000000003a9afe00 000000014003dd35 firebird!loopThread+0x1bb [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 6534] 0d (Inline Function) ---------------- firebird!anonymous-namespace'::ThreadArgs::run+0x5 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 78] 0e 000000003a9afef0 00007ffd7f12cab0 firebird!threadStart+0x65 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 97] 0f 000000003a9aff30 00007ffd80f68364 ucrtbase!o__realloc_base+0x60 10 000000003a9aff60 00007ffd81d05e91 KERNEL32!BaseThreadInitThunk+0x14 11 000000003a9aff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Thread 2:

Child-SP RetAddr Call Site

00 (Inline Function) ---------------- firebird!Firebird::LinkedList::getElement+0xb [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 1566] 01 (Inline Function) ---------------- firebird!Firebird::FreeObjectsFirebird::LinkedList,Firebird::LowLimits::allocateBlock+0x5c [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 1659] 02 00000000025df570 000000014002ced2 firebird!Firebird::MemPool::alloc+0xb6 [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 2245] 03 00000000025df5e0 000000014002cf19 firebird!Firebird::MemPool::allocate2+0x32 [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 2301] 04 00000000025df610 00000001400121db firebird!Firebird::MemPool::allocate+0x19 [z:\fb40\v4.0.2\src\common\classes\alloc.cpp @ 2336] 05 (Inline Function) ---------------- firebird!operator new+0xa [z:\fb40\v4.0.2\src\common\classes\alloc.h @ 366] 06 (Inline Function) ---------------- firebird!Firebird::ObjectsArray<Firebird::Array<char,Firebird::EmptyStorage >,Firebird::Array<Firebird::Array<char,Firebird::EmptyStorage > *,Firebird::InlineStorage<Firebird::Array<char,Firebird::EmptyStorage > *,8,Firebird::Array<char,Firebird::EmptyStorage > *> > >::add+0x11 [z:\fb40\v4.0.2\src\common\classes\objects_array.h @ 205] 07 00000000025df640 000000014000bb37 firebird!SRVR_multi_thread+0x35b [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 1656] 08 00000000025dfe90 000000014003dd35 firebird!inet_connect_wait_thread+0x77 [z:\fb40\v4.0.2\src\remote\server\os\win32\srvr_w32.cpp @ 413] 09 (Inline Function) ---------------- firebird!anonymous-namespace'::ThreadArgs::run+0x5 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 78] 0a 00000000025dfef0 00007ffd7f12cab0 firebird!threadStart+0x65 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 97] 0b 00000000025dff30 00007ffd80f68364 ucrtbase!o__realloc_base+0x60 0c 00000000025dff60 00007ffd81d05e91 KERNEL32!BaseThreadInitThunk+0x14 0d 00000000025dff90 0000000000000000 ntdll!RtlUserThreadStart+0x21

A lot of threads have the same call stack as Thread 9 Thread 9:

Child-SP RetAddr Call Site

00 000000005550d9e8 00007ffd7e3d75ff ntdll!ZwWaitForSingleObject+0x14 01 000000005550d9f0 00007ffd3ebe851c KERNELBASE!WaitForSingleObjectEx+0x8f 02 000000005550da90 00007ffd3eb7f747 Engine13!Firebird::SharedMemoryBase::eventWait+0x5c [z:\fb40\v4.0.2\src\common\isc_sync.cpp @ 801] 03 000000005550dac0 00007ffd3eb7c91f Engine13!Jrd::LockManager::wait_for_request+0x417 [z:\fb40\v4.0.2\src\lock\lock.cpp @ 3833] 04 000000005550dbb0 00007ffd3eb7c1df Engine13!Jrd::LockManager::grant_or_que+0xaf [z:\fb40\v4.0.2\src\lock\lock.cpp @ 2207] 05 000000005550dbf0 00007ffd3eaa1345 Engine13!Jrd::LockManager::enqueue+0x2cf [z:\fb40\v4.0.2\src\lock\lock.cpp @ 532] 06 000000005550dc70 00007ffd3eaa0dab Engine13!enqueue+0xa5 [z:\fb40\v4.0.2\src\jrd\lck.cpp @ 947] 07 (Inline Function) ---------------- Engine13!ENQUEUE+0x34 [z:\fb40\v4.0.2\src\jrd\lck.cpp @ 149] 08 000000005550dcf0 00007ffd3eab1955 Engine13!LCK_lock+0x13b [z:\fb40\v4.0.2\src\jrd\lck.cpp @ 676] 09 000000005550de60 00007ffd3eab3075 Engine13!Jrd::MonitoringSnapshot::MonitoringSnapshot+0x365 [z:\fb40\v4.0.2\src\jrd\monitoring.cpp @ 495] 0a 000000005550ed70 00007ffd3eab39d1 Engine13!Jrd::MonitoringSnapshot::create+0x65 [z:\fb40\v4.0.2\src\jrd\monitoring.cpp @ 423] 0b 000000005550edb0 00007ffd3eaf9e21 Engine13!Jrd::MonitoringTableScan::getFormat+0x11 [z:\fb40\v4.0.2\src\jrd\monitoring.cpp @ 109] 0c 000000005550ede0 00007ffd3e9c6ba5 Engine13!Jrd::VirtualTableScan::open+0x91 [z:\fb40\v4.0.2\src\jrd\recsrc\virtualtablescan.cpp @ 57] 0d 000000005550ee10 00007ffd3ea3093f Engine13!Jrd::ForNode::execute+0x95 [z:\fb40\v4.0.2\src\dsql\stmtnodes.cpp @ 5099] 0e 000000005550ee50 00007ffd3ea32274 Engine13!EXE_looper+0x19f [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 1389] 0f 000000005550ef30 00007ffd3ea31fab Engine13!looper_seh+0x34 [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 1497] 10 000000005550ef90 00007ffd3ea314ee Engine13!execute_looper+0xfb [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 1068] 11 000000005550eff0 00007ffd3ea7c07b Engine13!EXE_start+0x24e [z:\fb40\v4.0.2\src\jrd\exe.cpp @ 934] 12 000000005550f090 00007ffd3e93a6b0 Engine13!JRD_start+0x2b [z:\fb40\v4.0.2\src\jrd\jrd.cpp @ 9238] 13 000000005550f0c0 00007ffd3e93b6a0 Engine13!Jrd::DsqlDmlRequest::doExecute+0x70 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 713] 14 000000005550f2d0 00007ffd3e9381cf Engine13!Jrd::DsqlDmlRequest::execute+0x1d0 [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 868] 15 000000005550f390 00007ffd3ea932d5 Engine13!DSQL_open+0x38f [z:\fb40\v4.0.2\src\dsql\dsql.cpp @ 231] 16 000000005550f450 00007ffd3ea814d2 Engine13!Jrd::JStatement::openCursor+0x115 [z:\fb40\v4.0.2\src\jrd\jrd.cpp @ 5118] 17 000000005550f740 00007ffd5aa331c6 Engine13!Firebird::IStatementBaseImpl<Jrd::JStatement,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JStatement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JStatement,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IStatement > > > >::cloopopenCursorDispatcher+0x72 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 10029] 18 (Inline Function) ---------------- fbclient!Firebird::IStatement::openCursor+0x4f [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 1898] 19 000000005550f7c0 00007ffd5aa2769d fbclient!Why::YStatement::openCursor+0xc6 [z:\fb40\v4.0.2\src\yvalve\why.cpp @ 4436] 1a 000000005550f880 000000014001b942 fbclient!Firebird::IStatementBaseImpl<Why::YStatement,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YStatement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YStatement,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IStatement > > > >::cloopopenCursorDispatcher+0x6d [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 10029] 1b (Inline Function) ---------------- firebird!Firebird::IStatement::openCursor+0x42 [z:\fb40\v4.0.2\src\include\firebird\idlfbinterfaces.h @ 1898] 1c 000000005550f900 0000000140020fa0 firebird!rem_port::execute_statement+0x522 [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 3869] 1d 000000005550fb60 000000014001f66b firebird!process_packet+0x4a0 [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 4971] 1e 000000005550fe00 000000014003dd35 firebird!loopThread+0x1bb [z:\fb40\v4.0.2\src\remote\server\server.cpp @ 6534] 1f (Inline Function) ---------------- firebird!anonymous-namespace'::ThreadArgs::run+0x5 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 78] 20 000000005550fef0 00007ffd7f12cab0 firebird!threadStart+0x65 [z:\fb40\v4.0.2\src\common\threadstart.cpp @ 97] 21 000000005550ff30 00007ffd80f68364 ucrtbase!o__realloc_base+0x60 22 000000005550ff60 00007ffd81d05e91 KERNEL32!BaseThreadInitThunk+0x14 23 000000005550ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

— Reply to this email directly, view it on GitHubhttps://github.com/FirebirdSQL/firebird/issues/7328, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOZGLFDMGM4PHVCKDTD6AS3WBQQMVANCNFSM6AAAAAAQ4RBL6U. You are receiving this because you are subscribed to this thread.Message ID: @.***>

omachtandras avatar Oct 04 '22 18:10 omachtandras

András, looks like #5447 is going to solve exactly your case ;-) Could you please comment there whether the discussed solutions are appropriate for your task.

dyemanov avatar Oct 05 '22 04:10 dyemanov

A lot of threads have the same call stack as Thread 9

It means that a lot of user sessions query MON$ tables. Why???

Our application (multiple instances) uses MON$DATABASE to check whether the database is in read-only or not (for replication). (And while in the error-state I used FlameRobin to query some MON$tables)

SQL: Select "MON$REPLICA_MODE" AS READONLY From "MON$DATABASE"; Transaction: Read commited, rec_version, nowait, read

1Hace avatar Oct 05 '22 05:10 1Hace

You'd better use RDB$GET_CONTEXT('SYSTEM', 'REPLICA_MODE') instead of querying MON$DATABASE.

dyemanov avatar Oct 05 '22 05:10 dyemanov

Thanks for the quick reply, we will test it! But the 'hang'-issue is related to quering MON$-tables?

1Hace avatar Oct 05 '22 05:10 1Hace

Yes, it's possible (at least your stack trace shows that). Looks like some user attachment does not respond to the interrupt generated when polling is performed. However, I cannot be sure whether it's Thread 12 in your dump or some thread you omitted actually has a different stack than Thread 9. So some problem may really exist and my suggestion re. RDB$GET_CONTEXT is actually a workaround, not a solution.

dyemanov avatar Oct 05 '22 07:10 dyemanov

If you or someone else is interested, I have attached the call stack of all threads. all_threads_call_stack.txt

1Hace avatar Oct 05 '22 12:10 1Hace

A side note: because it is an application even better would be using of fb_info_replica_mode saving start transaction and query prepare.

aafemt avatar Oct 05 '22 12:10 aafemt

Hi Dmitry,

unfortunately, it is not enough for us to know how many users are in the database.

There are technical processes that do not count towards the user number, and we also check that someone does not log in twice.

András

@.***http://www.libra.hu/ CÉGÜNK A LIBRA CSOPORT TAGJA

OMACHT ANDRÁS fejlesztési igazgató

LIBRA Szoftver Zrt. 1113 Budapest, Karolina út 65. +36 (1) 255-3939 @.@.> | www.libra.huhttp://www.libra.hu

From: Dmitry Yemanov @.> Sent: Wednesday, October 5, 2022 6:46 AM To: FirebirdSQL/firebird @.> Cc: Omacht András @.>; Comment @.> Subject: Re: [FirebirdSQL/firebird] Quering MON$ table failes and causes 100% CPU (Issue #7328)

András, looks like #5447https://github.com/FirebirdSQL/firebird/issues/5447 is going to solve exactly your case ;-) Could you please comment there whether the discussed solutions are appropriate for your task.

— Reply to this email directly, view it on GitHubhttps://github.com/FirebirdSQL/firebird/issues/7328#issuecomment-1267934561, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOZGLFF3Y4ZMHS2K7OB5TUTWBUBZRANCNFSM6AAAAAAQ4RBL6U. You are receiving this because you commented.Message ID: @.***>

omachtandras avatar Oct 07 '22 07:10 omachtandras