Massive thread creation on listening for events -> unable to create new native thread

[siglesiasg]

Hello,

I've found a memory / thread leak when listening to blockchain events, in this case is when trying to listen for block events. Too many threads are created and an error: java.lang.OutOfMemoryError: unable to create new native thread is throwed resulting in a lock when processing events.

How to replicate Case:

I create a new channel and I execute lots of transactions in the channel so a lot of new blocks are generated > 1000. Then I start listening to that channel, using network.addBlockListener(startBlock, blockConsumer); where startBlock is = 0 and blockConsumer is a simple consumer to print the BlockEvent. When I launch the app in debug mode I see that lots of threads are created and waiting for the previous block to be processed.

As many threads as available blocks are being created:

Ending in a java.lang.OutOfMemoryError: unable to create new native thread

[bestbeforetoday]

Thank you for the issue report and investigation. Is it possible that you are hitting a failure in allocating threads used to deliver events because you have blocked processing of the events with a break-point in your debugger, or is this something you observe in normal (non-debugging) usage? I'm pretty sure I've successfully replayed many tens of thousands of events in the past, but my event listener was very simple and processed events very quickly.

It sounds like there might be scope to improve the way incoming events are handled internally to avoid locking preventing those events from being buffered ready for dispatch to listeners here:

https://github.com/hyperledger/fabric-gateway-java/blob/26c69ebc3ca5f23d0a8ee5efa9ffbd962a78f54b/src/main/java/org/hyperledger/fabric/gateway/impl/event/OrderedBlockEventSource.java#L64-L73

A side-effect to consider is that, depending on the implementation, this may allow that event buffer to grow unbounded and eventually cause out of memory errors.

[siglesiasg]

Hello Mark, thanks for the reply.

We've already seen this issue in production environment and I just had replicated in localhost to check what was going on. At this point we are storing the block events in an SQL so we get the JSON, we parse and we write it on to the database in a transactional way, every block takes about 20/100ms to be executed (depending on tx number). Not really big number.

To solve this issue, we've managed to read ahead the blocks one by one until last one using queryBlockByNumber function and when we finish iterating all available blocks we delegate control to the real event listener. Tricky workaround but is working for now.

I think thread generation is below the method you've posted here but I still don't understand why is necessary to have one thread per block

Thanks

[bestbeforetoday]

I don't think it is strictly necessary to create a new thread to deliver each block, although I guess the intent was to deal with event delivery asynchronously and not hold up any other processing. I'm surprised the threads aren't being allocated from a bounded pool.

You're correct that the thread generation is further down the stack, within the Channel registerBlockListener() method (in fabric-sdk-java) called here:

https://github.com/hyperledger/fabric-gateway-java/blob/26c69ebc3ca5f23d0a8ee5efa9ffbd962a78f54b/src/main/java/org/hyperledger/fabric/gateway/impl/event/ChannelBlockEventSource.java#L40

It might be better if this overload was used instead as a mechanism for delivering events into the fabric-gateway-java buffer before being dispatched to listeners, which I hope would avoid the thread creation:

https://github.com/hyperledger/fabric-sdk-java/blob/87e99f63091db3bc946a2bdfc0c387d985d53e93/src/main/java/org/hyperledger/fabric/sdk/Channel.java#L5806

String registerBlockListener(BlockingQueue<QueuedBlockEvent> blockEventQueue)