service-api icon indicating copy to clipboard operation
service-api copied to clipboard

Deep merge takes too long

Open mrbinwin opened this issue 3 years ago • 0 comments

Hello!

service api v.5.5.0

I run a deep merge of 2 launches: launch 1: 6274 test_items launch 2: 3204 test_items

It takes 1750 sec (29 minutes) to merge them. It happens because of this query: https://github.com/reportportal/service-api/blob/develop/src/main/java/com/epam/ta/reportportal/core/item/identity/TestItemUniqueIdGenerator.java#L87

The merge script pulls from db all parents for each test_item of each of the merging launches. So in my case it sends 9478 database queries and each query pulls 3 entity. Most of the time this script pulls the same test_items (the same suite, story, parent step). Because of usage of findAllById() method Hibernate can not use any kind of cache to improve the performance of these queries.

I've added a simple HashMap cache to check if it can work faster.

Deep merge without caching: 9478 db queries 1750 sec

With cache: 366 db queries 366 item names cached 27690 cache hits 35 sec

Long merges can be a problem because during high load a merge procees locks other database queries and it can lead to API connection pool overflow and this error: Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

mrbinwin avatar Aug 17 '21 15:08 mrbinwin