wiki icon indicating copy to clipboard operation
wiki copied to clipboard

feat: save data into pageTree table after processing a batch of pages

Open praveen-kumar85 opened this issue 1 year ago • 1 comments

While re-building a page tree, all the existing page details are fetched from the database and a tree array is created at the application side. Once the tree is created, the pageTree table is truncated & the objects in the tree array are saved to the pageTree table in chunks. When there are a lot of records present in the page table, it takes a lot of time to build the tree array at the application side causing the database connections to timeout and it is causing the rebuild-tree job to fail.

In order to avoid idle connections, data should be saved right after enough number of records are added to the tree array.

praveen-kumar85 avatar Mar 10 '23 08:03 praveen-kumar85

The downside of this approach is that the pageTree table will be empty or incomplete during the rebuild. Users will experience empty or broken navigation if the rebuild takes a long time to complete.

The worker maintains a pool of DB connections and this isn't a transaction. So whether the connection times out shouldn't matter in this case. A new one will be established automatically if needed.

NGPixel avatar Mar 17 '23 22:03 NGPixel