speeds up tablet mgmt iterator

[keith-turner]

Based on profiling tablet servers while running SplitMillionIT, made the following changes related to TabletManagementIterator

• Made TabletMgmtIterator extend WholeRowIterator instead of stack on top of it. This avoids encoding->decoding->encoding rows on the iterator stack.
• Avoided lookuping and parsing multiple table props for each tablet inorder to determine if it needs split. Moved this be done once per table.
• Changed a stream to compute file size sum into a loop.

[cshannon]

Did you compare the test time before and after the changes? Just curious on the performance difference.

[dlmarion]

Changed a stream to compute file size sum into a loop.

This is the Nth time where we are undo-ing stream operations for performance reasons...

[cshannon]

I just noticed #4563 , so really for a valid test for the time different should likely test after merging in both these changes and those changes.

[keith-turner]

Did you compare the test time before and after the changes? Just curious on the performance difference.

Yeah I did see a significant performance increase for scanning 1 million tablets. Sampling the logs where the manager logs how long a full scan of the metadata table took saw it was about 1.3 to 1.4 times faster. Suspect the change to wholerow iterator contributes most of that, but not sure. For each individual change the way I know it has an impact is that the code changed will be much less prominent in the profiling data. So I don't have timing for each thing, just see it drop off in the profiling data. Once something drops off in the profiling data, then other things will start to show up as more prominent.

[dlmarion]

We should probably run a full IT build on this PR before merging.

[keith-turner]

Without the changes from this PR and #4563 seeing SplitMillionIT take a total of 11 min 8 sec (upped the timeout). Below are some highlights from the logs.

2024-05-17T11:16:28,184 32 [functional.SplitMillionIT] INFO : Time to compact all tablets : 177606ms
2024-05-17T11:16:28,297 32 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 113ms
2024-05-17T11:17:04,720 32 [functional.SplitMillionIT] INFO : Time to clone table : 36422ms
2024-05-17T11:17:05,433 32 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 100 ondemand tablets for table id 2.
2024-05-17T11:17:27,405 32 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 22685ms
2024-05-17T11:20:35,323 32 [functional.SplitMillionIT] INFO : Time to merge all tablets : 187917ms
2024-05-17T11:20:36,123 745 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 1 ondemand tablets for table id 2.
2024-05-17T11:20:41,610 32 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 6287ms
2024-05-17T11:22:13,490 32 [functional.SplitMillionIT] INFO : Time to delete table : 91880ms

With the changes from this PR and #4563 seeing SplitMillionIT take a total of 9 min 44 sec. Its squeaking by under the 10 min threshold. Still seems like it used to be faster when running it locally. Below are some highlights. See noticeable improvements in compact and delete table, but not merge. I think merge is what I need to look into next. Also going to open an issue about improving the time to host ondemand tablets based on some thing I have seen. Below and above where it request hosting of 100 tablets and takes 10s of seconds that should be much much faster. Part of the reason that was slow was because of how long it takes to scan all of the metadata, which is what lead to opening this PR. However more could be done to improve on demand hosting.

2024-05-17T11:41:05,680 34 [functional.SplitMillionIT] INFO : Time to compact all tablets : 148425ms
2024-05-17T11:41:05,817 34 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 137ms
2024-05-17T11:41:32,089 34 [functional.SplitMillionIT] INFO : Time to clone table : 26271ms
2024-05-17T11:41:32,813 34 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 100 ondemand tablets for table id 2.
2024-05-17T11:41:45,478 34 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 13389ms
2024-05-17T11:44:49,188 34 [functional.SplitMillionIT] INFO : Time to merge all tablets : 183710ms
2024-05-17T11:44:49,445 747 [clientImpl.ClientTabletCacheImpl] DEBUG: Requesting hosting for 1 ondemand tablets for table id 2.
2024-05-17T11:44:50,221 34 [functional.SplitMillionIT] INFO : Time to verify 200 rows was 1033ms
2024-05-17T11:46:08,001 34 [functional.SplitMillionIT] INFO : Time to delete table : 77780ms

[dlmarion]

I wonder if there is a way to make Tablet hosting faster

[keith-turner]

I wonder if there is a way to make Tablet hosting faster

Have a possible way, not complete sure about it. Going to open an issue

[dlmarion]

MANAGER_TABLET_GROUP_WATCHER_INTERVAL defaults to 5s in MiniAccumuloClusterImpl. Can override in the test.

[keith-turner]

@dlmarion opened #4571. In that issue propose a possible way to speed up tablet hosting by replacing the requestToHost column w/ future and directly calling balancer and sending tablet server load tablet RPCs. Also in that issue documented how I was seeing the fallout from cloning 1 million tablets impact the hosting request because it tied up TGW. So was hoping we could completely decouple the hosting request from TGW inorder to speed it up. Realized setting future still allows the request to be fault tolerant.

[keith-turner]

Ran a full build with these changes and it was ok, some test failed but those were already failing before this change.