Add iceberg load test

[ahmedabu98]

Adding a load test for IcebergIO

Separating integration test (from #31220) into its own suite

[github-actions[bot]]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 68.55%. Comparing base (e2d6246) to head (1c5b4f0). Report is 5 commits behind head on master.

:exclamation: Current head 1c5b4f0 differs from pull request most recent head 50b2488

Please upload reports for the commit 50b2488 to get more accurate results.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #31392       +/-   ##
=============================================
- Coverage     71.40%   68.55%    -2.86%     
- Complexity     1474    14921    +13447     
=============================================
  Files           900     2636     +1736     
  Lines        114166   222073   +107907     
  Branches       1076    11825    +10749     
=============================================
+ Hits          81519   152234    +70715     
- Misses        30619    63644    +33025     
- Partials       2028     6195     +4167     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[ahmedabu98]

Performance test ran successfully with the following jobs:

Write (2024-05-28_14_38_09-6308924367920992234):

• Wrote 1 billion records
• Autoscaled to 125 workers
• Total elapsed time: 18.5 min

Read (2024-05-28_14_56_50-10726183500998931365):

• Read 1 billion records
• Autoscaled to 42 workers
• Total elapsed time: 21 min

[ahmedabu98]

R: @kennknowles

[github-actions[bot]]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[ahmedabu98]

Looks like populating the data did have an effect. Pipelines used a lot more workers but finished in roughly the same amount of time.

Write (2024-06-17_18_21_14-2727804973398206918)

• Wrote 1 billion records (2 TB)
• Autoscaled to 373 workers
• Total elapsed time: ~59 min (Actual write finished in 14 min. Took a long time to clean up and stop worker pool)

Read (2024-06-17_19_20_46-7349681781857878664)

• Read 1 billion records (2 TB)
• Autoscaled to 204 workers
• Total elapsed time: 13 min

[github-actions[bot]]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.