[KYUUBI #6832] Impl Spark DSv2 YARN Connector that supports reading YARN aggregation logs

[naive-zhang]

trafficstars

Why are the changes needed?

Initial implement Spark DSv2 YARN Connector that supports reading YARN aggregation logs and YARN apps

Close #6832.

How was this patch tested?

Add tests YarnAppQuerySuite, YarnCatalogSuite, and YarnLogQuerySuite

Was this patch authored or co-authored using generative AI tooling?

• [x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (b265ccb) to head (1d4d45d).

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6856   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         687     687           
  Lines       42463   42463           
  Branches     5796    5796           
======================================
  Misses      42463   42463           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[pan3793]

@naive-zhang it may take some time to merge the whole feature into the master, to speed up the process, you may want to split it into several PRs, for example, the first PR just include a YarnCatalog, and then each PR focus one table

[pan3793]

@naive-zhang Please let me know if you are still interested in this PR, if not, or if you don't have time to complete it, it's also good to let me know so that others who are interested in it can take over~

[davidyuan1223]

@naive-zhang Please let me know if you are still interested in this PR, if not, or if you don't have time to complete it, it's also good to let me know so that others who are interested in it can take over~

if there still no body response when i complete the authz with paimon, i wanna try this. currently, my team also think we need to monitor more detail, and i think maybe we could record the spark stage's shuffle read/write size/num in the extensions

[pan3793]

@davidyuan1223 Thanks for your interest in this feature. Given that the original author has no responds for a while, it's fine to take over. But let me extract the already done parts and merge them first, this may take a few days to complete.

[github-actions[bot]]

Thanks for the PR! This PR is being closed due to inactivity. This isn't a judgement on the merit of the PR in any way. If this is still an issue with the latest version of Kyuubi, please reopen it and ask a committer to remove the Stale tag!

Thank you for using Kyuubi!