[Backport 2.x] Refactor SparkQueryDispatcher

[opensearch-trigger-bot[bot]]

Backport d32cf94c132cbc4313566e92016a7840542c58dd from #2636.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.42%. Comparing base (294566f) to head (f40e94a).

Additional details and impacted files

@@            Coverage Diff            @@
##                2.x    #2643   +/-   ##
=========================================
  Coverage     95.42%   95.42%           
- Complexity     5170     5178    +8     
=========================================
  Files           491      492    +1     
  Lines         14598    14610   +12     
  Branches        974      969    -5     
=========================================
+ Hits          13930    13942   +12     
  Misses          643      643           
  Partials         25       25           

Flag	Coverage Δ
sql-engine	95.42% <100.00%> (+<0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[ykmr1224]

Backport is addressed by https://github.com/opensearch-project/sql/pull/2669