[WIP][CI] Upgrade pyspark to 4.0.0

[mshr-h]

As per title.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 87.52%. Comparing base (ccc3ada) to head (ade1271). Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #793   +/-   ##
=======================================
  Coverage   87.52%   87.52%           
=======================================
  Files          76       76           
  Lines        4625     4625           
  Branches      741      741           
=======================================
  Hits         4048     4048           
  Misses        375      375           
  Partials      202      202           

Flag	Coverage Δ
unittests	87.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.