onetl icon indicating copy to clipboard operation
onetl copied to clipboard

Add Spark 4.0 support

Open dolfinus opened this issue 1 year ago • 1 comments

Change Summary

  • Update CI matrix to include Spark 4.x. By default, it is not used for tests, unless some integration has been changed (like 2.x).
  • Spark 4.x also supports Java 22, include it to Readme & CI matrix.
  • Update JDBCConnection methods to be compatible with Spark 4.0 JDBCUtils methods signatures.
  • Update XML class get_packages() and check_if_supported() methods, as Spark 4.0 includes XML support.
  • Since Spark 4.0, DecimalType(38, 10).typeName() started returning decimal(38, 10) instead of decimal, which breaks some Oracle tests. Updated SparkTypeToHWM implementation to have direct mapping between Spark data type classes and HWM classes.

Notes:

  • Skip Excel tests on Spark 4.0, as there is no package version.

Related issue number

Checklist

  • [ ] Commit message and PR title is comprehensive
  • [ ] Keep the change as small as possible
  • [ ] Unit and integration tests for the changes exist
  • [ ] Tests pass on CI and coverage does not decrease
  • [ ] Documentation reflects the changes where applicable
  • [ ] docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change (see CONTRIBUTING.rst for details.)
  • [ ] My PR is ready to review.

dolfinus avatar Jul 25 '24 18:07 dolfinus

Codecov Report

:x: Patch coverage is 94.73684% with 3 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 93.43%. Comparing base (6d41000) to head (226549f). :warning: Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
onetl/_util/scala.py 60.00% 1 Missing and 1 partial :warning:
.../connection/db_connection/jdbc_mixin/connection.py 95.45% 0 Missing and 1 partial :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #297      +/-   ##
===========================================
- Coverage    93.44%   93.43%   -0.02%     
===========================================
  Files          234      234              
  Lines        10284    10308      +24     
  Branches       964      972       +8     
===========================================
+ Hits          9610     9631      +21     
- Misses         530      531       +1     
- Partials       144      146       +2     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Jul 25 '24 18:07 codecov[bot]