onetl
onetl copied to clipboard
Add Spark 4.0 support
Change Summary
- Update CI matrix to include Spark 4.x. By default, it is not used for tests, unless some integration has been changed (like 2.x).
- Spark 4.x also supports Java 22, include it to Readme & CI matrix.
- Update JDBCConnection methods to be compatible with Spark 4.0 JDBCUtils methods signatures.
- Update XML class
get_packages()andcheck_if_supported()methods, as Spark 4.0 includes XML support. - Since Spark 4.0,
DecimalType(38, 10).typeName()started returningdecimal(38, 10)instead ofdecimal, which breaks some Oracle tests. UpdatedSparkTypeToHWMimplementation to have direct mapping between Spark data type classes and HWM classes.
Notes:
- Skip Excel tests on Spark 4.0, as there is no package version.
Related issue number
Checklist
- [ ] Commit message and PR title is comprehensive
- [ ] Keep the change as small as possible
- [ ] Unit and integration tests for the changes exist
- [ ] Tests pass on CI and coverage does not decrease
- [ ] Documentation reflects the changes where applicable
- [ ]
docs/changelog/next_release/<pull request or issue id>.<change type>.rstfile added describing change (see CONTRIBUTING.rst for details.) - [ ] My PR is ready to review.
Codecov Report
:x: Patch coverage is 94.73684% with 3 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 93.43%. Comparing base (6d41000) to head (226549f).
:warning: Report is 1 commits behind head on develop.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| onetl/_util/scala.py | 60.00% | 1 Missing and 1 partial :warning: |
| .../connection/db_connection/jdbc_mixin/connection.py | 95.45% | 0 Missing and 1 partial :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## develop #297 +/- ##
===========================================
- Coverage 93.44% 93.43% -0.02%
===========================================
Files 234 234
Lines 10284 10308 +24
Branches 964 972 +8
===========================================
+ Hits 9610 9631 +21
- Misses 530 531 +1
- Partials 144 146 +2
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.