[yaml] Adding Spanner IO Providers for Beam YAML
- Added SpannerReadSchemaTransformProvider.java, a provider for reading from spanner.
- Modified SpannerWriteSchemaTransformProvider.java to incorporate error handling and fixed other issues.
- Modified MutationUtils.java to add a function "createRowFromMutation" which creates a Beam Row from a Mutation. This is used in the spanner write provider.
- Modified ErrorHandling.java to make it serializable.
- Modified standard_io.yaml to register the Spanner IO providers.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers
R: @damccorm
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers
@reeba212 #32008 was merged, so you should be able to rebase on main and make sure checks pass!
Codecov Report
Attention: Patch coverage is 0% with 48 lines in your changes missing coverage. Please review.
Project coverage is 57.15%. Comparing base (
64b4aa6) to head (8f9178e). Report is 7 commits behind head on master.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| sdks/python/apache_beam/io/gcp/spanner_wrapper.py | 0.00% | 34 Missing :warning: |
| sdks/python/apache_beam/yaml/integration_tests.py | 0.00% | 14 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #31987 +/- ##
============================================
- Coverage 57.34% 57.15% -0.19%
Complexity 1474 1474
============================================
Files 964 965 +1
Lines 153341 153435 +94
Branches 1076 1076
============================================
- Hits 87933 87699 -234
- Misses 63204 63537 +333
+ Partials 2204 2199 -5
| Flag | Coverage Δ | |
|---|---|---|
| python | 81.15% <0.00%> (-0.45%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Seems like the google-cloud-spanner version specified here conflicts with the version installed in the containers.
https://github.com/apache/beam/blob/master/sdks/python/setup.py#L444 https://github.com/apache/beam/blob/master/sdks/python/container/py38/base_image_requirements.txt#L72
At least one test suite is failing due to this. https://ge.apache.org/s/2eh6tfikgiwb2/console-log/task/:sdks:python:expansion-service-container:docker?anchor=258&page=1
cc: @tvalentyn
cc @Polber
@damccorm was working on rebuilding those - not sure the status
Looks like it is under review still: #32411
Yep, was just waiting on approval - just merged and it should be back to green