beam icon indicating copy to clipboard operation
beam copied to clipboard

[yaml] Adding Spanner IO Providers for Beam YAML

Open reeba212 opened this issue 1 year ago • 4 comments

  1. Added SpannerReadSchemaTransformProvider.java, a provider for reading from spanner.
  2. Modified SpannerWriteSchemaTransformProvider.java to incorporate error handling and fixed other issues.
  3. Modified MutationUtils.java to add a function "createRowFromMutation" which creates a Beam Row from a Mutation. This is used in the spanner write provider.
  4. Modified ErrorHandling.java to make it serializable.
  5. Modified standard_io.yaml to register the Spanner IO providers.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

reeba212 avatar Jul 25 '24 16:07 reeba212

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

github-actions[bot] avatar Jul 25 '24 17:07 github-actions[bot]

R: @damccorm

Polber avatar Jul 25 '24 19:07 Polber

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

github-actions[bot] avatar Jul 25 '24 19:07 github-actions[bot]

@reeba212 #32008 was merged, so you should be able to rebase on main and make sure checks pass!

Polber avatar Aug 27 '24 16:08 Polber

Codecov Report

Attention: Patch coverage is 0% with 48 lines in your changes missing coverage. Please review.

Project coverage is 57.15%. Comparing base (64b4aa6) to head (8f9178e). Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
sdks/python/apache_beam/io/gcp/spanner_wrapper.py 0.00% 34 Missing :warning:
sdks/python/apache_beam/yaml/integration_tests.py 0.00% 14 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #31987      +/-   ##
============================================
- Coverage     57.34%   57.15%   -0.19%     
  Complexity     1474     1474              
============================================
  Files           964      965       +1     
  Lines        153341   153435      +94     
  Branches       1076     1076              
============================================
- Hits          87933    87699     -234     
- Misses        63204    63537     +333     
+ Partials       2204     2199       -5     
Flag Coverage Δ
python 81.15% <0.00%> (-0.45%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 30 '24 19:08 codecov[bot]

Seems like the google-cloud-spanner version specified here conflicts with the version installed in the containers.

https://github.com/apache/beam/blob/master/sdks/python/setup.py#L444 https://github.com/apache/beam/blob/master/sdks/python/container/py38/base_image_requirements.txt#L72

At least one test suite is failing due to this. https://ge.apache.org/s/2eh6tfikgiwb2/console-log/task/:sdks:python:expansion-service-container:docker?anchor=258&page=1

cc: @tvalentyn

chamikaramj avatar Sep 12 '24 17:09 chamikaramj

cc @Polber

liferoad avatar Sep 12 '24 17:09 liferoad

@damccorm was working on rebuilding those - not sure the status

Polber avatar Sep 12 '24 17:09 Polber

Looks like it is under review still: #32411

Polber avatar Sep 12 '24 17:09 Polber

Yep, was just waiting on approval - just merged and it should be back to green

damccorm avatar Sep 12 '24 17:09 damccorm