beam icon indicating copy to clipboard operation
beam copied to clipboard

Add PytorchBatchConverter

Open yeandy opened this issue 2 years ago • 2 comments

Adding PytorchBatchConverter as discussed in https://github.com/apache/beam/issues/21440 as an initial step to integrate Batched DoFns into RunInference.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI.

yeandy avatar Sep 19 '22 20:09 yeandy

R: @TheNeuralBit

yeandy avatar Sep 19 '22 21:09 yeandy

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

github-actions[bot] avatar Sep 19 '22 21:09 github-actions[bot]

Codecov Report

Merging #23296 (9d537b8) into master (0d937d4) will decrease coverage by 0.27%. The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #23296      +/-   ##
==========================================
- Coverage   73.58%   73.31%   -0.28%     
==========================================
  Files         716      720       +4     
  Lines       95301    95766     +465     
==========================================
+ Hits        70127    70208      +81     
- Misses      23878    24262     +384     
  Partials     1296     1296              
Flag Coverage Δ
python 82.94% <0.00%> (-0.46%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...pache_beam/typehints/pytorch_type_compatibility.py 0.00% <0.00%> (ø)
sdks/python/apache_beam/io/gcp/bigquery_tools.py 73.30% <0.00%> (-12.43%) :arrow_down:
...s/interactive/dataproc/dataproc_cluster_manager.py 71.72% <0.00%> (-5.70%) :arrow_down:
.../python/apache_beam/testing/test_stream_service.py 88.09% <0.00%> (-4.77%) :arrow_down:
sdks/python/apache_beam/io/gcp/bigquery.py 71.31% <0.00%> (-2.94%) :arrow_down:
sdks/python/apache_beam/internal/gcp/json_value.py 85.50% <0.00%> (-2.90%) :arrow_down:
...on/apache_beam/runners/dataflow/dataflow_runner.py 80.88% <0.00%> (-2.12%) :arrow_down:
sdks/python/apache_beam/typehints/schemas.py 91.89% <0.00%> (-1.96%) :arrow_down:
sdks/python/apache_beam/runners/runner.py 72.39% <0.00%> (-1.85%) :arrow_down:
...che_beam/runners/interactive/interactive_runner.py 90.50% <0.00%> (-0.89%) :arrow_down:
... and 48 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov[bot] avatar Oct 13 '22 14:10 codecov[bot]

PTAL @TheNeuralBit

yeandy avatar Oct 14 '22 16:10 yeandy

It looks like codecov is unhappy since this file looks untested. Presumably because we don't have pytorch installed in the run generating coverage data. I confirmed this test is exercised on Jenkins: image

TheNeuralBit avatar Oct 14 '22 21:10 TheNeuralBit

Thanks for the small fix and confirmation!

yeandy avatar Oct 17 '22 17:10 yeandy