beam icon indicating copy to clipboard operation
beam copied to clipboard

Add support for FLOAT to Python RowCoder

Open TheNeuralBit opened this issue 2 years ago • 6 comments

Part of #19815

This adds support for encoding the schema type FLOAT to Python's RowCoder. This is added in a new coder, SinglePrecisionFloatCoder, designed to be compatible with Java's FloatCoder.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI.

TheNeuralBit avatar Aug 08 '22 20:08 TheNeuralBit

@lostluck it looks like Go might be using the same coder for single-precision floats as for double-precision, is that right? From the Go PreCommit:

13:58:13 --- FAIL: TestStandardCoders (0.03s)
13:58:13     --- FAIL: TestStandardCoders/beam:coder:row:v1#06 (0.00s)
13:58:13         fromyaml_test.go:42: Failed "{beam:coder:row:v1 
13:58:13             
13:58:13             f_float$8c97b6c5-69e5-4733-907b-26cd8edae612 [] false}": err decoding "\x01\x00\x00\x00\x00\x00": decoding a struct { F_float float32 "beam:\"f_float\"" }
13:58:13             	caused by:
13:58:13             error decoding double field
13:58:13             	caused by:
13:58:13             EOF
13:58:13 FAIL

TheNeuralBit avatar Aug 08 '22 21:08 TheNeuralBit

Yes that does seem to be the case: https://github.com/apache/beam/blob/cc0b446509fc07c48f4157c12189852fce62e817/sdks/go/pkg/beam/core/graph/coder/row_encoder.go#L211-L214

Would you be opposed to me sending a change to encode as single-precision in that case?

TheNeuralBit avatar Aug 08 '22 21:08 TheNeuralBit

Codecov Report

Merging #22626 (bcb3822) into master (7a9bb76) will decrease coverage by 0.00%. The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master   #22626      +/-   ##
==========================================
- Coverage   74.19%   74.19%   -0.01%     
==========================================
  Files         709      709              
  Lines       93499    93524      +25     
==========================================
+ Hits        69370    69387      +17     
- Misses      22852    22860       +8     
  Partials     1277     1277              
Flag Coverage Δ
python 83.59% <80.00%> (-0.01%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ks/python/apache_beam/coders/coders_test_common.py 98.54% <ø> (ø)
sdks/python/apache_beam/coders/coders.py 88.00% <63.63%> (-0.35%) :arrow_down:
sdks/python/apache_beam/coders/coder_impl.py 93.91% <85.71%> (-0.06%) :arrow_down:
sdks/python/apache_beam/coders/row_coder.py 94.64% <100.00%> (+0.14%) :arrow_up:
sdks/python/apache_beam/coders/slow_stream.py 94.69% <100.00%> (+0.19%) :arrow_up:
sdks/python/apache_beam/internal/metrics/metric.py 93.00% <0.00%> (-1.00%) :arrow_down:
...hon/apache_beam/runners/direct/test_stream_impl.py 93.28% <0.00%> (-0.75%) :arrow_down:
...eam/runners/portability/fn_api_runner/execution.py 92.44% <0.00%> (-0.65%) :arrow_down:
...ks/python/apache_beam/runners/worker/sdk_worker.py 88.94% <0.00%> (-0.16%) :arrow_down:
...on/apache_beam/runners/dataflow/dataflow_runner.py 82.87% <0.00%> (-0.14%) :arrow_down:
... and 3 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov[bot] avatar Aug 08 '22 21:08 codecov[bot]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @AnandInguva for label python.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

github-actions[bot] avatar Aug 08 '22 22:08 github-actions[bot]

Well I tried to modify Go's RowCoder but couldn't manage to get the test working right. Instead I've just skipped this case, and filed https://github.com/apache/beam/issues/22629 to track.

TheNeuralBit avatar Aug 08 '22 23:08 TheNeuralBit

Run Java PreCommit

TheNeuralBit avatar Aug 09 '22 00:08 TheNeuralBit

@AnandInguva do you have time to review this?

TheNeuralBit avatar Aug 10 '22 16:08 TheNeuralBit

@TheNeuralBit I am on vacation. I will be back on Aug 22nd.

AnandInguva avatar Aug 10 '22 17:08 AnandInguva

assign to next reviewer

TheNeuralBit avatar Aug 10 '22 18:08 TheNeuralBit

R: @tvalentyn

TheNeuralBit avatar Aug 10 '22 18:08 TheNeuralBit

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

github-actions[bot] avatar Aug 10 '22 18:08 github-actions[bot]

Oops Valentyn is also out

R: @ryanthompson591

TheNeuralBit avatar Aug 10 '22 22:08 TheNeuralBit

Run Go PreCommit

TheNeuralBit avatar Aug 12 '22 18:08 TheNeuralBit