beam
beam copied to clipboard
Add support for FLOAT to Python RowCoder
Part of #19815
This adds support for encoding the schema type FLOAT to Python's RowCoder. This is added in a new coder, SinglePrecisionFloatCoder
, designed to be compatible with Java's FloatCoder
.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
@lostluck it looks like Go might be using the same coder for single-precision floats as for double-precision, is that right? From the Go PreCommit:
13:58:13 --- FAIL: TestStandardCoders (0.03s)
13:58:13 --- FAIL: TestStandardCoders/beam:coder:row:v1#06 (0.00s)
13:58:13 fromyaml_test.go:42: Failed "{beam:coder:row:v1
13:58:13
13:58:13 f_float$8c97b6c5-69e5-4733-907b-26cd8edae612 [] false}": err decoding "\x01\x00\x00\x00\x00\x00": decoding a struct { F_float float32 "beam:\"f_float\"" }
13:58:13 caused by:
13:58:13 error decoding double field
13:58:13 caused by:
13:58:13 EOF
13:58:13 FAIL
Yes that does seem to be the case: https://github.com/apache/beam/blob/cc0b446509fc07c48f4157c12189852fce62e817/sdks/go/pkg/beam/core/graph/coder/row_encoder.go#L211-L214
Would you be opposed to me sending a change to encode as single-precision in that case?
Codecov Report
Merging #22626 (bcb3822) into master (7a9bb76) will decrease coverage by
0.00%
. The diff coverage is80.00%
.
@@ Coverage Diff @@
## master #22626 +/- ##
==========================================
- Coverage 74.19% 74.19% -0.01%
==========================================
Files 709 709
Lines 93499 93524 +25
==========================================
+ Hits 69370 69387 +17
- Misses 22852 22860 +8
Partials 1277 1277
Flag | Coverage Δ | |
---|---|---|
python | 83.59% <80.00%> (-0.01%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted Files | Coverage Δ | |
---|---|---|
...ks/python/apache_beam/coders/coders_test_common.py | 98.54% <ø> (ø) |
|
sdks/python/apache_beam/coders/coders.py | 88.00% <63.63%> (-0.35%) |
:arrow_down: |
sdks/python/apache_beam/coders/coder_impl.py | 93.91% <85.71%> (-0.06%) |
:arrow_down: |
sdks/python/apache_beam/coders/row_coder.py | 94.64% <100.00%> (+0.14%) |
:arrow_up: |
sdks/python/apache_beam/coders/slow_stream.py | 94.69% <100.00%> (+0.19%) |
:arrow_up: |
sdks/python/apache_beam/internal/metrics/metric.py | 93.00% <0.00%> (-1.00%) |
:arrow_down: |
...hon/apache_beam/runners/direct/test_stream_impl.py | 93.28% <0.00%> (-0.75%) |
:arrow_down: |
...eam/runners/portability/fn_api_runner/execution.py | 92.44% <0.00%> (-0.65%) |
:arrow_down: |
...ks/python/apache_beam/runners/worker/sdk_worker.py | 88.94% <0.00%> (-0.16%) |
:arrow_down: |
...on/apache_beam/runners/dataflow/dataflow_runner.py | 82.87% <0.00%> (-0.14%) |
:arrow_down: |
... and 3 more |
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more
Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer
:
R: @AnandInguva for label python.
Available commands:
-
stop reviewer notifications
- opt out of the automated review tooling -
remind me after tests pass
- tag the comment author after tests pass -
waiting on author
- shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
Well I tried to modify Go's RowCoder but couldn't manage to get the test working right. Instead I've just skipped this case, and filed https://github.com/apache/beam/issues/22629 to track.
Run Java PreCommit
@AnandInguva do you have time to review this?
@TheNeuralBit I am on vacation. I will be back on Aug 22nd.
assign to next reviewer
R: @tvalentyn
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
Oops Valentyn is also out
R: @ryanthompson591
Run Go PreCommit