beam icon indicating copy to clipboard operation
beam copied to clipboard

Implement a `Top` partitioner

Open hjtran opened this issue 1 year ago • 11 comments

This diff implements a new parittioners module which includes a Top partitioner. Top replicates the combiners.Top behavior.

This is a WIP. I initially implemented this for my particular case which is global-window-only, but I should probably go back through this and add API for Smallest, Largest, PerKey etc. Posting this early though in case anyone has any comments

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

hjtran avatar Oct 23 '23 16:10 hjtran

reviewer: @damccorm

hjtran avatar Oct 23 '23 16:10 hjtran

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

github-actions[bot] avatar Oct 23 '23 16:10 github-actions[bot]

Codecov Report

Attention: Patch coverage is 0% with 33 lines in your changes are missing coverage. Please review.

Project coverage is 38.36%. Comparing base (38720a3) to head (206042e). Report is 1360 commits behind head on master.

Files Patch % Lines
sdks/python/apache_beam/transforms/partitioners.py 0.00% 33 Missing :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #29106       +/-   ##
===========================================
- Coverage   72.22%   38.36%   -33.87%     
===========================================
  Files         684      687        +3     
  Lines      100856   101693      +837     
===========================================
- Hits        72846    39010    -33836     
- Misses      26434    61107    +34673     
  Partials     1576     1576               
Flag Coverage Δ
python 29.97% <0.00%> (-52.86%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Oct 23 '23 16:10 codecov[bot]

Codecov Report

Merging #29106 (206042e) into master (38720a3) will decrease coverage by 33.87%. Report is 446 commits behind head on master. The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           master   #29106       +/-   ##
===========================================
- Coverage   72.22%   38.36%   -33.87%     
===========================================
  Files         684      687        +3     
  Lines      100856   101693      +837     
===========================================
- Hits        72846    39010    -33836     
- Misses      26434    61107    +34673     
  Partials     1576     1576               

Flag Coverage Δ python 29.97% <0.00%> (-52.86%) ⬇️ Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ sdks/python/apache_beam/transforms/partitioners.py 0.00% <0.00%> (ø) ... and 316 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

How exactly does the codecov bot invoke tests? I suspect that the new module should have ~100% coverage but the new test probably is getting missed somehow by codecov

hjtran avatar Oct 23 '23 16:10 hjtran

How exactly does the codecov bot invoke tests? I suspect that the new module should have ~100% coverage but the new test probably is getting missed somehow by codecov

Yeah, the codecov bot is frustratingly wrong sometimes, but especially on new code it seems? Your tests did run - https://ci-beam.apache.org/job/beam_PreCommit_Python_Coverage_Commit/4639/testReport/apache_beam.transforms.partitioners_test/ - so I'm not too worried about the bad signal

damccorm avatar Oct 23 '23 21:10 damccorm

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Dec 26 '23 12:12 github-actions[bot]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Feb 26 '24 12:02 github-actions[bot]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Apr 28 '24 12:04 github-actions[bot]

Codecov Report

Attention: Patch coverage is 0% with 33 lines in your changes missing coverage. Please review.

Project coverage is 38.36%. Comparing base (38720a3) to head (206042e). Report is 2655 commits behind head on master.

Files Patch % Lines
sdks/python/apache_beam/transforms/partitioners.py 0.00% 33 Missing :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #29106       +/-   ##
===========================================
- Coverage   72.22%   38.36%   -33.87%     
===========================================
  Files         684      687        +3     
  Lines      100856   101693      +837     
===========================================
- Hits        72846    39010    -33836     
- Misses      26434    61107    +34673     
  Partials     1576     1576               
Flag Coverage Δ
python 29.97% <0.00%> (-52.86%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Apr 28 '24 12:04 codecov-commenter

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Jun 28 '24 12:06 github-actions[bot]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Aug 28 '24 12:08 github-actions[bot]