beam icon indicating copy to clipboard operation
beam copied to clipboard

Example of Online Clustering

Open shub-kris opened this issue 2 years ago • 3 comments

This PR aims to illustrates an example of performing online clustering using Stateful Processing and RunInference.

The entire implementation is divided into two different pipelines:

  1. write_data_to_pubsub_pipeline pushes data to PubSub
  2. clustering_pipeline reads data from PubSub and does online clustering

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI.

shub-kris avatar Sep 19 '22 08:09 shub-kris

R: @andyxiexu @damccorm

shub-kris avatar Sep 19 '22 08:09 shub-kris

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

github-actions[bot] avatar Sep 19 '22 08:09 github-actions[bot]

Codecov Report

Merging #23289 (de6b36d) into master (5520fe0) will decrease coverage by 0.01%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #23289      +/-   ##
==========================================
- Coverage   73.45%   73.44%   -0.02%     
==========================================
  Files         718      718              
  Lines       95528    95558      +30     
==========================================
+ Hits        70171    70180       +9     
- Misses      24061    24082      +21     
  Partials     1296     1296              
Flag Coverage Δ
python 83.16% <ø> (-0.03%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...s/interactive/dataproc/dataproc_cluster_manager.py 71.72% <0.00%> (-5.70%) :arrow_down:
sdks/python/apache_beam/internal/gcp/auth.py 73.33% <0.00%> (-5.34%) :arrow_down:
...pache_beam/runners/interactive/interactive_beam.py 81.70% <0.00%> (-0.64%) :arrow_down:
sdks/python/apache_beam/runners/direct/executor.py 96.46% <0.00%> (-0.55%) :arrow_down:
...hon/apache_beam/runners/worker/bundle_processor.py 93.30% <0.00%> (-0.25%) :arrow_down:
sdks/python/apache_beam/transforms/util.py 96.06% <0.00%> (ø)
...dks/python/apache_beam/options/pipeline_options.py 94.36% <0.00%> (+0.02%) :arrow_up:
...ks/python/apache_beam/runners/worker/sdk_worker.py 89.09% <0.00%> (+0.15%) :arrow_up:
...eam/runners/interactive/interactive_environment.py 92.02% <0.00%> (+0.30%) :arrow_up:
...che_beam/runners/interactive/interactive_runner.py 91.77% <0.00%> (+0.38%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov[bot] avatar Sep 19 '22 09:09 codecov[bot]

@damccorm linting and formatting fixed

shub-kris avatar Sep 27 '22 11:09 shub-kris