pravega-samples
pravega-samples copied to clipboard
pravega
Issue 191: Add at-least-once stream processing example
These examples are intended to illustrate how at-least-once semantics can be achieved with Pravega. In particular, these illustrative examples do not use Apache Flink.
In the event that a single worker dies, the segments that it was reading will be automatically assigned to another worker. However, this code does not transfer the state between workers, leaving the state inconsistent.
Also, from what point does the reader read the newly assigned segments? Does it begin from the last checkpoint?
@claudiofahey due to issues we had in previous releases merging develop into master, we have changed the release approach and the current development branch is dev. We plan to delete develop to avoid confusions, so could you please re-open this PR against dev? Thanks!
The latest commit should provide at-least-once semantics in a clean way using only Pravega. I believe the code is complete but it does not have any automated tests yet. We also need to update the README to describe how this works.
This PR is ready for review. For a description of this PR, see https://github.com/claudiofahey/pravega-samples/blob/issue-191-streamprocessing/pravega-client-examples/src/main/java/io/pravega/example/streamprocessing/README.md.
Unit and integration tests can be run with ./gradlew pravega-client-examples:build.
@RaulGracia, I have addressed all of your comments. However, it appears that there is an intermittent timeout failure in the test kill5of6Test. I will continue to investigate.
@claudiofahey maybe the failure you are seeing are related to the sporadic failures we see in Pravega core repo related to tests involving Pravega standalone: https://github.com/pravega/pravega/issues/5864
@claudiofahey I tested the sample and it works. Also compiled with JDK11 and JDK8 and it also works (./gradlew installDist). But I also observed the failure in test kill5of6Test. Please, let's try to get this test passing so we can merge this one soon.
FYI, I ran the integration 20 times and all have passed:
for i in {1..20}; do ./gradlew clean test; done |& tee -a /tmp/test.log
grep "SUCCESSFUL" /tmp/test.log
BUILD SUCCESSFUL in 3m 42s
BUILD SUCCESSFUL in 3m 31s
BUILD SUCCESSFUL in 3m 56s
BUILD SUCCESSFUL in 3m 41s
BUILD SUCCESSFUL in 3m 47s
BUILD SUCCESSFUL in 6m 25s
BUILD SUCCESSFUL in 4m 18s
BUILD SUCCESSFUL in 4m 34s
BUILD SUCCESSFUL in 4m 25s
BUILD SUCCESSFUL in 4m 32s
BUILD SUCCESSFUL in 4m 16s
BUILD SUCCESSFUL in 4m 16s
BUILD SUCCESSFUL in 6m 29s
BUILD SUCCESSFUL in 4m 5s
BUILD SUCCESSFUL in 4m 8s
BUILD SUCCESSFUL in 4m 6s
BUILD SUCCESSFUL in 4m 26s
BUILD SUCCESSFUL in 4m 29s
BUILD SUCCESSFUL in 4m 2s
BUILD SUCCESSFUL in 3m 37s
@claudiofahey tests passed for me. I had to upgrade the Scala version of Spark in the Hadoop sample, as build was broken by one of the previous PRs. But now looks good, thanks!.
@tkaitchuck please, would you mind to do another review to either approve it or request more changes?
@claudiofahey can you address @tkaitchuck comments so we can close this one? I think we are close.
@claudiofahey any updates addressing @tkaitchuck feedback? This one is actually looks close to get merged, wondering if it is worth it to do a last push for it.