risingwave icon indicating copy to clipboard operation
risingwave copied to clipboard

feat(meta): make complete barrier non-async in worker loop

Open wenym1 opened this issue 1 year ago • 0 comments

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Previously after we collect a barrier, we call complete_barrier to commit the a new hummock snapshot and also do some post-collection work to update the catalog metadata. The complete_barrier is an async method, which involves works like doing IO to meta store backend, and acquiring async lock. The complete_barrier call will in-place block the global barrier manager worker loop, which may block the handling of other events.

In this PR, we change to spawn a task a do the complete_barrier work, so that the call on it will be non-blocking. We also refactor the method to split it into two parts, one called update_snapshot, which updates the hummock snapshot, and the other called update_tracking_jobs, which updates the information of tracked create streaming jobs.

Checklist

  • [ ] I have written necessary rustdoc comments
  • [ ] I have added necessary unit tests and integration tests
  • [ ] I have added test labels as necessary. See details.
  • [ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
  • [ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • [x] All checks passed in ./risedev check (or alias, ./risedev c)
  • [ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • [ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • [ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

wenym1 avatar Feb 18 '24 07:02 wenym1