faust icon indicating copy to clipboard operation
faust copied to clipboard

fix(tables): Handle None events in recovery process

Open level09 opened this issue 9 months ago • 1 comments

Problem

The _slurp_changelogs in faust/faust/tables/recovery.py method in the recovery process doesn't properly handle None events that can occasionally be received from the changelog queue. When a None event is encountered, the code attempts to process it, which leads to errors since None doesn't have the expected attributes and methods.

Solution

This PR adds a simple check at the beginning of event processing to detect and skip None events. The implementation:

  1. Silently checks if the received event is None
  2. For None events, checks if there are any active partitions or tables to recover
  3. If there's nothing to recover, signals recovery end and sleeps to avoid a busy loop
  4. Otherwise, skips processing the None event and continues to the next iteration

Benefits

  • Prevents errors when None events are encountered
  • Avoids busy loops in the recovery process when there's nothing to recover
  • Maintains clean logs in production by handling edge cases silently
  • Improves overall stability of the recovery process

Testing

Tested in a production environment with Kafka where None events were occasionally being received from the changelog queue. The fix successfully prevents errors and maintains stable operation.

level09 avatar Mar 07 '25 17:03 level09

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.09%. Comparing base (ff75c0b) to head (00b5b0e).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #657   +/-   ##
=======================================
  Coverage   94.09%   94.09%           
=======================================
  Files         102      102           
  Lines       11100    11100           
  Branches     1196     1196           
=======================================
  Hits        10444    10444           
  Misses        557      557           
  Partials       99       99           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Mar 08 '25 00:03 codecov[bot]