`SetTable`s are missing `on_clear`
mode has a method to clear all the elements in a set (https://github.com/faust-streaming/mode/blob/c0aa58181432402dca30aa5978179383863a185a/mode/utils/collections.py#L546-L548) which calls an .on_clear callback.
This .on_clear callback method is empty by default (https://github.com/faust-streaming/mode/blob/c0aa58181432402dca30aa5978179383863a185a/mode/utils/collections.py#L535-L536)
However, faust does not implement on_clear at all.
This means that the in-memory data structure is cleared but the change is not reflected in the changelog topic. Upon recovery elements that were previously cleared may re-appear. I think the fix would be very simple:
- adding a new operation here: https://github.com/faust-streaming/faust/blob/4a09533d3cb54801238545b4fa17fa74ac130138/faust/tables/sets.py#L34-L36
- implementing
on_clearsimilar to https://github.com/faust-streaming/faust/blob/4a09533d3cb54801238545b4fa17fa74ac130138/faust/tables/sets.py#L77-L78 maybe setting the value to a tombstone? - clearing the set when replaying the messages from the changelog https://github.com/faust-streaming/faust/blob/4a09533d3cb54801238545b4fa17fa74ac130138/faust/tables/sets.py#L95-L111
@cristianmatache @wbarnha , Since on_key_del is implemented for Table (https://github.com/faust-streaming/faust/blob/master/faust/tables/table.py#L86-L94) Could we simply (naively) implement this in terms of __delitem__?,
e.g. something like, in Table define
def on_clear(self):
for k in self.data.keys():
del self[k]