feature: Filter from messages in descending order

[saada]

When using the message filtering feature, there's a way to filter messages from the start offset and find the oldest matches within the count specified. However, the reverse starting from the latest offset doesn't seem possible?

This seems to be the current supported behavior...

for (offset=START_OFFSET, offset<=LATEST_OFFSET, offset++):
    if filter(messages[offset]):
        results.push_to_top(messages[offset])
        returned_count++
        if returned_count >= MAX_RESULTS:
            break;

Whereas we’re looking for

for (offset=LATEST_OFFSET, offset>=START_OFFSET, offset--):
    if filter(messages[offset]):
        results.push_to_bottom(messages[offset])
        returned_count++
        if returned_count >= MAX_RESULTS:
            break;

Yes, it’s technically possible to zero in on the desired set in the example by changing the custom start offset but it’s a really noisy feedback loop

[weeco]

Hey @saada , the reason for this is that it's not possible with Kafka to consume backwards. So what we do is to start at an offset X and consume forwards. If you chose "Latest 500 messages" this means that we will first get the offset from 500 messages before the end and then we will start consuming forwards again. The problem with search filters is that we usually don't know how much we have to consume until your search request is satisfied.

Currently we do not store a bunch of messages in memory, but instead we stream all found results via Websockets to the frontend. So indeed it's not as straightforward as iterating the batch of results in the backwards order. If we wanted to do that we have to ensure that we do it in a way that does not consume too much memory in the backend. For instance we could always consume the most recent 500 messages by consuming "highWaterMark-500" forwards. add the found results to the stack and then continue with the next most recent 500 messages before that until we have reached either the earliest offset or the number of expected results. Is this what you envision?

[saada]

Yes. That's what I have in mind. Batches of 500s until we hit the desired number of matches. VS the current behavior which is: search the last 500 messages regardless of match count.