Newly alive messages post invis can take long time to find

[devshorts]

If we have the scenario where you have an invis message that is holding everyone up (i.e. a 10 day worker message) and you are getting millions of messages behind it, then when we try and find the next invis message we are going to scan through all those messages.

Instead I propose we do some better optimizations for looking for the next invis (and messages that have become alive post-invis behind it).

We could leverage the fact that the repair worker is finalizing buckets behind him, HOWEVER he will not delete a bucket that has an unacked message in it, This message causes the bucket to dangle. We can track buckets that are dangling (they may not be consecutive) and jump from dangling bucket to dangling bucket like a linked list in order to find the next potentially invis or non-invis message.

Imagine we had a table like

CREATE TABLE dangling_buckets(
    queueid text,
    dangling_bucket bigint,
    PRIMARY KEY (queueid, dangling_bucket) 
);

This creates a sorted queue of dangling bucket pointers by queue id which we can use as metadata to hop the invis pointer forwarder faster

That said, we also need to make sure that when a previously delivered message is finally acked, if it is in a dangling bucket and the bucket is all ack'd and tombstoned we should delete the bucket. As it stands, given the lifecycle of a queue we can have many dangling but actually complete buckets which will cause queue delete to be really slow.

Regardless of if we use this metadata table for invis jumping or not, we should create it anyways for proper cleanup. As messages are acked they should check if they are in the danglng bucket, and if all messages in the dangling bucket are completed then delete the bucket.

Invis should also check the bucket and validate that if all messages in a dangler are actually complete then kill the bucket and remove the dangler.

This way we can do lazy bucket cleanup on reads

[jakeswenson]

@devshorts is working on refactoring the invisibility logic into a invisibility strategy pattern. Which will make it easier to replace this logic with whatever invisible message strategy we want.