NET-865: enable full gap filling

[harbu]

Checklist

• [ ] Has passing tests that demonstrate this change works
• [ ] Updated Changelog
• [ ] Updated Documentation

[linear[bot]]

NET-865 re-enable full gap filling

Currently upon reaching a gap, a client will try to fill the gap 10 times. If after those 10 attempts, it is unable to fill the gap, it will give up and proceed forward. However, it will also skip filling any new gaps that occurred while it was filling the original gap.

It would be better if it tried to fill the new gaps that occurred while it was filling the original gap. It used to originally work this way (if my memory serves me), but was changed due to the gap filling working so slow that the user of the client felt that the subscription had gotten "stuck".

I would propose we change the behavior back to how it was before. To avoid the "stuck" situation, we should lower the default wait timeouts and gap filling retries attempts instead. But indeed there is a risk here of buffer growing indefinitely.

Some more fancier solutions

• Concurrent gap filling
• Dynamic gap filling based on buffer watermarks

[harbu]

Addressed PR comments @teogeb, please re-review

[teogeb]

Replaced by https://github.com/streamr-dev/network/pull/1628