estuary icon indicating copy to clipboard operation
estuary copied to clipboard

Narrow down the contents we queue toCheck on startup

Open neelvirdy opened this issue 2 years ago • 0 comments

We shouldn't queue every pinned content to be rechecked on startup. We should only need to queue contents that we may need to make more deals for.

Could we track active_deal_count per content and filter for active_deal_count < REPLICA_TARGET?

Maybe we just batch this process so we dont oversaturate the queue on startup when we're behind?

https://github.com/application-research/estuary/blob/33cc5415491f5dafa92239d2bd1524fa789aedb4/contentmgr/queue.go#L136

neelvirdy avatar Dec 09 '22 17:12 neelvirdy