phiresky
@sunaurus It seems like an idea but what would you use as a heuristic? Since you do want it to catch up as quick as possible and you don't want...
> "1 activity per second" That seems far too low tbh. When i implemented the queue my reference was 50-100 activities/second (with horizontal scaling though). Can you say how many...
What you say seems reasonable though maybe as a dynamic limit: Set the speed to 10x what the actual activity speed in the DB is. The question is just where...
> check the number of available workers Interesting idea, what might also work is use the db pool stats and start throttling if the pool has no available connections https://docs.rs/deadpool/latest/deadpool/struct.Status.html...
Can you maybe add detail of what you think exactly the issue was for you? Was it that the rest of the site wasn't able to do any DB queries?...
> database was just not ready for the extra load that came from full speed catch-up federation That's interesting because it also means that by just reducing the pool size...
> One problem is with [get_community_follower_inboxes()](https://github.com/LemmyNet/lemmy/blob/main/crates/db_views_actor/src/community_follower_view.rs#L45). Do you mean get_instance_followed_community_inboxes? I don't think `get_community_follower_inboxes` is used anymore
I think you misunderstood what I wrote, this PR doesn't increase the number of requests at all. In fact, it should more evenly distribute the load throughout the day
Ah you're actually right - for instances that are dead this would increase the load because they are never updated. It's "only" 2000 instances on lemmy.ml but yes. This would...
> Thats exactly what the updated column is, so no need for a new column. Well no because it's only set when the instance is alive, not when it's dead....