Shouldn't only leader node remove outdated docs from nodes collection?

[luk-kaminski]

Expected Behavior

Only leader node remove outdated docs from nodes collection.

Current Behavior

All nodes do it in AbstractNodeService#ping() method every second. It means that in 16-nodes-big cluster each second Mongo is asked 16 times to remove stale node information. It seems to be unnecessary. It may potentially create funny issues when client has stale_leader_timeout set to different values in different nodes.

Possible Solution

Only leader calls dropOutdated() method.

Context

May be loosely related to HS-2290103299.

Your Environment

• Graylog Version: 5.2.3
• MongoDB Version: 5

[luk-kaminski]

@Emy-01 -> Bernd is opting for leaving this as it is. Please see all the comments in this PR for reference. I'd say we should close this issue with "Won't Do" resolution or leave it in Core Team's backlog.

[kmerz]

@bernd Could you please close this issue with a comment, if you are in favor of leaving it as it is?

[bernd]

@kmerz I removed it from the AX team's project. Let's keep it open in the core team's backlog for now.