Helix Incorrectly Reports Pending Messages

[ankitsultana]

We use the following MBean to track Helix Pending messages in Pinot Servers:

CLMParticipantReport:MonitorType=ParticipantMessageMonitor,ParticipantName=some_instance_id:

We have often seen that this MBean reports that there are Helix pending messages, but there's no thread processing any Helix messages, nor are there any messages in ZK.

We don't have a repro at the moment, but based on inputs we can try to see if we can find any common signals. One scenario where we saw this is documented in the Pinot ticket here: https://github.com/apache/pinot/issues/11626