Large UDP payloads can be silently dropped

Due to nature of UDP protocol data could be silently dropped if reached specific limits. As the data/metrics are dropped silently usually we alerted by teams noticing missing metrics.

One idea discussed is to put some ignorable/optional metric indicating the number of bytes or metrics sent over the wire so FFWD agent could detect the loss and emit a metric to be alerted on.

Good suggestion. Or a checksum if we want to be really stingy with bandwidth.

@malish8632 do we know that limit ?

Yes the default is ~25MB. Check troubleshooting section in protobuf docs here.

Well, I wanted more datapoint from teams that are experiencing the problem.I doubt anybody is sending 25MB in the packet. I think the packets drop before reaching UDP max. I noticed, but I am not certain, that anything higher than 2048 bytes is being dropped. I will look into the issue when I get a chance.