wormhole icon indicating copy to clipboard operation
wormhole copied to clipboard

Node: Observation batching

Open bruce-riley opened this issue 1 year ago • 1 comments
trafficstars

This PR introduces the batching of observations by adding the SignedObservationBatch gossip message. It modifies the guardian to wait up to one second in order to batch as many observations as possible (up to 4000) into a single gossip message. This not only reduces gossip traffic but also the load on the guardians and applications that listen to observations.

This PR is meant to go hand-in-hand with PR #4000, which further improves gossip traffic by splitting it across different topics, allowing the guardians and gossip listeners (like spies and flies) to only subscribe to the topics they need.

NOTE: Once the gossip network cutover is complete, support for SignedObservation (individual observation) messages will be dropped, but in the meantime, applications that listen to observations need to support both message streams.

bruce-riley avatar Jun 10 '24 16:06 bruce-riley

A concern has been raised regarding possible latency impacts of this change. Therefore I am converting this PR back to draft until those concerns can be addressed.

bruce-riley avatar Jul 05 '24 15:07 bruce-riley

The availability of Pyth's market data with as little latency as possible is a very strong usecase to warrant not being part of observation batching, and so they should be excluded from it.

aadam-10 avatar Jul 30 '24 17:07 aadam-10

This PR is being superseded by PR #4066, so this one is being closed.

bruce-riley avatar Aug 07 '24 19:08 bruce-riley