si icon indicating copy to clipboard operation
si copied to clipboard

feat(edda): impl a `CompressingStream` to reduce number of Edda requests

Open fnichol opened this issue 7 months ago • 0 comments

This change introduces a CompressingStream into the Edda server which wraps the underlying Edda requests subscription. It reads message from the upstream subscription, and attempts to "compress" multiple API requests into a single, new compressed request.

The simplified happy path for the stream is as follows:

  1. Reading the first message from the subscription

    The message is parsed to determine how many messages are pending behind this message. That pending number plus the first message determine the "read window" for the state machine. That is, how many messages to immediately consume before compressing.

  2. Parsing the first message into an API request

    The message is parsed into an API request using the ApiTypesNegotiate functionality.

  3. Reading the next message from the subscription in the read window

    The state is looped back upon until all messages in the read window are consumed.

  4. Parsing the next message into an API request in the read window

    After reading the message, it is parsed into an API request, then it returns to reading the next message (in 3) until all messages in the read window are read and parsed.

  5. Compressing multiple API requests into a single compressed request

    All the API requests are fed into the CompressedRequest::from_requests() function, which reduces multiple API requests into one CompressedRequest type. This is where the "compression" takes place.

  6. Deleting a message from the Jetstream stream

    This state is looped back upon until all Jetstream messages are deleted, as they have been successfully read, parsed, and compressed.

  7. Converting request into final message to yield from [Stream]

    The final CompressedRequest type is serialized into a LocalMessage which implements the naxum::MessageHead trait, thus enabling this stream to be the incoming stream for a Naxum app.

fnichol avatar May 30 '25 06:05 fnichol