Azure Fluid Relay Service size limits

[BrianHuf]

Issue Category

Can't find an answer to a specific question.

Details

I'm looking to understand if there are op and/or summary size limits on FRS.

I reviewed https://docs.microsoft.com/en-us/azure/azure-fluid-relay/ and didn't find an answer.

While testing a large Property DDS scenario two errors were noticed. First, errors were found when commit()'ing large changes. This was avoided by committing more frequently (i.e., smaller ops). Then after some time, the historian would return an HTTP 413 (Payload Too Large).

To further explore, an automated test was created (https://github.com/dstanesc/FluidPatterns/blob/master/libs/fluid-util/src/size.test.ts)

Based on testing, it appears

• op limit is 1 MB
• summary limit is 10 MB

Is this correct?
Is it configurable?

[tylerbutler]

@ahmedbisht Is this on your radar?

[tylerbutler]

Related to #10201.

[vladsud]

Sorry for not getting back faster, but here is my best understanding:

• Op batch size: 1Mb (limit comes from socket.io & Kafka limits that are both 1Mb)
• FRS summary size: 30Mb.

Here are couple good pieces of news:

• We are working on op compression as mitigation, and couple other techniques to reduce the pain of hitting 1Mb limit. We are not sure yet if we will go as far as fully fixing this area (i.e. have infinite limits, at least in terms of implementation, we might still have some policy putting limits here), but we will substantially raise the limits here. Our experiments show that payloads up to 5Mb will go through, after that - it depends on actual payload content (how well it compresses).
• Summary size: One of our external partners works on blob compression that will have similar effect (as above) on summary limits. The right fix here is to make summaries more incremental (they are incremental in runtime, but some DDSs are not incremental, so single change causes DDS to submit full state). Fixing this across the board will take longer, but I'd suggest you mention what DDSs you use that cause trouble.

[microsoft-github-policy-service[bot]]

This PR has been automatically marked as stale because it has had no activity for 60 days. It will be closed if no further activity occurs within 8 days of this comment. Thank you for your contributions to Fluid Framework!