Live refresh causes significant traffic and frequent “offline” message

[schoel-bis]

• [x] I have searched the issues of this repository and believe that this is not a duplicate.

Description 😯

In the case of our project in Tunisia, project list data is ~360kB in size. Live update reloads that entire amount every 5 seconds. This has two negative consequences:

• It causes ~4MB of extra traffic for each open UI per minute
• On slow connections, it often causes timeouts or other connection failures which will in turn cause an “offline” notice to pop up in the lower left corner of the screen, although the application is not actually off-line, only the connection could no't sustain the strain.

A few ideas for possible fixes/mitigations:

• Add a compression middleware to reduce the amount of data that goes over the wire
• Reduce the amount of data transferred for these updates (not sure if that is really possible, though)
• Change the logic such that only new and updated projects are transmitted, e.g. by including a cut-off timestamp in the request
• While at that: use WebSockets or at least long polling instead of simple polling to get those changes to the client

How to reproduce 🕹

1. Add a significant number of projects
2. Navigate to project overview
3. Watch network monitor and, on a slow or throttled, or otherwise unreliable connection, observe the “offline” notifications popping up

Your Environment 🌎

Tech	Version
Frontend	v1.?.?
API	v1.?.?
Blockchain	v1.?.?
Export-Service	v1.?.?
Email-Service	v1.?.?
Storage-Service	v1.?.?

[SamuelPull]

When implementing longer polling interval/pagination/etc. keep in mind that some E2E tests depend on 5 second live update 🤐 Opening #1653 for "quick win", but I would like to keep this issue open until a better solution is implemented. Opening #1665 to give users an option to turn off live refresh

[galethil]

@schoel-bis could you please provide more details about how many projects do you have in Tunisia?

[schoel-bis]

That's 117 projects currently. The total size of the project list downloaded every 5 sec (before applying the new configuration from #1638) is 371.09 kB.

[SamuelPull]

@schoel-bis Thank you for bringing this to our attention, and for the ideas. Websockets are unfortunately of question, some partners can't use them. All users have now an option to turn off live update.

We're working on a new endpoints for projects and subprojects with pagination, hopefully for the 2.10.0 release.