Scaling Guardian to enterprise workloads

[anvabr]

Problem description

The trajectory of growth in Guardian user numbers suggests that in the near future Guardian might see non-trivial active user numbers, such that single instances would need to support thousands of simultaneous users. There are concerns about the feasibility of scaling Guardian instances to such workloads with acceptable resource (hardware) costs.

Requirements

• make necessary changes to Guardian and document hardware (DevOps) configuration such that deployed instances would satisfy performance parameters specified in the 'acceptance criteria' section below.
• review and where needed revamp Guardian web application FrontEnd (FE) <-> BackEnd (BE) communication (i.e. FE updates are too frequent and thus too taxing on BE, they need to be reasonably batched and sent together).
• potentially introduce caching in FE, between FE and BE, and possibly between BE and the Database (DB) and/or Hedera & Inter-Planetary File System (IPFS) - for examples there are cases when immutable documents, schemas, etc which don't change there are unnecessarily reloaded or resent.
• implement memory usage optimisation, potentially in-memory compression to make sure the services don't blow up the Random Access Memory (RAM) of the cloud "hardware".
• Prove with practical testing that Guardian can fulfil acceptance criteria requirements below.

Definition of done

• Guardian capacity planning docs reflect the real scalability profile of Guardian (based on testing experience)
• Documentation updated accordingly

Acceptance criteria

• Guardian can scale to 2000+ simultaneous active users, all operating on the same instance actively performing their policy and system tasks
• Guardian DB can scale to 100s of GB of data
• No changes to the functionality of Guardian is visible to the users