Performance optimizations for Cardano transactions signature/proof

[jpraynaud]

Why

Now that we are signing transactions on a Mithril test network for the Cardano mainnet, we want to identify bottlenecks and prepare optimizations for the signature/proving of the Cardano transactions. This process will be iterative.

What

Here are the areas that we need to optimize:

• Minimum impact of the activation of the new type of data signed for Cardano transactions on the release-mainnet network:
  • Avoid blocking the production of certificates for current signed entity types
  • Implement warm-up strategy for new comers in the network that takes advantage of the 2 first epochs downtime
  • Proofs generation does not impact the production of certificates for the network
• Minimum footprint on the signer infrastructure (disk space, memory, CPU requirements):
  • Assess the maximum disk space required by the Cardano transactions store on the signer once pruning is activated (in GB)
  • Make sure this limit is not exceeded during warm-up phase
  • Assess the memory and CPU consumption of the signer during import/signature of the transactions (maybe offload memstores of Merkle trees on disk for the signature on the signer? this would avoid too much memory consumption)
  • Update the current Mithril signer footprint documentation
• Maximum throughput of proofs served by the aggregator REST API:
  • Assess/implement optimizations needed to enhance the proof serving throughput (step by step, iterative, most efficient optimizations at first)
  • Model the expected traffic coming from a third party provider serving transactions by address for e.g. a light wallet:
    • Computation based on their average/peak traffic
    • Model proofs generation throughput (and number of transactions to prove per proof)
    • Model certificate download throughput
  • Caching strategy for transaction proofs and certificate on the client for optimal pressure on the aggregator
  • Determine maximum number of transactions allowed when creating a proof
  • Horizontal scaling strategy in worst case scenario where maximum throughput of an aggregator is not enough (e.g. aggregator 'slaves' for serving proofs)

How

Minimum impact when activated on release-mainnet:

• [x] Impact analysis
• [x] Design throttle proofs generation strategy
• [x] Design warm-up strategy

Maximum proofs generation throughput

• [x] Assess current performances of the aggregator proof route #1687
• [x] Prototype first round of optimizations for increasing throughput #1687
• [x] Implement optimizations:
  • [x] Resource pooling for block range Merkle map #1698
  • [x] Merkle map proof computation optimization #1730 and #1722
  • [x] Database connection pooling #1760
  • [x] Parallelize sub Merkle trees creation and Merkle map proof computation #1756
  • [x] ~~Pre-compute sub Merkle proofs and use them in the block range Merkle map?~~
  • [x] ~~Cache last N certified transactions (database records, ...)~~ (Later if needed)
• [x] Implement boundaries on request payload sent to prover route #1757
• [x] Design and implement a stress test environment

Minimum SPO infrastructure footprint

• [x] Optimize pruning/warmup on the signer for optimal disk space usage #1692 #1766
• [ ] Merkle tree multiple storage (Sync, non Sync, low memory)?
• [ ] Benchmark signer resources consumption (CPU, memory, ...)
• [ ] Update the Mithril signer footprint documentation

Client optimizations

• [ ] Design and implement client caching strategy
• [ ] TBD

Infrastructure optimizations

• [ ] Plan VM upgrade and associated costs?