Release UTxO-HD v0.1

[dnadales]

Previous work

• :heavy_check_mark: Consensus implementation ready (both the in-memory and LMDB backend).
• :heavy_check_mark: Feature branch integrated into a cardano-node branch.
• :heavy_check_mark: System-level benchmarks for the in-memory backend done.
  • :heavy_check_mark: The value-only workload benchmark showed no regressions for the metrics of interest (forging, peer-propagation, end-to-end propagation). :warning: However, this feature demands more resources. Ad-hoc measurements on a running node show we consume about 2.5 GB extra with the in-memory solution.
  • :heavy_check_mark: Like in the value-only benchmark, the Plutus workload benchmark did not show any regression for the metrics of interest; however, it also showed an increase in resource usage.
• :no_entry_sign: System-level benchmarks for the LMDB backend blocked (see Depends On below).
• :heavy_check_mark: We managed to run a UTxO-HD capable node in legacy mode, which maintains the baseline memory usage while keeping all the ledger states in memory (as the current node does). This legacy mode is not production-ready because it requires further integration and testing.
• :heavy_check_mark: We finished the redesign of the Ledger DB API, which the LSM-tree backend will require, but, in addition, it might enable us to design an in-memory backend that will keep the baseline performance and resource requirements without requiring the code duplication and additional maintenance effort the legacy mode does.
• :heavy_check_mark: We integrated the existing implementations (in-memory and LMDB into the new Ledger DB API).
• :heavy_check_mark: We implemented the new in-memory backend.

Current work

• :hammer_and_wrench: P&T is working on the LMDB benchmarking setup. the current benchmark setup uses virtualized I/O, which renders the results unreliable. The benchmarking setup needs to be adapted accordingly.
• :hammer_and_wrench: Perform ad-hoc and system-level benchmarks on the new in-memory backend.
• Re-enable Consensus microbenchmarks (mempool) or establish a new baseline if system-level benchmarks show good performance.
• Refine the structure of the Git history for future reference.
• Merge the behaviour-preserving parts of the feature branch into main.
• Document the feature and inform the community (eg through blogposts).

Current implementation status

• UTXO-HD code is rebased on top of cardano-node 8.9.1
• UTXO-HD node is runnable in three modes:
  • V1InMemory (not recommended)
  • V2InMemory
  • V1LMDB
• It successfully syncs, replays and answers queries
• Branches containing the current implementation:
  • https://github.com/IntersectMBO/ouroboros-consensus/pull/977
  • https://github.com/IntersectMBO/cardano-api/pull/494
  • https://github.com/IntersectMBO/cardano-node/pull/5726

Roadmap

• [x] https://github.com/IntersectMBO/ouroboros-consensus/issues/905
• [x] https://github.com/IntersectMBO/ouroboros-consensus/issues/906
• [x] https://github.com/IntersectMBO/ouroboros-consensus/issues/907
• [x] https://github.com/IntersectMBO/ouroboros-consensus/issues/908
• [x] #5697
• [ ] https://github.com/IntersectMBO/ouroboros-consensus/issues/923
• [ ] https://github.com/IntersectMBO/ouroboros-consensus/issues/909
• [ ] https://github.com/IntersectMBO/ouroboros-consensus/issues/910
• [ ] https://github.com/IntersectMBO/ouroboros-consensus/issues/911
• [ ] https://github.com/IntersectMBO/ouroboros-consensus/issues/912
• [ ] https://github.com/IntersectMBO/ouroboros-consensus/issues/126
• [ ] https://github.com/input-output-hk/ouroboros-consensus/issues/28

Requirements

• [ ] We should provide an in-memory alternative with equal or better performance and resource requirements than the current node.
• [ ] Default backend should be in-memory.
• [ ] No impact on Conway release.
• [ ] Release should be signed off by P&T.
• [ ] Release should be signed off by SDET.
• [ ] The UTxO-HD feature should be explained and advertised to the community through different channels and appropriately documented.
• [ ] The feature should not impact the Conway release.

Impact

• No migration mechanism will be implemented: users will need to re-apply the chain (which will take several hours but is less time-consuming than syncing from scratch).
• [ ] https://github.com/input-output-hk/cardano-db-sync/issues/1296

[dnadales]

Memory usage comparison using baseline 8.1.2 and 8.2.1-pre against UTxO-HD with both backends after 11 days running (2 corresponds to in-memory, 3 to LMDB). NOTE: we have not run a node whose memory is restricted to 8GB. This might be a very useful data point.

Edge #    Version                Extra Config                Post Replay RSS    +~3 day RSS    +~11 day RSS
1         8.1.2                                              13.47 GiB          13.88 GiB      13.88 GiB
2         8.2.1 HD (52e9e448)                                15.82 GiB          16.15 GiB      16.39 GiB
3         8.2.1 HD (52e9e448)    --lmdb-ledger-db-backend    9.50 GiB           9.74 GiB       10.43 GiB
4         8.2.1-pre                                          13.28 GiB          13.98 GiB      14.01 GiB

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[dnadales]

• The latest preemtive sync showed the following results: It is important to note that the log for the baseline node was 7.7MBs whereas the UTXO-HD runs resulted in 4.1GBs. This was done with the legacy tracing system in which eliding tracers might be at fault for so much output and at least part of the regression.

[jasagredo]

Closing in favor of #5918.