Build and ship flat storage

[exalate-issue-sync[bot]]

Epic to track work to be done to build and ship flat storage. Also tracks work that should be done to estimate the expected performance improvement.

Existing documentation:

https://nearinc.atlassian.net/wiki/spaces/EAP/pages/58556568/Flat+Storage

https://docs.google.com/document/d/1Tla2JtHHc6rJwZPUOF0skoaLiExrcV1PW2F65llfHG8/edit#heading=h.2r15ioo581ig

Related works and discussions:

https://nearinc.atlassian.net/browse/CRT-8 and subsequently linked tasks

https://nearinc.atlassian.net/browse/CRT-86

https://near.zulipchat.com/#narrow/stream/329092-pagoda.2Fstorage.2Fprivate/topic/Action.20receipt.20undercharging

[Longarithm]

(deleted)

[bowenwang1996]

@Longarithm is the progress in this tracking issue up to date?

[Longarithm]

Yes. The PR for catchups needs some polishing to be merged https://github.com/near/nearcore/pull/8193, tests for migration and proper flat state removal are TODO, other items weren't started yet.

[pugachAG]

Progress update (created together with @Longarithm):

• Refactoring #8577: spent 2 weeks to agree on the plan, 2 more weeks to implement
• Tests #8500: not started, need 1 week
• Limit memory consumption of deltas #8436: 3 weeks spent on discussions and experiments, 1 more week to implement
• Debug tools: implementation started by Marcin #8531, need 1 more week to polish it and add functionality
• Rollout tasks: not started, 1 week remains
• Params estimator support: done, but Jakob might require more assistance on cost estimations
• Some work to finish Flat Storage NEP: a couple of days

Total progress for Q1: 45%

cc @walnut-the-cat

[walnut-the-cat]

#8531 is now merged. Does it complete #8398?

[Longarithm]

Yes, closing

[jakmeier]

A new issue related to FS: https://github.com/near/nearcore/issues/8791

I don't think we should block FS on it, since it functionally does not depend on it. We just have to make sure #8791 also gets done before the next release cut, which should be more than enough time.

[Longarithm]

We completed the work for flat storage reads. There are some remaining tasks to release it in 1.34, which I listed in #8827. We plan to spend some time on it in Q2, as 1.34 is expected to happen in Q2.