onflow
Public key de-duplication
Objective
optimize public key storage - remove duplicate public keys
Key results
Rough estimates based on June 2025 mainnet snapshot:
- remove ~48% of all public keys stored
- reduce state size and disk used by ~20 GB
- reduce number of registers by ~68 million
- TBD after deployment:
- reduce peak memory use on EN (this can be ~3x state size reduction but that isn't guaranteed due to external factors like Go garbage collection, unrelated components/processes, etc.)
Some other considerations are mentioned in the issue opened on June 26, 2022.
DACI
| Driver | Approver | Consulted | Informed |
|---|---|---|---|
| Technical: @fxamacker, EM: @j1010001 | @Kay-Zee, @dete | @zhangchiqing , @janezpodhostnik | @vishalchangrani |
Work Done
In addition to deduplication with migration (PR #7738) to run during spork, we also completed work on runtime deduplication (PR #7829 automatically detects duplicate public keys being added and stores them in deduplicated form).
- #7738
- #7829
- #7834
- #7835
- #7841
- #7875
- #7878
- #7925
remove ~46% of all public keys stored reduce memory and storage used by ~19 GB reduce registers stored by ~65M
@j1010001 I updated the above with a newer estimate based on June 2025 mainnet snapshot:
- remove ~48% of all public keys stored
- reduce state size and disk used by ~20 GB
- reduce number of registers by ~68 million
- TBD after deployment:
- reduce peak memory use on EN (this can be ~3x state size reduction but that isn't guaranteed due to external factors like Go garbage collection, unrelated components/processes, etc.)
Some other considerations are mentioned in the issue I opened on June 26, 2022.
so we have like 48% dupes for public keys ?
PS: I was surprised at first but it seems it is totally possible, https://www.flowscan.io/tx/9a35d1fcfd746a9617483db11501953b9acff376f26b2f4b8416b58c7fb92c1c?tab=script
I don't have much faith in developers, but sometimes I am still surprised :)
Status update:
Janez and I agree on the approach discussed on July 21 and I will keep Janez in the loop if any changes to what we discussed becomes necessary to make sure this doesn't conflict with what Janez has planned for concurrent execution, fvm, etc.
Timeline:
- July 17-18: refreshed on this topic and started getting familiar with relevant code
- July 21: created informal doc for discussion with Janez and we had very good meeting about it:
- goals, current status & some characteristics of existing public keys
- proposed changes to data structure
- proposed changes to code
- July 21: worked on mock migration using proposed format change until ~8:30 PM Central
Created this Google sheet of accounts with 10K+ keys. I am trying to figure who these accounts belong to and understand the reason why they had to define so many keys on the account.
BTW, for each register we eliminate, some non-register nodes in the trie get eliminated too.
Created this Google sheet of accounts with 10K+ keys. I am trying to figure who these accounts belong to and understand the reason why they had to define so many keys on the account.
@vishalchangrani thanks for putting together that Google sheet!
I wanted to share an update here on Friday but I didn't reach a good stopping point with work until after 9 PM Central.
Last week, I continued improving the data format and deduplication. I need to verify the result of these improvements, but it looks like we might be able to reduce millions more registers than the previously discussed ~68 million.
As a given, the data format and deduplication removes registers but it doesn't remove/delete/lose data we need on mainnet (e.g., the public key + seqeuence number combination are not lost).
I'm currently writing validation code and will use it to verify deduplicated data. After that, I'll run the proof-of-concept migration using the updated data format & deduplication logic. Then I will begin to optimize migration to reduce spork downtime.
I'll schedule the next meeting with Janez after I gather more data and have verified results to share. I'm making progress and don't want to create needless meetings about unverified results.
On Friday (Aug 1), I resumed work on validations and haven't run into any showstoppers yet. The validations have been finding typical issues and I finished fixing them around 9 PM Central on Friday.
The updated data format and deduplication can eliminate roughly 85-86 million registers (much better than the previously discussed 68 million registers).
However, the improved approach needs to pass additional validations. It also needs a sanity check from Janez because it involves data format changes to existing registers (e.g., changes to account status registers).
I will schedule a meeting with Janez next week after gathering more data, creating and running more validations, and writing a summary to go over during the meeting.
If we don't find any showstoppers or major changes, then I will proceed with next steps: integrating the deduplication into migration, measuring real migration duration and RAM usage to see if optimizing migration is worthwhile, etc. I will also update FVM to use the new data format.
This morning (Aug 6), I wrapped up creating and running additional validations using the updated data format and deduplication.
I shared a summary of changes (e.g., data format changes since our first meeting) to go over with Janez. We scheduled our meeting for Friday, Aug 8 because Janez is swamped with urgent work (e.g., this week's HCUs, etc.)
This afternoon, I found a deduplication edge case that can cause account status register size to grow too large, so I've been resolving that until ~9:30 PM Central. The fix worked and didn't change data format.
Some Details and Next Steps
The deduplication during migration intentionally avoids 100% deduplication for some cases if the overhead of the deduplicated state would use more storage than allowing some duplicate keys.
I will resume wrapping up work on post-migration deduplication heuristics which can automatically deduplicate keys when accounts add new duplicate keys. Since accounts can have tens of thousands or over a hundred thousand existing keys, the data format and heuristics will obviously not strive for 100% detection rate of duplicate keys.
The latest data format already includes most (if not all) of the changes needed for post-migration deduplication heuristics but I want to wrap it up ASAP to avoid the need to change the data format again.
If we don't find any showstoppers or big changes needed this Friday, I will begin incorporating the deduplication logic and data format into a migration program and run it on m1 early next week to see what nworkers setting we want to use with m3. After that, I can look into running migration on m3 to see how long the unoptimized migration takes.
Currently, the estimated number of registers reduced is ~86 million.
- Continue implementing the key dedup migration (all outstanding technical issues have been resolved).
- Test migration on mainnet data to figure out how long it takes on an m1 machine.
- Continue implementing runtime (post spork) public key deduplication in FVM.
Thanks Vishal, I was going to post that update here but you beat me to it. 😄
For a bit more detail, I finished creating and running:
- PoC of migration deduplication
- PoC of post-spork (runtime) deduplication
Last Friday, Janez and I had a meeting so I can share the latest changes to the data format and deduplication so he can provide a sanity check.
Currently, I am working on turning PoCs into relatively production-ready code. This step involves some refactoring and taking parts of the PoC and integrating it into the onflow/flow-go code (e.g., flow's util program).
I'm adding more tests during this part of the work, which may identify issues that need to be fixed (like the one I found and fixed on Friday afternoon).
After that, I will run it on m1 to find out RAM requirements. Then we can run it on m3 with the appropriate nworkers setting for the available RAM.
Not mentioned today, but it would also be useful to gather before/after migration stats for each account to see which accounts might need special attention.
I continued adding tests until about 9 PM last night and I'm now comfortable running initial m1 migration before adding more tests.
Today, I'm working on wrapping up the work to take parts of the PoCs and integrating that code into onflow/flow-go code (e.g., the util program). This involves taking proof-of-concept code and making it suitable for production use (not just cut/paste).
Unless there are surprises, I will try to run migration on my m1 in benchnet today or tomorrow (Aug 15). After that, I will open PRs and run trial migration on the shared m3 using the settings identified by m1 run(s).
After running trial migrations on m1 and m3, I need to take the PoC runtime (post-spork) deduplication code, integrate it into onflow/flow-go, and make the code suitable for production use.
Public key deduplication (PKD) migration step (within the more time-consuming full migration) is only ~4.5 minutes on m1 using nworkers=64 🎉.
So, I'm holding off on optimizing it (unless requested).
Number of registers (aka payloads) reduced by PKD migration is as expected: 86 million.
I'm currently preparing to open the PR for PKD migration (maybe as soon as tomorrow). If needed, we can also try running PKD migration on a faster m3 using nworkers=64.
- Total migration time = 1 hour 8 minutes of which dedup taks 4mins25sec (on
m1machine). - Next, we will rerun the test on an m3 machine to see how much speed up we achieve.
Total migration time = 1 hour 8 minutes of which dedup taks 4mins25sec (on m1 machine). Next, we will rerun the test on an m3 machine to see how much speed up we achieve.
Migration on m3 using the same July 2025 mainnet snapshot and nworkers=64 flag:
- migration program: https://github.com/onflow/flow-go/pull/7738
- 53 minutes 4 seconds total migration duration
- load state from disk, etc.
- 2 minutes 19 seconds 🎉 for public key deduplication migration
- build trie, save state to disk, etc.
The util program running migration used 954 GB RAM (max RSS) on m3 with 1.9 TB RAM. This is more than m1 with 1.4 TB RAM, maybe due to Go (golang) garbage collection being less aggressive due to having more free RAM available on the system.
Busy with higher priority work, so I copied some of my wip draft writeup I had about this project's results and impact.
Actual Results (Oct. 22, 2025 Mainnet Spork)
For more technical details and impact (as time allows to document), see https://github.com/fxamacker/draft-notes-about-deduplicating-public-keys
Results contributed significant headroom for devops to update server configurations, test, etc. and reduce the size of servers to lower operational costs. 🎉
Not sure if related, but the temporary memory usage bump (mainnet27-EN2) on Oct. 23, 2025 might be from internal testing (heavy scripts) that was mentioned by a colleague just minutes before the start of the spike (orange line).
| Reduction | Notes | |
|---|---|---|
| Public Keys | 77.6 million (53.1%) | better than our goal of 46-48% 🎉 |
| Payloads (aka Registers) | 86.1 million (16%) | better than our goal of 65-68 million 🎉 |
| Payload Size | 8.9 GB | not estimated but better than private expectations 🎉 |
| MTrie Vertices | 210 million (16%) | matches expected 2x-3x payload count reduction 🎉 |
| MTrie Size | 28.8 GB (7.2%) | better than our goal of 19-20 GB 🎉 |
| EN Checkpoint Size | 21.7 GB (6.2%) | better than our goal of 19-20 GB 🎉 |
| EN RAM Usage | ~150 GB | not estimated but better than private expectations 🎉 |
| DB, caches, indexers, etc. | TBD | components handling payloads on AN, EN, etc. |
I designed the migration to be fast, so the key deduplication only took 5m36s using ten workers (nworkers=10) within the roughly 1 hour load/setup/run_migrations/save stages that are shared (amortized across all migrations during a spork).
A non-obvious result is the addition of more tests related to account keys in general (not specific to deduplication). Identifying and adding missing tests is needed for maintainable systems.
I heard speedup after the spork is better than expected, but multiple changes were deployed at the same time, and it is harder to separate this project's impact on speed from other projects.