subspace
inject balances from json into chainspec
It is required to keep the chain spec file manageable with thousands of balances. The JSON file is NOT final as it only contains farmer rewards and no other allocations (team, fdn, etc.). Tested on devnet on Oct 9.
Code contributor checklist:
- [x] I have read, understood and followed contributing guide
Is there a way to just store the file hash in the node/git, and download the latest file when the node starts? Or distribute the file with the node build?
This is supposed to be a part of the chain spec, so it will be awkward to distribute it separately and I'd really prefer to not have it depend on the download from somewhere.
At the same time this large blob will end up copied into the actual mainnet chain spec, which in turn means our chain spec will be permanently huge. So we will remove the file shortly afterwards, but the ballooned chain spec will stay with us basically forever.
Repo size is a smaller concern for me, this is a really important information to have persisted in history. But I agree that we should wait for the very final version rather than adding multiple revisions of this large file.
I have added the file, even if not final, to test the "real deal" and wanted us to align on a preferred way to handle such case, because that is the size of the file we will be dealing with on mainnet. If we decide on format/workflow here, in the next PR I can just swap the allocations file for the final one. There is no way to avoid ballooned mainnet chainspec, however, these are not strictly necessary for testnet, so we could save some there.
Perhaps adding the file to a separate repro and using it as git submodule can help to reduce the mono repo size.
Perhaps adding the file to a separate repro and using it as git submodule can help to reduce the mono repo size.
It really doesn't if you need to clone submodule for code to compile anyway :slightly_smiling_face:
It also seems like all the amounts end in 15 zeroes, so maybe we could do that multiplication in code?
In the final file it will be closer to 8 trailing zeroes, but I would rather not introduce into the code assumptions about precision of underlying file.
There is no way to avoid ballooned mainnet chainspec, however, these are not strictly necessary for testnet, so we could save some there.
I think changing to arrays and removing spaces will help keep the size of the file down. (I don’t want to remove new lines, because that makes diffs much harder to see.)
Is there anything we need to test on testnet that would be missed by having a small chain spec file? It’s usually better to keep them as similar as possible, unless we’re testing mainnet the same as testnet before every release. (I’ve run into mainnet-only bugs in other distributed networks before.)
(I don’t want to remove new lines, because that makes diffs much harder to see.)
In the next iteration, the file will change basically on every line since the precision will be higher and post-merge there should not ever be a diff to genesis balances.
(I don’t want to remove new lines, because that makes diffs much harder to see.)
In the next iteration, the file will change basically on every line since the precision will be higher and post-merge there should not ever be a diff to genesis balances.
Sure, we can always run it through jq to format and then do a diff. So whatever you want to do to minify is fine with me!
The last commit reduces the json file significantly since we decided to round everyone to a whole token, so dropped all the trailing zeros, which also made deserialization simpler. This should now be at a much better state for review.
@teor2345 Please refer to the thread for the source data. A large amount of 1 is due to our recent decision to round everyone up to the nearest whole token, so everyone who got <1 was rounded to 1. The distribution is as expected and something we have observed in testnets (for example here). The total of the json should match the total of the sheet at 65,695,698.00 The json is still missing about a dozen big allocations, such as company treasury, foundation treasury, team grants etc
Yep, that checks out:
$ cat crates/subspace-node/src/genesis_allocations.json | jq 'map(.[1])|add'
65695698
Edit: the number of entries also checks out, minus the 9 zero-valued entries.
The only odd thing I found was a suspiciously round number of unique amounts.
$ ... | jq 'unique_by(.[1])|length' 4000Is it possible that the data generation script hit some kind of limit, and started ignoring different amounts once it had 4000 of them? (It seems unlikely, but so does having exactly 4000 unique reward amounts.)
This also checks out, I copied the values directly from the spreadsheet:
$ pbpaste | sort -u | wc -l
4001
(The 4001st value is zero.)
Since the spreadsheet is a rounded-up total of other spreadsheet fields, it's not scripted, so it can't have hit some kind of script limit. (And also the totals match, so we haven't skipped any value.)
The allocations should be final now and totaling to 650 000 000 (65 and seven zeros).
Squash on merge is not an option with merge queue unfortunately. It is best to squash explicitly to make sure we have a single version of the large file in the repo.
Ugh I f-ed up the squash. Going to close this PR and make a fresh one with a single commit