Replace public keys with those compressed

[s8sato]

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/01/08 19:58] How many bytes used per account stored in memory in wsv?

S. Sato, [2025/01/09 0:20] Approximately 2.2GiB resident memory difference is observed for 2^20 (1Mi) account-asset pairs, which means 2200~2300B for a pair. Code (https://github.com/s8sato/iroha/commit/17bfacd8d2e0436b050251546b387cde076a39e7)

Assumptions:

• No difference except in the world state: compared in the initial state of 0 blocks
• Every account accompanies one asset

I might investigate further to be more precise

S. Sato, [2025/01/09 0:47] We might have to store multihash instead of PublicKey as well as to dedup ids

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/01/09 1:12] Do we store compressed public keys?

S. Sato, [2025/01/09 1:28] I guess no, it consumes 8+296 bytes while e.g. ed25519 multihash is essentially 32 bytes

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/01/09 14:18] We can compress the public key I guess as we just need a bit showing if y is even or odd and not the whole y coord

[s8sato]

In addition to PublicKey size and duplicate IDs between keys and values, it would be a major factor in memory usage that mv::storage::Storage.revert holds a snapshot to handle soft forks

[s8sato]

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/03/13 23:51] Regarding addresses and keys, we should use compressed public keys and generate addresses with checksums and network bytes, so there are a few tasks associated with account structure.

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/03/13 23:57] By compressed, I mean that on the eliptic curve, you don't need to store the entire y value, just the x + a bit that shows if it is positive y or negative y (though it could also depend on the curve used).

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/03/13 23:58] Like in secp256k1 used by bitcoin:

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/03/13 23:58]

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/03/13 23:58] the curve is mirrored on the y axis, so if you know the x value and you know if y is positive or negative, then you can calculate y, so bitcoin doesn't store the full public keys anymore, just the "compressed" keys

I don’t ask for crypto. M◉⃤|<𓊖༒𓊖 Takemiya 武宮誠 (Sora.org & SORAMITSU), [2025/03/13 23:59] It saves a lot of space actually, but at the cost of some computation of course (but it is pretty simple math, at least for secp256k1. Not sure about ed25519 or other arbirtrary curves we may support)

[dima74]

So currently Account struct consumes ~344 bytes, and when stored in World it takes ~1006 bytes per account. (In World it takes more because of key/value duplication, and because of concread/BTreeMap internal structure)

---

Potential memory optimizations:

• Optimize PublicKey struct size:
  • Store PublicKey in compact form (#5445)
  • Inline algorithm in PublicKeyCompact (#5452)
  • ? Make size of PublicKey 40 bytes, first two variants (Ed25519 and Secp256k1) should be stored inline (without heap), other two variants (BlsNormal and BlsSmall) will be stored on heap.
  • ? Use Arc for PublicKeyCompact::algorithm_and_payload
  • ? Use custom ConstVec implementation for PublicKeyCompact::algorithm_and_payload (there is no need to store capacity, we can get it from algorithm)
  • ? Introduce map World::public_keys of type something like Map<PublicKey, u64> and use those u64 id in all other maps like World::accounts
• Remove key/values duplication in World::accounts and similar maps (#5463)

---

Analysis of Account and PublicKey structs:

// total 344
struct Account {
    // total 320
    id: AccountId,
    // self 24
    metadata: Metadata,
}

// self 24 + heap 296 = total 320
struct AccountId {
    // self 16
    domain: DomainId,
    // self 8 + heap 296
    signatory: PublicKey,
}
    
// self 8 + heap 296
struct PublicKey(Box<PublicKeyInner>);

// 296
enum PublicKeyInner {
    // 192  (encoded - 32)
    Ed25519(ed25519::PublicKey),
    // 104  (encoded - 33)
    Secp256k1(secp256k1::PublicKey),
    // 144  (encoded - 48)
    BlsNormal(bls::BlsNormalPublicKey),
    // 288  (encoded - 96)
    BlsSmall(bls::BlsSmallPublicKey),
}

[dima74]

Copying Sato comment from #5445 for reference:

Great—please also revisit these prior discussions:

• Combining keys and values only at the interface: Redundant identifiers in storage values #5034
• Separating primary keys and foreign keys, treating them as reference counters: Data integrity in World #4672

[dima74]

Overview of memory usage after optimizations:

• Structs in World. This is similar to usage of real Iroha.
  • Account — 336 bytes per account
  • Asset — 422 bytes per asset
  • Nft — 406 bytes per nft
• Struct as is. This is just for comparison. In World structs consume more because they are stored in BTreeMap, where keys are stored multiple times in intermediate nodes.
  • Account — 108 bytes
  • Asset — 132 bytes
  • Nft — 174 bytes