Have all the special storage keys in a single place

[xlc]

I am working on a feature of Chopsticks to decode all the storage keys. There are some keys requires special handling as they are not included in metadata.

Currently we have

https://github.com/paritytech/polkadot-sdk/blob/86228fa45e4c64642f7210cf44c40cc84ae17537/substrate/primitives/storage/src/lib.rs#L192-L216

https://github.com/paritytech/polkadot-sdk/blob/86228fa45e4c64642f7210cf44c40cc84ae17537/substrate/frame/support/src/traits/metadata.rs#L163-L167

https://github.com/paritytech/polkadot-sdk/blob/86228fa45e4c64642f7210cf44c40cc84ae17537/substrate/frame/support/src/storage/transactional.rs#L34-L37

https://github.com/paritytech/polkadot-sdk/blob/86228fa45e4c64642f7210cf44c40cc84ae17537/substrate/primitives/consensus/grandpa/src/lib.rs#L65-L67

https://github.com/paritytech/polkadot-sdk/blob/86228fa45e4c64642f7210cf44c40cc84ae17537/polkadot/primitives/src/v6/mod.rs#L242-L251

Please let me know if I missed anything

This makes it really hard to implement a generic and all-catching storage decoder. Can we move all the Substrate well known keys to sp_core::storage::well_known_keys? The polkadot one are all in a single place so that's good.

[xlc]

The key format is also inconsistent. Some of them are :xxx and some of them are :xxx: but I guess we are already too late to change them.

We should also update the docs to include encoded hex and the expected value type.

[bkchr]

The polkadot one are all in a single place so that's good.

The Polkadot ones are not special keys, these are just the normal FRAME storage keys.

[xlc]

:relay_dispatch_queue_remaining_capacity is definitely a special key but yeah everything else seems just storage keys encoded. Maybe we could refactor them in such why we code the string and have macro to convert them to hex in compile time? So it is more obvious where the hex is coming from.

[bkchr]

:relay_dispatch_queue_remaining_capacity

Ahh you got me :P

Maybe we could refactor them in such why we code the string and have macro to convert them to hex in compile time?

In a perfect way we would not need them at all. Or better abstracted. But we already have the hash macros, so it should actually work.

[ggwpez]

I think we should use storage_alias to have type safety.

[dharjeezy]

is this issue just all about creating a cratewell_known_keys in sp_storage and moving the keys specified into it and then refractoring? is there anything else to be done asides that? @ggwpez

[ggwpez]

It seems a bit odd to me that the keys for runtime specific things are specified in sp_storage and not a frame crate.
But since sp-storage cannot depend on frame, we could re-export all well known keys somewhere in frame-support.

Not sure if that is ideal, better ideas?

and then refractoring

Please first add a deprecation warning. If you want to remove something.

[dharjeezy]

@ggwpez can we then put all these keys in the frame support storage crate?

[ggwpez]

We can re-export them there yes, but they are also needed in lower level crates, so they need to stay in there as well.

I guess a common re-export module is the best we can do now.

[dharjeezy]

We can re-export them there yes, but they are also needed in lower level crates, so they need to stay in there as well.

I guess a common re-export module is the best we can do now.

i don't seem to quite get what you mean by a common re-export module? what do you mean? @ggwpez

[ggwpez]

mod well_known_keye {
    pub use sp_storage::...
    pub use sp_consensus_grandpa::...
}

and so on. Although it would probably not cover the ones from polkadot primitives, since what would create a dependency cycle again.

Not sure what @bkchr was thinking.

[bkchr]

It seems a bit odd to me that the keys for runtime specific things are specified in sp_storage and not a frame crate.

Because these keys are not FRAME specific. These are Substrate well known keys for communicating data to the node. Things like the code should not require to first call into the runtime.

[bkchr]

Generally, maybe just adding a page in the polkadot sdk docs is a better idea. I don't see any reason for adding them somewhere to re-export them, while this ticket is just about being able to find these keys/having a place to look them up.

[dharjeezy]

so, no need to get this done anymore? @bkchr

[bkchr]

As I said above, I think there should be a guide/reference doc added to the sdk docs.

[aurexav]

Recently, I create this ~~removed~~.

I'm interesting to add the well know storage keys there. It might be an option.

And also #324. Subscan team is struggling with the custom pallet account across different chains. 🤷🏻‍♂️

[bkchr]

And also #324. Subscan team is struggling with the custom pallet account across different chains. 🤷🏻‍♂️

But can they not read the PalletId from the constants?

[aurexav]

But can they not read the PalletId from the constants?

Yep, that would be the plan B. But a static registry is better.

[bkchr]

But a static registry is better.

But why?