[Features] Reintroduce sub-du

[ruseinov]

Resolves https://github.com/paritytech/substrate/issues/12409

At the moment this is just a copy-paste with updated dependencies.

• [ ] follow-up with devops: wrap this in a simple HTTP server application (external repo), deploy it on state.polkadot.network or something like that.

[ggwpez]

So this loops through the pallet+storage items and calculates their keys?
I wonder how many keys the first approach missed. There are special keys likes :CODE: which would not be included.
Maybe looping through the trie keys and doing a best-effort reverse lookup could find them.

I also vaguely remember a TS project which could be web-deployed, but cant find it now…

[kianenigma]

I wonder how many keys the first approach missed. There are special keys likes :CODE: which would not be included. Maybe looping through the trie keys and doing a best-effort reverse lookup could find them.

This only looks at the storage items that are in the metadata, so yeah any custom key is missed, and probably fine. Exposing well_known_keys over metadata seems totally reasonable. Although it won't really fix the problem.

[ruseinov]

bot rebase

[paritytech-processbot[bot]]

Rebased

[dvdplm]

• wrap this in a simple HTTP server application (external repo), deploy it on state.polkadot.network or something like that.

Perhaps a dumb question but what is the purpose of having this on the internet, as opposed to have people run a local binary? Just as a community utility thing?

[ruseinov]

Perhaps a dumb question but what is the purpose of having this on the internet, as opposed to have people run a local binary? Just as a community utility thing?

Yeah, just a remote api that would conveniently show latest storage of a given network. Possibly with some sort of dumb cache to make sure multiple requests within a short period of time are handled without issues.

[ruseinov]

bot rebase

[paritytech-processbot[bot]]

Rebased

[ruseinov]

bot rebase

[paritytech-processbot[bot]]

Rebased

[ggwpez]

Last time I tried it it would not connect because of some RPC error.
Did you try it yet @ruseinov ?

[ruseinov]

Last time I tried it it would not connect because of some RPC error. Did you try it yet @ruseinov ?

Not yet, getting to it now

[ruseinov]

bot rebase

[paritytech-processbot[bot]]

Rebased

[ruseinov]

@kianenigma I have removed sub-storage.

In the current implementation I am using wrappers done in remote-externalities builder to achieve faster results. Once all of this works - we can discuss having those helpers someplace else shared by the two.

But actually making a few methods public might not hurt remote-externalities much.

This is still a WIP, any suggestions welcome.

[kianenigma]

In the current implementation I am using wrappers done in remote-externalities builder to achieve faster results.

~~This is not really correct. All you need is:~~

~~1. an RPC call to get state_getMetadata~~ ~~2. an RPC call to get state_storageSize (or similar)~~

~~Both of which are provided neatly by substrate RPC utilities.~~

I should first review the code, which I will do asap.

[kianenigma]

@niklasad1 it might not be bad idea if you entirely take this over, as Roman is busy with other stuff (that's way more high priority).

[stale[bot]]

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

[ruseinov]

@niklasad1 it might not be bad idea if you entirely take this over, as Roman is busy with other stuff (that's way more high priority).

Yeah, not possible for me to proceed with this atm. But I'll get back to this if it's not done by the time I'm finished with the other stuff.

[kianenigma]

@harrysolovay would the idea behind this be a good tool to build with CAPI?

[harrysolovay]

This is absolutely something that could be implemented as a Capi pattern lib. Let's try it out & follow up here.