Build event types using metadata's type registry

[NunoAlexandre]

The problem

Currently, users of the gsrpc library need to manually define all the event types the chain they connect with offers. If they fail to define any of the event types or forget to update them, whenever said event is encountered and a deserialisation is attempted, it will fail and the whole thing crumbles with an error such as:

panic: unable to find field Elections_NewTerm for event #0 with EventID [8 0]

Note that this may only happen way after having the gsrpc client running, and not at runtime.

Desired outcome

By supporting metadata v14, we know have a complete description of all the types. Ideally, a user of gsrpc would connect to any chain and be able to have the Event types of said chain defined on the fly and never have its execution failing mid way due to a missing field or event type all together.

Foreseeable edge cases

• What happens if gsrpc connects to a chain for a very long time and an update to the event types takes place in the meantime?

Proposals

Let’s brainstorm different proposal ideas. Their id does not imply any order.

Proposal A - Code generation

We offer a CLI command of sorts that connects to a given chain and generates an events.go.generated file that defines all the event types defined by said chain.

The user is then in charge of merging that code into the final events.go file (or whatever filename they use).

Advantages

• Hassle-free, complete event types for a specific chain
• Can be run independently and thus support different chains
• Even tho it only works for chains running metadata >= v14, it does not get in the way for older versions

Pitfalls

• Still require some error-prone user-action to merge the generated event types with the user codebase
• Requires running through the process every time an event type changes

Proposal B - Type-subset and fallback to Map

We have the user defining the sunset of events they are interested in working with and, instead of failing for unsupported events, we create a duck-type-like map with an event’s data that can still be used by the user.

Advantages

• Useful fallback, duck-typing approach to events the user is not interested in

Pitfalls

• Error-prone user-action by having to define the right types correctly
• Still requires the user to manually and update their type definitions
• Requires defining new structs for new Event types the user may want to start supporting
• Silent failing Say that a defined type T had a field a which is now called b; with this setup, we would fallback to use a map as a bad of data, which would not be handled as a T .

Useful links

• Original issue: Add missing event types to EventRecords · Issue #52 · centrifuge/go-substrate-rpc-client · GitHub

• ChainBridge/substrate.go at f0343fd61bfb49ff5a7f539ec824a258271f04c9 · centrifuge/ChainBridge · GitHub

• ChainBridge/events.go at master · centrifuge/ChainBridge · GitHub

[NunoAlexandre]

@mikiquantum @vedhavyas @mustermeiszer @tankofzion Appreciate your thoughts, specially regarding my formulation of the problem. Please let me know if I am seeing that correctly. Any other feedback is welcome as well!

[mikiquantum]

@NunoAlexandre Thanks for triggering this topic. Here are my two cents:

What happens if gsrpc connects to a chain for a very long time and an update to the event types takes place in the meantime?

We can have a fallback case where the client automatically re-fetches the metadata and update the types in case of parsing error.

I believe that we have enough information about the types that we do not need to go for a static approach like the one in Proposal A. Proposal B, seems more complete (and a bit more complex). In addition, some of the Pitfalls of Proposal B are common for Proposal A as well. Types that a user cares about will need to be defined up front at compilation time and if there is a change in the parameter name or type we will most likely fail deserializing it and user will need to revisit their types they are interested in. The main and big advantage is that we will not miss events that we care about.

We can def, at start-up time, fail if the user types defined are not present or compatible on the metadata.

[mustermeiszer]

I am also in favor of approach B. Although, I am at no means strong in opinion here, neither the best person to add valuable input. The metadata v14 of gsrpc will break with older versions anyways, right?

[vedhavyas]

Either approach is fine at the moment @NunoAlexandre. With generics being added to go in 1.18, I think it should pretty much replace the code gen part if there was any.

As for the events, we can get all the events from metadata, no? then user we can provide a default EventRecord or can replace with custom one, we cross those events with the ones in metadata. So using that and what @mikiquantum said, we can fail early and let them add manually to the custom records or bring up to this repo for core substrate frame events that we may have missed.

I think we should also start looking at using Go 1.18 generics support. I see that as a path forward to have strong typed library without using code gen or currenlty bringing their own types. Since metadata has everything, we can provide default types and fail for types we can expect from the user.

[NunoAlexandre]

Thank you @mikiquantum @mustermeiszer @vedhavyas for the valuable feedback, really 🙏

Features

With all of your feedback considered, I see the following features:

1. When connecting to a chain, check whether the user-defined event types are well implemented. If they are not, fail early with a useful error message that points the user to the problematic event/field.

2. For event types that we can match the metadata to a static user-defined type, we parse them to that corresponding strong type; For orphan event types, we fallback gracefully to a duck-typing approach by creating a map-like value with all the event fields and corresponding values.

Timeline

Thinking of breaking this work into chunks, I would think of these steps:

1.1 - Create a function that, given a metadata v14 type registry, creates an AST (or something along those lines) with all the event types' information.

1.2 - Create a function that can do the same as 1. but for the static user-defined event types.

1.3 - Plug 1 and 2 together so that we can verify that the user-defined event types are correct following to the metadata we fetched and fail fast if the verification fails.

2 - Implement a fallback-like Map value for the events that can't be parsed to a static, user-defined event type.

1.1, 1.2., and 1.3. are all sub-parts of the first use case described above, and half of the second one, but they can still be done in parts, so that we avoid sticking with very long PRs.

The block 2. is not fundamental and can be done as a follow up step since it's a fallback for a use case the user should not, in principle, be very interested in, or else the would have defined a statically defined type for the event at hand.

Note: @mustermeiszer raised a good point above regarding how metadata versions < v14 are handled. I don't see an issue in keeping the same behaviour for those older versions and introducing the new features for >= v14, but if you see any issue in doing that please point it out 👍

[csknk]

Thanks for working on this issue - I have bumped into problems with user-defined types several times now while interacting with Polkadot. @NunoAlexandre - what do you think is the best "manual" solution to use as a short-term workaround?

In other words, what is currently the best way to determine chain-specific custom types?

[cdamian]

Done in - https://github.com/w3f/Grants-Program/pull/1281