go-substrate-rpc-client Add dynamic extrinsic signing support

The changes in this PR add a new type of extrinsic that can have a dynamic payload and signature, based on the information in the metadata.

Test script can be found here.

Jul 03 '24 15:07 cdamian

Could you consider adding a version of SubmitExtrinsic that accepts a extrinsic.DynamicExtrinsic as well?

Jul 04 '24 06:07 henrikdent

@henrikdent I think the final version will be a breaking change where we completely drop support for the old version, we have not decided yet since we still have to test this with other chains such as AssetHub.

Jul 10 '24 16:07 cdamian

Tried to use this branch, but get an error:

panic: creating payload: signed extension 'StorageWeightReclaim' is not supported

while doing this:

	err = ext.Sign(
		keyPair,
		meta,
		extrinsic.WithEra(types.ExtrinsicEra{IsImmortalEra: true}, genesisHash),
		extrinsic.WithNonce(types.NewUCompactFromUInt(uint64(accountInfo.Nonce))),
		extrinsic.WithTip(types.NewUCompactFromUInt(0)),
		extrinsic.WithSpecVersion(rv.SpecVersion),
		extrinsic.WithTransactionVersion(rv.TransactionVersion),
		extrinsic.WithGenesisHash(genesisHash),
		extrinsic.WithMetadataMode(extensions.CheckMetadataModeDisabled, extensions.CheckMetadataHash{Hash: types.NewEmptyOption[types.H256]()}),
	)

Jul 11 '24 08:07 akme

Tried to use this branch, but get an error:

panic: creating payload: signed extension 'StorageWeightReclaim' is not supported

while doing this:

	err = ext.Sign(
		keyPair,
		meta,
		extrinsic.WithEra(types.ExtrinsicEra{IsImmortalEra: true}, genesisHash),
		extrinsic.WithNonce(types.NewUCompactFromUInt(uint64(accountInfo.Nonce))),
		extrinsic.WithTip(types.NewUCompactFromUInt(0)),
		extrinsic.WithSpecVersion(rv.SpecVersion),
		extrinsic.WithTransactionVersion(rv.TransactionVersion),
		extrinsic.WithGenesisHash(genesisHash),
		extrinsic.WithMetadataMode(extensions.CheckMetadataModeDisabled, extensions.CheckMetadataHash{Hash: types.NewEmptyOption[types.H256]()}),
	)

@akme we were not aware of that signing extension, can you please let us know what chain this is so we can look into it?

Jul 11 '24 11:07 cdamian

@akme we were not aware of that signing extension, can you please let us know what chain this is so we can look into it?

It happens with wss://rococo-asset-hub-rpc.polkadot.io when I try to sign this: types.NewCall(meta, "Assets.transfer", assetIdSerialized, bob, types.NewU128(*bal)) or types.NewCall(meta, "Balances.transfer_keep_alive", bob, types.NewUCompact(bal))

Jul 11 '24 11:07 akme

@akme added another signing option WithAssetID that you should add when signing the extrinsic.

I tested it successfully using a simple remark call on the chain that you mentioned.

In a modified version of the above mentioned test script I added:

...

err = ext.Sign(
		BobKeyRingPair,
		meta,
		extrinsic.WithEra(types.ExtrinsicEra{IsImmortalEra: true}, genesisHash),
		extrinsic.WithNonce(types.NewUCompactFromUInt(uint64(accountInfo.Nonce))),
		extrinsic.WithTip(types.NewUCompactFromUInt(0)),
		extrinsic.WithSpecVersion(rv.SpecVersion),
		extrinsic.WithTransactionVersion(rv.TransactionVersion),
		extrinsic.WithGenesisHash(genesisHash),
		extrinsic.WithMetadataMode(extensions.CheckMetadataModeDisabled, extensions.CheckMetadataHash{Hash: types.NewEmptyOption[types.H256]()}),
		extrinsic.WithAssetID(types.NewEmptyOption[types.AssetID]()),
	)
	
...

Please let me know if you encounter any issues.

Jul 11 '24 23:07 cdamian

It can't work either

@akme added another signing option WithAssetID that you should add when signing the extrinsic.

I tested it successfully using a simple remark call on the chain that you mentioned.

In a modified version of the above mentioned test script I added:
...

err = ext.Sign(
		BobKeyRingPair,
		meta,
		extrinsic.WithEra(types.ExtrinsicEra{IsImmortalEra: true}, genesisHash),
		extrinsic.WithNonce(types.NewUCompactFromUInt(uint64(accountInfo.Nonce))),
		extrinsic.WithTip(types.NewUCompactFromUInt(0)),
		extrinsic.WithSpecVersion(rv.SpecVersion),
		extrinsic.WithTransactionVersion(rv.TransactionVersion),
		extrinsic.WithGenesisHash(genesisHash),
		extrinsic.WithMetadataMode(extensions.CheckMetadataModeDisabled, extensions.CheckMetadataHash{Hash: types.NewEmptyOption[types.H256]()}),
		extrinsic.WithAssetID(types.NewEmptyOption[types.AssetID]()),
	)
	
...
Please let me know if you encounter any issues.

Jul 12 '24 06:07 pingpingpang001

Please let me know if you encounter any issues.

It worked for Balances.transfer_keep_alive, I was able to send ROC between accounts. It didn't work for Assets.transfer:

Ext - 0x8102840066f34a40ef3d840e5354e1c7c3362179ab03d286d8ad10ec54fbaab1a4de415401348c8e5412b800d3ac62152676a43a75efebeb08c1bb3aa4223b293ac7fa6f126ac1c9f15b9ff0262002c74fb1d682d6bebb7eafb0352952016303cf03c8158c0018000000320810890000000011928a7729d09635de0b8a9d8e0e528aa910b00057e1cffa1ca6b9e3e279125f10270000000000000000000000000000
panic: Verification Error: Runtime error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4f79d7 - <unknown>!rust_begin_unwind
    1: 0x4a9f17 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x1b11b0 - <unknown>!TaggedTransactionQueue_validate_transaction

should I pass an assetId here extrinsic.WithAssetID(types.NewEmptyOption[types.AssetID]())? another question is how to get a hash/txid of extrinsic?

Jul 12 '24 07:07 akme

Would you create a gist about test Balances.transfer_keep_alive ? Mine get the same error on this method. Thank you so much.

Please let me know if you encounter any issues.

It worked for Balances.transfer_keep_alive, I was able to send ROC between accounts. It didn't work for Assets.transfer:
Ext - 0x8102840066f34a40ef3d840e5354e1c7c3362179ab03d286d8ad10ec54fbaab1a4de415401348c8e5412b800d3ac62152676a43a75efebeb08c1bb3aa4223b293ac7fa6f126ac1c9f15b9ff0262002c74fb1d682d6bebb7eafb0352952016303cf03c8158c0018000000320810890000000011928a7729d09635de0b8a9d8e0e528aa910b00057e1cffa1ca6b9e3e279125f10270000000000000000000000000000
panic: Verification Error: Runtime error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4f79d7 - <unknown>!rust_begin_unwind
    1: 0x4a9f17 - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x1b11b0 - <unknown>!TaggedTransactionQueue_validate_transaction
should I pass an assetId here extrinsic.WithAssetID(types.NewEmptyOption[types.AssetID]())? another question is how to get a hash/txid of extrinsic?

Jul 12 '24 08:07 pingpingpang001

Would you create a gist about test Balances.transfer_keep_alive ? Mine get the same error on this method. Thank you so much.

func Example_makeASimpleTransfer(keyPair signature.KeyringPair) {
	// This sample shows how to create a transaction to make a transfer from one an account to another.

	// Instantiate the API
	api, err := gsrpc.NewSubstrateAPI("wss://rococo-asset-hub-rpc.polkadot.io")
	if err != nil {
		panic(err)
	}

	meta, err := api.RPC.State.GetMetadataLatest()
	if err != nil {
		panic(err)
	}

	// Create a call, transferring 12345 units to Bob
	bob, err := types.NewMultiAddressFromHexAccountID("0x8eaf04151687736326c9fea17e25fc5287613693c912909cb226aa4794f26a48")
	if err != nil {
		panic(err)
	}

	// 1 unit of transfer
	bal, ok := new(big.Int).SetString("1000000000000", 10)
	if !ok {
		panic(fmt.Errorf("failed to convert balance"))
	}

	c, err := types.NewCall(meta, "Balances.transfer_keep_alive", bob, types.NewUCompact(bal))
	if err != nil {
		panic(err)
	}

	ext := extrinsic.NewDynamicExtrinsic(&c)

	genesisHash, err := api.RPC.Chain.GetBlockHash(0)
	if err != nil {
		panic(err)
	}

	rv, err := api.RPC.State.GetRuntimeVersionLatest()
	if err != nil {
		panic(err)
	}

	key, err := types.CreateStorageKey(meta, "System", "Account", keyPair.PublicKey)
	if err != nil {
		panic(err)
	}

	var accountInfo types.AccountInfo
	ok, err = api.RPC.State.GetStorageLatest(key, &accountInfo)
	if err != nil || !ok {
		panic(err)
	}

	err = ext.Sign(
		keyPair,
		meta,
		extrinsic.WithEra(types.ExtrinsicEra{IsImmortalEra: true}, genesisHash),
		extrinsic.WithNonce(types.NewUCompactFromUInt(uint64(accountInfo.Nonce))),
		extrinsic.WithTip(types.NewUCompactFromUInt(0)),
		extrinsic.WithSpecVersion(rv.SpecVersion),
		extrinsic.WithTransactionVersion(rv.TransactionVersion),
		extrinsic.WithGenesisHash(genesisHash),
		extrinsic.WithMetadataMode(extensions.CheckMetadataModeDisabled, extensions.CheckMetadataHash{Hash: types.NewEmptyOption[types.H256]()}),
		extrinsic.WithAssetID(types.NewEmptyOption[types.AssetID]()),
	)

	if err != nil {
		panic(err)
	}



	encodedExt, err := codec.EncodeToHex(ext)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Ext - %s\n", encodedExt)

	sub, err := api.RPC.Author.SubmitAndWatchDynamicExtrinsic(ext)
	if err != nil {
		panic(err)
	}
	defer sub.Unsubscribe()

	for {
		select {
		case st := <-sub.Chan():
			extStatus, _ := st.MarshalJSON()
			fmt.Printf("Status for transaction - %s\n", string(extStatus))
		case err := <-sub.Err():
			panic(err)
		}
	}
}

Keep in mind that your go.mod should point to the commit from this branch feature/add-dynamic-extrinsic-signing-support. I've just added this line at the end of my go.mod file and made go mod tidy:

replace github.com/centrifuge/go-substrate-rpc-client/v4 => github.com/centrifuge/go-substrate-rpc-client/v4 v4.2.1-0.20240711233432-c7949e1f6b9a

Jul 12 '24 08:07 akme

Your code works for Balances.transfer_allow_death on westend. Westend metadata don't have Assets.transfer . I'll check the difference between your and mine . Thank you very much

Would you create a gist about test Balances.transfer_keep_alive ? Mine get the same error on this method. Thank you so much.

func Example_makeASimpleTransfer(keyPair signature.KeyringPair) {
	// This sample shows how to create a transaction to make a transfer from one an account to another.

	// Instantiate the API
	api, err := gsrpc.NewSubstrateAPI("wss://rococo-asset-hub-rpc.polkadot.io")
	if err != nil {
		panic(err)
	}

	meta, err := api.RPC.State.GetMetadataLatest()
	if err != nil {
		panic(err)
	}

	// Create a call, transferring 12345 units to Bob
	bob, err := types.NewMultiAddressFromHexAccountID("0x8eaf04151687736326c9fea17e25fc5287613693c912909cb226aa4794f26a48")
	if err != nil {
		panic(err)
	}

	// 1 unit of transfer
	bal, ok := new(big.Int).SetString("1000000000000", 10)
	if !ok {
		panic(fmt.Errorf("failed to convert balance"))
	}

	c, err := types.NewCall(meta, "Balances.transfer_keep_alive", bob, types.NewUCompact(bal))
	if err != nil {
		panic(err)
	}

	ext := extrinsic.NewDynamicExtrinsic(&c)

	genesisHash, err := api.RPC.Chain.GetBlockHash(0)
	if err != nil {
		panic(err)
	}

	rv, err := api.RPC.State.GetRuntimeVersionLatest()
	if err != nil {
		panic(err)
	}

	key, err := types.CreateStorageKey(meta, "System", "Account", keyPair.PublicKey)
	if err != nil {
		panic(err)
	}

	var accountInfo types.AccountInfo
	ok, err = api.RPC.State.GetStorageLatest(key, &accountInfo)
	if err != nil || !ok {
		panic(err)
	}

	err = ext.Sign(
		keyPair,
		meta,
		extrinsic.WithEra(types.ExtrinsicEra{IsImmortalEra: true}, genesisHash),
		extrinsic.WithNonce(types.NewUCompactFromUInt(uint64(accountInfo.Nonce))),
		extrinsic.WithTip(types.NewUCompactFromUInt(0)),
		extrinsic.WithSpecVersion(rv.SpecVersion),
		extrinsic.WithTransactionVersion(rv.TransactionVersion),
		extrinsic.WithGenesisHash(genesisHash),
		extrinsic.WithMetadataMode(extensions.CheckMetadataModeDisabled, extensions.CheckMetadataHash{Hash: types.NewEmptyOption[types.H256]()}),
		extrinsic.WithAssetID(types.NewEmptyOption[types.AssetID]()),
	)

	if err != nil {
		panic(err)
	}



	encodedExt, err := codec.EncodeToHex(ext)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Ext - %s\n", encodedExt)

	sub, err := api.RPC.Author.SubmitAndWatchDynamicExtrinsic(ext)
	if err != nil {
		panic(err)
	}
	defer sub.Unsubscribe()

	for {
		select {
		case st := <-sub.Chan():
			extStatus, _ := st.MarshalJSON()
			fmt.Printf("Status for transaction - %s\n", string(extStatus))
		case err := <-sub.Err():
			panic(err)
		}
	}
}

Keep in mind that your go.mod should point to the commit from this branch feature/add-dynamic-extrinsic-signing-support. I've just added this line at the end of my go.mod file and made go mod tidy:

replace github.com/centrifuge/go-substrate-rpc-client/v4 => github.com/centrifuge/go-substrate-rpc-client/v4 v4.2.1-0.20240711233432-c7949e1f6b9a

Jul 12 '24 08:07 pingpingpang001

@akme you can find a gist for executing the Assets.transfer call here.

You can also get a signed extrinsic from - https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo-asset-hub-rpc.polkadot.io#/extrinsics by selecting the extrinsic that you want, adding the arguments to the call, go to "Submit Transaction", disable "sign and submit", set lifetime to 0, hit "Sign (no submission)". You will then get an encoded signed extrinsic. This is what I usually do when debugging these kind of issues.

Jul 12 '24 11:07 cdamian

@akme you can find a gist for executing the Assets.transfer call here. Awesome, your gist worked! Thank you!

You can also get a signed extrinsic from - https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo-asset-hub-rpc.polkadot.io#/extrinsics by selecting the extrinsic that you want, adding the arguments to the call, go to "Submit Transaction", disable "sign and submit", set lifetime to 0, hit "Sign (no submission)". You will then get an encoded signed extrinsic. This is what I usually do when debugging these kind of issues. Thanks for the info!

Could you please help me to understand a way to get an ID of extrinsic that I send with the help of your gist? I need to sign and send, then watch for exact extrinsic and create an url, e.g. for an explorer.

Jul 12 '24 11:07 akme

Could you please help me to understand a way to get an ID of extrinsic that I send with the help of your gist? I need to sign and send, then watch for exact extrinsic and create an url, e.g. for an explorer.

@akme not sure I understand your request, but feel free to open a different issue with more details though.

Jul 12 '24 12:07 cdamian

@akme not sure I understand your request, but feel free to open a different issue with more details though.

Sure, thanks :)

Jul 12 '24 12:07 akme

Another new SignedExtension "PrevalidateAttests" in v1.2.6 for polkadot

extrinsic: {
        type: 858
        version: 4
        signedExtensions: [
          {
            identifier: CheckNonZeroSender
            type: 860
            additionalSigned: 35
          }
          {
            identifier: CheckSpecVersion
            type: 861
            additionalSigned: 4
          }
          {
            identifier: CheckTxVersion
            type: 862
            additionalSigned: 4
          }
          {
            identifier: CheckGenesis
            type: 863
            additionalSigned: 12
          }
          {
            identifier: CheckMortality
            type: 864
            additionalSigned: 12
          }
          {
            identifier: CheckNonce
            type: 866
            additionalSigned: 35
          }
          {
            identifier: CheckWeight
            type: 867
            additionalSigned: 35
          }
          {
            identifier: ChargeTransactionPayment
            type: 868
            additionalSigned: 35
          }
          {
            identifier: PrevalidateAttests
            type: 869
            additionalSigned: 35
          }
          {
            identifier: CheckMetadataHash
            type: 870
            additionalSigned: 33
          }
        ]
      }

Jul 15 '24 04:07 wooyeeyii

Polkadot need SignedExtensionName PrevalidateAttests

Jul 15 '24 07:07 pingpingpang001

Polkadot need SignedExtensionName PrevalidateAttests

I am looking into this now and will post an update as soon as I have it.

Jul 22 '24 10:07 cdamian

Polkadot need SignedExtensionName PrevalidateAttests

Please try again using the latest version here.

Jul 22 '24 12:07 cdamian

Hey @cdamian I am using the method "NetworkMembership.nominate" in creating a NewCall for a custom polkadot chain and it is throwing panic: creating payload: signed extension 'CheckNetworkMembership' is not supported, can you help me with this out in this as I think this PR could also help with dynamic support to custom call methods. Thanks!

Jul 23 '24 21:07 kartikaysaxena

Hey @cdamian I am using the method "NetworkMembership.nominate" in creating a NewCall for a custom polkadot chain and it is throwing panic: creating payload: signed extension 'CheckNetworkMembership' is not supported, can you help me with this out in this as I think this PR could also help with dynamic support to custom call methods. Thanks!

@kartikaysaxena Hey, can you please tell me what chain this is so I can check?

Jul 24 '24 08:07 cdamian

Hey @cdamian I am using the method "NetworkMembership.nominate" in creating a NewCall for a custom polkadot chain and it is throwing panic: creating payload: signed extension 'CheckNetworkMembership' is not supported, can you help me with this out in this as I think this PR could also help with dynamic support to custom call methods. Thanks!

@kartikaysaxena Hey, can you please tell me what chain this is so I can check?

https://github.com/dhiway/cord

Jul 24 '24 09:07 kartikaysaxena

@kartikaysaxena is there a websocket address that I can use in the polkadot JS app, such as - https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fpolkadot.api.onfinality.io%2Fpublic-ws#/explorer?

Jul 24 '24 12:07 cdamian

Here it is @cdamian wss://sparknet.cord.network/

Jul 24 '24 17:07 kartikaysaxena

Hi @kartikaysaxena , according to the signed extra found here, there are no additional fields required in the payload or signature. Please test using the latest version here and let me know if you encounter any issues.

~I will try to test this locally using the guide found on the github repo you provided earlier.~ Tested successfully using the local chain.

Jul 24 '24 20:07 cdamian

@cdamian I tested using this PR, and it gives

panic: Verification Error: Runtime error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x28e21e - <unknown>!rust_begin_unwind
    1: 0x2571ea - <unknown>!core::panicking::panic_fmt::he07f4fcfc0e8e78f
    2: 0x1d8120 - <unknown>!TaggedTransactionQueue_validate_transaction

Also to let you know that we actually directly don't call NetworkMembership.Nominate but wrap it inside a sudo call for example

	call, err := types.NewCall(meta, "NetworkMembership.nominate",keyringPair.PublicKey)
	if err != nil {
		panic(err)
	}

	sudoCall, err := types.NewCall(meta, "Sudo.sudo", call)
	if err != nil {
		panic(err)
	}

Also can you please share the demo code how you did it?

Chain Logs panicked at /Users/Kartikay2/Downloads/target/cord/runtimes/loom/src/lib.rs:1307:1: Bad input data provided to validate_transaction: Codec error

Jul 25 '24 15:07 kartikaysaxena

@kartikaysaxena This gist is what I used when testing this, with a local docker container:

docker run -d -p 9944:9944 -p 30333:30333 -p 9933:9933 -p 9615:9615 -v cord:/tmp/data dhiway/cord:develop --dev --rpc-external --rpc-cors all

Please note that the NetworkMembership.nominate call you're trying to make takes 2 arguments, as per the extrinsic found here.

Wrapping the call in sudo seems OK, we did something similar for batches here.

Jul 25 '24 18:07 cdamian

Thank you very much @cdamian! It now works.

Jul 25 '24 21:07 kartikaysaxena

hey @cdamian, I've encountered the CheckMetadataHash issue as well and following all the conversations till here, just want to know what is the current status of the fix? as I tried the latest commit of feature/add-dynamic-extrinsic-signing-support branch and still got the same error. Thank you 🙏

The error for me is:

Error:      	Expected nil, but got: &fmt.wrapError{msg:"submission of extrinsic failed: Verification Error: Runtime error: Execution failed: Runtime panicked: Bad input data provided to validate_transaction: Could not decode `UncheckedExtrinsic.0`:\n\tCould not decode `CheckMetadataHash::mode`:\n\t\tCould not decode `Mode`, variant doesn't exist\n", err:(*rpc.jsonError)(0x14002706210)}

Aug 08 '24 15:08 freddyli7

@freddyli7 can you please share a URI for this chain so we can test this?

Aug 08 '24 15:08 cdamian