General Message Queue Pallet

[gavofyork]

Closes #6129

A general pallet for paginated queuing of messages from origins. This makes minimal assumptions on the nature of an origin and message and is generic over one with the other being a blob. Each origin has its own queue and queues are serviced in a fair manner.

This pallet is aware of, and well-bounded regarding, weights and in particular PoV weight. The weight_limit provided to service_queues will be efficiently respected.

The queuing mechanism is designed to be performant and efficient with regards to compute, MEL estimation and proof size and work well for an unpredictable assortment of message sizes. It is also designed to be easily configurable to allow for optimisation of page size.

This is intended to be used with an XCM adapter struct implementing the ProcessMessage trait, to be made at a later time in the Polkadot codebase as a single, general queue for XCMP, DMP, UMP and local dispatch/queuing.

TODO:

• [ ] Benchmarks & weights.
  • [ ] Refactor service_queue into a separate inner call and ensure weight is respected on ancillary compute/storage access. See Scheduler pallet for an example of this.
• [ ] Overweight dispatch.
• [ ] Stale/ancient page removal.
• [ ] Comprehensive tests.
• [ ] Docs.

[ggwpez]

Do you think it would make sense to have a StoragePagedList as primitive or just tailor it in this MR?

[gavofyork]

How could it be made a primitive?

[ggwpez]

bot rebase

[paritytech-processbot[bot]]

Rebased

[ggwpez]

bot rebase

[paritytech-processbot[bot]]

Rebased

[gavofyork]

Final reviews would be helpful @eskimor @ggwpez

[gavofyork]

bot rebase

[paritytech-processbot[bot]]

Rebased

[gavofyork]

@eskimor an approval would be good so we can move things along.

[eskimor]

@eskimor an approval would be good so we can move things along.

On it. Had a few other things to take care of and that code is a bit outside of my usual domain, so a proper review takes me a bit. I should be through by tomorrow though.

[ggwpez]

/cmd queue -c bench-bot $ pallet dev pallet_message_queue

[command-bot[bot]]

@ggwpez https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2138777 was started for your command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_message_queue. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 39-3547ab8e-4a55-43dc-a1c8-99410d2d403e to cancel this command or /cmd cancel to cancel all commands in this pull request.

[command-bot[bot]]

@ggwpez Command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_message_queue has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2138777 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2138777/artifacts/download.

[gavofyork]

@joao-paulo-parity done.

[gavofyork]

bot merge

[paritytech-processbot[bot]]

Error: 1 review requesting changes and 2 approving reviews by reviewers with write access.

[joao-paulo-parity]

bot merge

[paritytech-processbot[bot]]

Waiting for commit status.

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-release-analysis-v0-9-36/1529/1