Ergonomic multi-block operations

[kianenigma]

I have started writing multi-block code in a few occasions already. There are multiple ways to abstract it, but this is the ultimate one:

fn multi_block_function(n: T::BlockNumber) -> Weight {
	// this only executes once. 
	let pre_process = foo(); 
		
	// will generate some storage items that stores the current block number,
	// with some match statements that re-executes this code for the coming 10 blocks. 
	#[pallet::multi_block]
    for i in 0..10 {
		// this is executed 10 times, over the course of 10 blocks
		// variable `n` should be shadowed to the correct value here.
		let _ = bar(n)?;
	} 
	
	// this is exeucted once, right after the last iteration.
	let post_process = baz();
}

This is merely an idea at this stage and there are a lot of caveats, which should be discussed in depth before an attempt at implementation. One I have more concrete ideas about this I will move it to the frame vision project.

[kianenigma]

@Ank4n

[xlc]

What we really want is async operations. We already have proper abstraction for it with years of experiences and I want to make sure we don't repeat the same mistakes and introduce more race condition bugs

[ggwpez]

What we really want is async operations.

What kind of operations? For example iterators on StorageMaps? So that StorageMap::iter_keys().map( would be multi-block operation.
Or rather wrapping a whole migration code block in some kind of async macro and using storage operations as yield points?

multi_block! {
    // migration code that will be split into multiple blocks.
    StorageMap::translate(t);
}

I think this is more complicated than having some kind of block number counter.
What are the race conditions do you see with this approach?

[xlc]

The main reason we want to split some operations to multiple blocks is due weight limit, not just because we want to do it due to business reason (which I think is out of scope for this issue).

Before a language natively supports async operations, we use callbacks to implement it. So it will be something like this

fn do_work(i: u32) {
  // do something with i
}

fn get_work() -> Option<u32> {
 // return next work
}

fn get_and_do_work() -> Weight {
  if let Some(work) = get_work() {
    do_work(work)
     multi_block_helper::execute_or_queue(Call::get_and_do_work)
    return weight_info::get_and_do_work(work)
  }
  return weight_info::get_work()
}

fn migration() {
   multi_block_helper::execute_or_queue(Call::get_and_do_work)
}

Later, if we somehow get async support and some weight integration, we can make it

#[weight = weight_info::do_work(i)]
fn do_work(i: u32) {
  // do something with I
}

#[weight = weight_info::get_work()]
fn get_work() -> Option<u32> {
 // return next work
}

async fn migration() {
  while let Some(work) = get_work() {
     do_work(work).await // check if remaining weight is enough, then execute, or defer it to next block
  }
   // code here will be executed after all the work is completed
}

[ggwpez]

Okay so there are a few other cases where multi-block or multi-call aka. async operations would be useful:
https://github.com/paritytech/substrate/pull/12367#discussion_r991535381 (uses remove_prefix) https://github.com/paritytech/substrate/pull/12310 (deletes a StorageMap which could produce a too high weight for one call/block)

So it is not limited to migrations where this problem arises…