[RFC][Planned Feature] Programmable Transactions

[tnowacki]

I'll try to keep this rather brief. There are probably a ton of possible things here that I haven't explained very well so please ask questions! And there are a ton of possible extensions or small changes we could make, so please give feedback on that too :)

Proposal

Programmable Transactions are a new type of transaction that will replace both batch transactions and normal transactions (with the exception of special system transactions). These transactions will allow for a series of Commands (mini transactions of sorts) to be executed, where the results of commands can be used in following commands. The effects of these commands will be "all-or-nothing", meaning if one command fails, they all fail; just like is done today inside of Move calls.

The goal here is that this would serve as an easy target for SDK tools to implement the Pay* transactions, to apply limits to Coin inputs, to construct non-primitive Move inputs, to use Shared objects more than once, etc. The goal though is still to also be simple. We are not going to target repetition or looping. We are also not going to allow returning of references from Move. In both cases, it greatly complicates the runtime behavior and rules. We have a great language already for those cases, Move. To address problems solved by loops or by complicated reference usage, we will be looking to at improvements to the module publishing flow. Specifically, we might add the ability to run code from a package without publishing it; a "one-off" package of sorts.

Here is a sketch of the programmable transaction kind, which will be contained in a TransactionData struct (so that's where all the gas and sender information will be). Keep in mind:

No end user or developer should be expected to program with this directly!

(unless you really want to of course)

In the short term, the SDK and other tools will generate these transactions for simple things like Pay* or Move calls. In the long term, we will build tools and abstractions that use this system as a target. We might start with exposing this directly in a JSON-like transaction builder perhaps, but we will strive to build a tool or very small DSL that compiles to this format instead.

struct ProgrammableTransactionKind {
  inputs: Vec<CallArg>, // these are the input objects or primitive values
  commands: Vec<Command>,
}

// we will pick reasonable integer types here depending on the limits for number
enum Argument {
  // the gas coin. The gas coin can only be used by-ref, except for with `TransferObjects`, which
  // can use it by-value.
  GasCoin,
  // one of the input objects or primitive values (from inputs above)
  Input(u16),
  // the result of another command.
  Result(u16),
  NestedResult(u16),
}

// each variant might want its own type
enum Command {
  // the MoveCall here can be any entry function with the existing rules
	// or *any public function*. The public functions can have return values,
  // but cannot return any references
  MoveCall {
    package: ObjectRef,
    module: Identifier,
    function: Identifier,
    type_arguments: Vec<TypeTag>,
    arguments: Vec<Argument>,
  },
  // (Vec<forall T:key+store. T>, address)
	// sends n-objects to the specified address
  // objects must have store (public transfer) and the previous owner
  // must be an address
  TransferObjects(Vec<Argument>, Argument),
  // coin operations
  // (&mut Coin<T>, u64) -> Coin<T>
  // splits of some amount into a new coin
  Split(Argument, Argument),
	// (&mut Coin<T>, Vec<Coin<T>>)
	// merges n-coins into the first coin
  Merge(Argument, Vec<Argument>),
}

struct CommandResultValue {
  // A Move type
  type_: Type,
  // derivable from the type_, but explicitly called out here as they will
  // be used in the semantics
  abilities: AbilitySet,
  // BCS bytes
  value: Vec<u8>,
}

// The running result of all commands
type CommandResults = Vec<Option<CommandResultValue>>;

Semantics

Each Command takes Arguments, and produces a series of outputs, CommandResult, which are added to the running series of results, CommandResults.

The inputs: Vec<CallArg> are objects or pure values that can be used in commands (by-value or by-reference). The gas coin will be a special sort of input that with restricted usage. The results The outputs of commands are also accessible as inputs to future commands, with Argument::Result or Argument::NestedResult.

The rules for an individual CommandResultValue depends on the abilities of the value. When used by-value, values with copy are copied; otherwise, the value is “consumed” or “transferred” and the entry in the CommandResult associated with that value is set to None. At the end of the transaction, any value without drop must have been transferred in this way. In other words, at the end of executing all commands, all CommandResultValues remaining in the CommandResults have the drop ability. For unused inputs, the same rules apply as apply for MoveCall , i.e. the contents might change but the owner does not.

With the whole picture in mind, we can look a bit more in depth at each Argument variant

• GasCoin like the input objects, does not need to be used. At the beginning of executing the series of commands, we will withdraw the maximum amount of gas (as specified from the transaction parameters) from the gas coin. Any unused gas will be returned to the coin at the end of execution.
  • To simplify the unused gas logic, the GasCoin cannot be used by-value with MoveCall or Merge. However, it can be used by-value with TransferObjects, and doing so is critical for implementing TransferSui or Pay*
• Input can be either an object, a vector of objects, or a series of BCS bytes. Since these rules are largely unchanged from the current Move Call transaction, we will not go more in depth.
• Result and NestedResult are used for accessing the values of previous transactions. Each value will have a Move type associated with it, which will be strongly typed. This means that the result cannot be used or transmuted to some other value, even if BCS compatible.

For execution, the commands are processed sequentially, where results are accessed by the index system described above. Any errors stop execution and prevent all state changes (though gas will be charged). Similar to the handling of linear/affine values in Move or Rust, values (inputs or results) can be used either by-reference or by-value. This usage is automatic and depends on the signature of the operation. Mutable reference &mut inputs must be uniquely used in that command. Immutable reference inputs & can be used more than once, but cannot be used owned. Using by-value invalidates the memory location. Meaning if it is an input it cannot be used again, and if it is a CommandResultValue we will set it’s location to None. A bit more in detail for each command:

• MoveCall will use its inputs by-value or by-reference according to the Move type of the parameter.
• TransferObjects use its objects and address by-value.
• Split uses its coin by mutable reference &mut and a u64 amount by-value.
• Merge takes its first argument by reference &mut, but the rest are by-value.
• Any &mut usage updates its cell after execution.
• By-value usage invalidates the cell.
• Reference usage & leaves the cell untouched.

At the end of execution of the commands:

• Any remaining input (or the GasCoin) has it’s ownership unchanged.
• Unused gas is returned to the GasCoin regardless of (possibly new) owner.
  • Recall that the GasCoin cannot be destroyed
• For any CommandResultValue it must have drop or it is an error.
  • Astute readers might notice that we copy any value with copy, so what about values with copy and not drop? We can claim that the last usage of any value with copy also invalidates the memory location (transfers the ownership). Such a rule is not observably different from also ignoring values with just copy at the end of execution.

Examples

Pay* Transactions

// PaySui
gas: vec![/* gas coins */, ...],
inputs: vec![addr0],
commands: vec![
  TransferObjects(vec![GasCoin], Input(0)),
]

// PayAllSui
gas: vec![/* gas coins */, ...],
inputs: vec![amount0, addr0, amount1, addr1, amount2, addr2, ...],
commands: vec![
  Split(GasCoin, Input(0)),
  TransferObjects(vec![Result(0)], Input(1)),
  Split(GasCoin, Input(2)),
  TransferObjects(vec![Result(2)], Input(3)),
  Split(GasCoin, Input(4)),
  TransferObjects(vec![Result(4)], Input(5)),
  ...
]

// PayAllSui
gas: vec![/* gas coins */, ...],
inputs: vec![coin0, amount0, addr0, coin1, amount1, addr1, coin2, amount2, addr2, ...],
commands: vec![
  Split(Input(0), Input(1)),
  TransferObjects(vec![Result(0)], Input(2)),
  Split(Input(3), Input(4)),
  TransferObjects(vec![Result(2)], Input(5)),
  Split(Input(6), Input(7)),
  TransferObjects(vec![Result(4)], Input(8)),
  ...
]

Coin Limit

Lets say we want to call a::m::foo(coin: &mut Coin<_>, ctx: &mut TxContext) with a coin C but limit the amount possibly withdrawn to 100

gas: vec![...],
inputs: vec![C, 100]
commands: vec![
  Split(Input(0), Input(1)),
  // pardon the psuedo syntax
  MoveCall("a::m::foo", Result(0))
  Merge(Input(0), Result(0))
]

Shared Object Interplay

We could imagine having two shared dex objects A and B. We want to exchange some SUI in A, take that coin and exchange it in B, and take it back to A for some SUI again.

Assuming a function exchange<C1, C2>(&mut Dex, coin: Coin<C1>): C2

gas: vec![...],
inputs: vec![A, B, 100],
commands: vec![
  // take 100 SUI
  Split(GasCoin, /* 100 */Input(2)),
  // exchange it for X
  MoveCall("exchange<SUI, X>", vec![/* A */Input(0), Result(0)]),
  // exchange the X for Y
  MoveCall("exchange<X, Y>", vec![/* B */Input(1), Result(1)]),
  // exchange the Y for SUI
  MoveCall("exchange<Y, SUI>", vec![/* A */Input(0), Result(2)),
  // merge it
  Merge(GasCoin, vec![Result(3)])
]

Deprecation of SingleTransactionKind and TransactionKind::Batch

Programmable Transactions can cover all variants of SingleTransactionKind except for the system related ones, such as ChangeEpoch and Genesis. Those system related transactions might be fine in their own SystemTransactionKind enum, or can remain in SingleTransactionKind if we do not deprecate the enum.

While all (non-system) single transaction types can be covered by programmable transactions, it might be cumbersome to do so. Especially depending on the the final layout of the ProgrammableTransaction struct itself. All transactions non Pay* transactions become just a single Command. The Pay* transactions are a sequence of Merge, Split, and TransferObjects (depending on the specific Pay* variant).

Similarly, this system can replace Batch as currently Batch is just a special case where no results are used in future commands.

[PaulFidika]

This sounds great! So is essentially a replacement for the script transactions that existed in Diem? If I wanted to write these transactions by hand, they would be written in Rust? They look simple enough.

The most obvious downside is that only values, not references, can be shared across move-calls. Having dynamically composable Move calls that can share references would be amazing if you could engineer it. In particular, I've invented the 'extend' pattern which Capsules uses extensively, it works like this:

module addr_a::mod_a {
   struct TypeA {
      id: UID
   }

   public fun extend(a: &mut TypeA): &mut UID {
      &mut a.id
   }
}

module addr_b::mod_b {
   struct Key has store, copy, drop {}
   public fun add_field(uid: &mut UID) {
      dynamic_field::add(uid, Key { }, true);
   }
}

Essentially, module-a exposes its struct TypeA with the function extend, which returns a mutable reference to its UID. Module-B accepts any &mut UID, and then adds its own field to it, using its own Key, so only Module-B can read / edit that field.

If we wanted to compose these two functions together, a programmable batch transaction would be the obvious solution; we'd simply call into A, get the &mut UID, then all into B, and viola, we're done.

This can obviously also be hard-coded into Move, with its own dedicated A -> B function. However, we must create a separate function for each possible combination, i.e., if there are 100 module-A's, and 100 module-B's, then we must hardcode 100 x 100 = 10,000 separate functions (lol). If we could simply return &mut UID with programmable transactions however, none of that would be necessary.

How feasible do you think it would be to allow Programmable Transactions to accept references in addition to values? This sort of behavior was, I believe, possible with Diem script transactions.

[tnowacki]

For clarity: Programable transactions functions can take reference arguments, but not return them. We could maybe extend it in the future to allow such things, but maybe expanding on this point would help

So is essentially a replacement for the script transactions that existed in Diem?

This isn't quite the case. I would view programmable transactions as something new/different. They are a way of composing both builtin/native transaction types and Move calls. And the intention is to be very lightweight for tooling.

We will add the command Publish(/* modules */ Vec<Vec<u8>>) to the Command enum, and will likely also add arguments to it, so the init functions can take arguments, something like Publish(Vec<(Vec<u8>, Vec<Argument>)>).

We have discussed adding ScriptPublish(Vec<(Vec<u8>, Vec<Argument>)>) which would basically call the init functions of the modules, but without publishing them. However, such "script" modules would not be able to define new types.

[PaulFidika]

For clarity: Programable transactions functions can take reference arguments, but not return them. We could maybe extend it in the future to allow such things, but maybe expanding on this point would help

Yeah I'd really like to be able to pass references between function calls within the same Programmable Transaction; how much harder would that be to implement? Is the issue that we can't hold references in the Results vector?

We have discussed adding ScriptPublish(Vec<(Vec<u8>, Vec<Argument>)>) which would basically call the init functions of the modules, but without publishing them. However, such "script" modules would not be able to define new types.

With this, maybe we could dynamically create a init function that calls into A, gets a mutable reference, then passes it into B. I imagine we'd have some script that generates these functions on the fly on the client-side. So with this sort of behavior we could achieve what we're looking for with the extend pattern I described above, but it's a little more clunky and involved than if Programmable Transactions were allowed to return references.

[tnowacki]

Is the issue that we can't hold references in the Results vector?

It would be a relatively major overhaul of parts of the VM APIs. The core of the problem is updating mutable references.

So with this sort of behavior we could achieve what we're looking for with the extend pattern I described above, but it's a little more clunky and involved than if Programmable Transactions were allowed to return references.

I like to think about this as a spectrum, one end we have simplicity for tool generation and on the other we have program expressiveness. So we have programmable transactions on the one end and ScriptPublish on the other. We can then sort of push each end into the middle over time: Programmable transactions get more expressive and these "scripts" get easier to generate. Though I think the middle of that spectrum will be awkward for a long time

[PaulFidika]

It would be a relatively major overhaul of parts of the VM APIs. The core of the problem is updating mutable references.

Do you think it'll be worth the effort? I think it will; I can help some if you want--sorry I'm basically always just bugging you to build stuff haha.

I like to think about this as a spectrum, one end we have simplicity for tool generation and on the other we have program expressiveness. So we have programmable transactions on the one end and ScriptPublish on the other. We can then sort of push each end into the middle over time: Programmable transactions get more expressive and these "scripts" get easier to generate. Though I think the middle of that spectrum will be awkward for a long time

If programmable transactions can use (im)mutable references, would there be a point to ScriptPublish? What would be something you can do with ScriptPublish that you cannot do with programmable transactions? I can't think of any, because ScriptPublish won't be able to define new structs to begin with.

[FrankC01]

My 0.02:

Expressiveness trumps simplicity in this case besides, tools can work the usability end on client side.

As far as expressiveness you are then faced with how the declarations align to concurrency so in a pseudo sense I want to express that Task1 and Task2 should operate concurrently but Task3 must wait on the results from both to execute:

Pseudo

Task1 (T1):

• PayAllSui
• GetObject for combined coin if call succes

Task2 (T2):

• Move Call
• GetObject X if move call success else fail (rollback)

Task3 (waits on T1 and T2) - won't get here if any of either fail:

• Get field value from GetObject step in T2
• Get fat coin reference from in T1
• MoveCall using information from T1 & T2

Scope values:

• Declarations outside of task or 'global' and can be marked immutable (read only) or mutable (input/output)
• For mutable globals ability to define scalars, collections (vectors, maps/dictionaries) and associative functions related to collection types
• Declarations inside of task and 'mimic' the global kinds but are not visible across tasks

[tnowacki]

It would be a relatively major overhaul of parts of the VM APIs. The core of the problem is updating mutable references.

Do you think it'll be worth the effort? I think it will; I can help some if you want--sorry I'm basically always just bugging you to build stuff haha.

We can add the feature eventually if we think it is worthwhile, so it's not off the table, but is cut from the design at least for the first version

I like to think about this as a spectrum, one end we have simplicity for tool generation and on the other we have program expressiveness. So we have programmable transactions on the one end and ScriptPublish on the other. We can then sort of push each end into the middle over time: Programmable transactions get more expressive and these "scripts" get easier to generate. Though I think the middle of that spectrum will be awkward for a long time

If programmable transactions can use (im)mutable references, would there be a point to ScriptPublish? What would be something you can do with ScriptPublish that you cannot do with programmable transactions? I can't think of any, because ScriptPublish won't be able to define new structs to begin with.

Control flow and branching come to mind first.

But I could also see more generally execution being more efficient with Move the larger and larger your "program" becomes.

[PaulFidika]

Control flow and branching come to mind first.

But I could also see more generally execution being more efficient with Move the larger and larger your "program" becomes.

This is a good point; I imagine you could build script transactions that mint an NFT, and if the NFT does not have the properties you want, call abort and revert the transaction. (As an NFT ecosystem, we'd have to think of to avoid users doing this.)

What are we calling this ScriptPublish functionality? I imagine we could create some macro that creates move code from a template, and then runs it as a ScriptPublish function; how is this different from Script transactions in Diem? Will this functionality be available at the same time as Programmable transactions?

I'd really like to be able to manage mutable references within Programmable transactions; but if it has to wait I understand. I can make do until then, and ScriptPublish can work as well (although it's more complex).

[tnowacki]

What are we calling this ScriptPublish functionality? I imagine we could create some macro that creates move code from a template, and then runs it as a ScriptPublish function; how is this different from Script transactions in Diem? Will this functionality be available at the same time as Programmable transactions?

Nothing finalized yet, but we might just call them "scripts"? Or anonymous modules? Not sure what is best. But in terms of templating or code or anything, it will just be a module with an init function.

And no timeline on this feature yet, sorry.

I'd really like to be able to manage mutable references within Programmable transactions; but if it has to wait I understand. I can make do until then, and ScriptPublish can work as well (although it's more complex).

For references, remember you can have mutable reference inputs, just no outputs. If that helps

[ericEnjoy]

It seems programmable transaction would be helpful for my issue. https://github.com/MystenLabs/sui/issues/8526 I want to implement a batch txn to support NFT buying, but I got some barries that list in issue above. If one txn result could be used in following txn with programmable txn, it will be feasible for batch buying now. Am I understanding this correctly?

[tnowacki]

It seems programmable transaction would be helpful for my issue. #8526 I want to implement a batch txn to support NFT buying, but I got some barries that list in issue above. If one txn result could be used in following txn with programmable txn, it will be feasible for batch buying now. Am I understanding this correctly?

You listed three issues in your issue regarding batch. And some of those will be solved, but some won't be.

batch transactions are atomic, not just a combination of single transaction

This will remain true for programmable transactions

the same object can only appear once in a batch transaction

This will be fixed by programmable transactions! That's one of the main features :)

subtransactions cannot use objects generated or transferred from previous subtransactions

You will only be able to use objects generated if they are passed out as a return value to the programmable transaction runtime. There is no way to access the objects created or anything. However! Keep in mind you can call non-entry public functions, which might help in getting to those return values.

[tnowacki]

All code for this has now landed! So I will mark as closed.

[samlaf]

Since this has landed, how can we give it a test spin? Can't find examples anywhere, and none of the rust/ts sdks, nor the CLI seem to have it implemented in v0.28.

Found these issues which seem to still be open for the cli and rust sdk, but can't find any for the ts sdk.