[Move] Third-Party Package upgrades

[sblackshear]

Principles

Package Upgrades offer builders a way to evolve packages, bringing value to users by adding features or fixing bugs. However, if not carefully designed, those same tools can be used to exploit user trust. Package Upgrades on Sui are designed according to the following principles to limit the potential for exploits without limiting the features available to builders:

• P1. Upgrades should respect object ownership, and not offer a way for other parties (including the package publisher) to change owned objects without explicit authorization from the owner.
• P2. Users should be able to provide informed consent on the ways a package can change after they start using it (how it can be updated, who can authorize updates to it, etc). I.e. this information should be available before value is locked in a package.
• P3. Upgrades should present a bounded risk to the user: The set of possible risks cannot grow over time so that a user locks value in a package, and is then exposed to greater risk at a later point in time.
• P4. When a design decision could either favor the package publisher or its user, opt to favor the user, as the publisher generally has more agency.

Technical Constraints

There are also technical details specific to Sui and Move that impact the design:

• T1. Packages are Immutable objects, meaning that the contents of a package at a particular ID cannot change after it is published. This restriction exists to support move calls in the single-owner fast path.
• T2. Packages can only access fields on the types they define (this includes reading, writing, packing and unpacking those fields). This property is used to maintain encapsulation and is enforced by the Move VM.

Summary of Protocol Changes

A package upgrade will introduce a new object on-chain (T1) which can access objects that were created by older versions of that package. Although it is an object at a new ID, its version field will track its package version.

• To make this work, the package's address on-chain (changes for the upgraded package relative to the original) is decoupled from its self-address (stays the same) which is part of the fully-qualified type name of all types accessible by the package (T2).
• A new field -- published-at -- will be introduced to package manifests, to designate a package's on-chain address, while the addresses in the [addresses] section of the manifest will continue to represent package addresses for type resolution and access purposes.

An upgraded package must be compatible with past versions of itself (See "What can I Upgrade?" below), to allow packages that were originally published calling an older version of that package to be able to run against newer versions.

Upgrades are purely additive, they do not modify any existing on-chain state:

• Existing on-chain objects from a package are unchanged by upgrades (P1).
• Users of old versions of a package will need to explicitly update their call-sites (both entry function calls in off-chain code, and public function calls on-chain) to make use of the upgraded package (P2).
• The old version of a package will also continue to exist, and users are free to continue using it (P4).

Upgrades are be governed by policies which are defined as Move code that works with newly introduced types (see "Summary of Framework Changes", below):

• Packages will issue an UpgradeCap when published, which issues UpgradeTickets authorizing specific upgrades.
• Upgrades will require an UpgradeTicket and will produce an UpgradeReceipt which is used to update the metadata in the originating UpgradeCap. This allows the permissions for upgrades to be managed by other Move packages (e.g. a K-of-N policy, or allow upgrades from a list of allowed addresses, etc).
• The ticket and receipt must be used in the same transaction that created them, so this API depends on #7790, to allow a single transaction to call a move function to get a ticket, run an upgrade, and then call another move function to consume the receipt.
• The explorer will indicate where a package's UpgradeCap is (whether it exists, who owns it, if it's being managed by another object) so that it is always clear what future upgrades are possible on the package (P2).
• UpgradeCap also specifies which parts of a package can be upgraded (compatible changes, just additive changes, just dependencies). The set of admissible changes can only get smaller with time (P3).

Summary of Framework Changes

Upgrades will introduce a new package to the framework package.move, containing UpgradeCap, UpgradeTicket, UpgradeReceipt, and their associated API:

module sui::package {
    use sui::object::{Self, ID, UID};

    /// Capability controlling the ability to upgrade a package.
    struct UpgradeCap has key, store {
        id: UID,
        /// (Mutable) ID of the package that can be upgraded.
        package: ID,
        /// (Mutable) The number of upgrades that have been applied 
        /// successively to the original package.  Initially 0.
        version: u64,
        /// What kind of upgrades are allowed.
        policy: u8,
    }

    /// Permission to perform a particular upgrade (for a fixed version of 
    /// the package, bytecode to upgrade with and transitive dependencies to 
    /// depend against).
    ///
    /// An `UpgradeCap` can only issue one ticket at a time, to prevent races 
    /// between concurrent updates or a change in its upgrade policy after 
    /// issuing a ticket, so the ticket is a "Hot Potato" to preserve forward 
    /// progress.
    struct UpgradeTicket {
        /// (Immutable) ID of the `UpgradeCap` this originated from.
        cap: ID,
        /// (Immutable) ID of the package that can be upgraded.
        package: ID,
        /// (Immutable) The policy regarding what kind of upgrade this ticket 
        /// permits.
        policy: u8,
        /// (Immutable) Digest of the bytecode and transitive dependencies
        /// that will be used in the upgrade.
        digest: vector<u8>,
    }

    /// Issued as a result of a successful upgrade, containing the 
    /// information to be used to update the `UpgradeCap`.  This is a "Hot 
    /// Potato" to ensure that it is used to update its `UpgradeCap` before 
    /// the end of the transaction that performed the upgrade.
    struct UpgradeReceipt {
        /// (Immutable) ID of the `UpgradeCap` this originated from.
        cap: ID,
        /// (Immutable) ID of the package after it was upgraded.
        package: ID,
    }

    /// Update any part of the package (function implementations, add new
    /// functions or types, change dependencies)
    const COMPATIBLE: u8 = 0;

    /// Add new functions or types, or change dependencies, existing 
    /// functions can't change.
    const ADDITIVE: u8 = 1;

    /// Only be able to change dependencies.
    const DEP_ONLY: u8 = 2;

    /// Tried to set a less restrictive policy than currently in place.
    const ETooPermissive: u64 = /* ... */;

    /// This `UpgradeCap` has already authorized a pending upgrade.
    const EAlreadyAuthorized: u64 = /* ... */;

    /// This `UpgradeCap` has not authorized an upgrade.
    const ENotAuthorized: u64 = /* ... */;

    /// Trying to commit an upgrade to the wrong `UpgradeCap`.
    const EWrongUpgradeCap: u64 = /* ... */;

    /// The most recent version of the package, increments by one for each 
    /// successfully applied upgrade.
    public fun version(cap: &UpgradeCap): u64 {
        cap.version
    }

    /// The most permissive kind of upgrade currently supported by this 
    /// `cap`.
    public fun upgrade_policy(cap: &UpgradeCap): u8 {
        cap.policy
    }

    /// Restrict upgrades through this upgrade `cap` to just add code, or 
    /// change dependencies.
    public entry fun only_additive_upgrades(cap: &mut UpgradeCap) {
        restrict(cap, ADDITIVE)
    }

    /// Restrict upgrades through this upgrade `cap` to just change
    /// dependencies.
    public entry fun only_dep_upgrades(cap: &mut UpgradeCap) {
        restrict(cap, DEP_ONLY)
    }

    /// Discard the `UpgradeCap` to make a package immutable.
    public entry fun make_immutable(cap: UpgradeCap) {
        let UpgradeCap { id, package: _, version: _ } = cap;
        object::delete(id);
    }

    /// Issue a ticket authorizing an upgrade to a particular new bytecode 
    /// (identified by its digest).  A ticket will only be issued if one has 
    /// not already been issued, and if the `policy` requested is at least as 
    /// restrictive as the policy set out by the `cap`.
    ///
    /// The `digest` supplied and the `policy` will both be checked by
    /// validators when running the upgrade.  I.e. the bytecode supplied in 
    /// the upgrade must have a matching digest, and the changes relative to 
    /// the parent package must be compatible with the policy in the ticket 
    /// for the upgrade to succeed.
    public fun authorize_upgrade(
        cap: &mut UpgradeCap,
        policy: u8,
        digest: vector<u8>
    ): UpgradeTicket {
        let id_zero = object::id_from_address(@0x0);
        assert!(cap.package != id_zero, EAlreadyAuthorized);
        assert!(policy >= cap.policy, ETooPermissive);

        let package = cap.package;
        cap.package = id_zero;

        UpgradeTicket {
            cap: object::id(&cap),
            package,
            policy: cap.policy,
            digest,
        }
    }

    /// Consume an `UpgradeReceipt` to update its `UpgradeCap`, finalizing 
    /// the upgrade.
    public fun commit_upgrade(
        cap: &mut UpgradeCap, 
        receipt: UpgradeReceipt,
    ) {
        let UpgradeReceipt { cap: cap_id, package } = receipt;

        assert!(object::id(&cap) == cap_id, EWrongUpgradeCap);
        assert!(object::id_to_address(&cap.package) != @0x0, ENotAuthorized);

        cap.package = package;
        cap.version = cap.version + 1;
    }

    fun restrict(cap: &mut UpgradeCap, policy: u8) {
        assert!(cap.policy <= policy, ETooPermissive);
        cap.policy = policy;
    }
}

What can I Upgrade?

Upgraded packages must maintain compatibility with all their previous versions, so a package published to run with version V of a dependency can run against some later version, V + k. This property is guaranteed by doing a link and layout compatibility check during publishing, enforcing the following:

Compatible Changes

• Adding new types, functions and modules.
• Changing the implementations of existing functions.
• Changing the signatures of non-public functions.
• Removing ability constraints on type parameters.
• Adding abilities to struct definitions.

Incompatible Changes

• Changing the signatures of public functions.
• Changing the layout of existing types (adding, removing or modifying field names or types).
• Adding new ability constraints on type parameters.
• Removing abilities from structs.

How will I...?

Examples of how common operations work with the introduction of package upgrades.

Publish and distribute a package

Publishing and distribution works similarly to how it did before, but with the following changes:

• The manifests for on-chain dependencies need to include a published-at address -- this is the address at which the package is found, which could now be different from the self-address of the dependency.
• Publishing will produce an UpgradeCap object which gives permission to the bearer to make future updates to the package. By default this will be returned to the sender, but in more complex use cases, it can be restricted, wrapped, or discarded as part of the publish transaction (using #7790) to set the upgrade policy up-front.
• If the package is going to be used as a dependency of other packages, the published-at field in its own manifest needs to be updated to point to the address it was just published to.
• This is in addition to updating the [addresses] section replacing the 0x0 entries for package self-address(es) with the newly published address.

Upgrade a package

In the simplest case, you own an UpgradeCap for a package, and will be able to call upgrade on a move package with it:

$ sui client upgrade --gas-budget 0x... --cap 0x...

This behaves similarly to a publish, except that it makes sure that the --cap argument points to a valid UpgradeCap that you have permission to modify, that the bytecode being published is compatible with the version of the package that is being guarded by the cap, and registers the newly published package as an upgrade of this old package.

In cases where the UpgradeCap is guarded by a more complex upgrade policy (See "Choose when upgrades are allowed", below), the upgrade transaction will need to be built using #7790, using the following steps:

1. Build the bytecode for the upgraded package.
2. Calculate the digest of this bytecode (this is a sha256 hash of the module bytecode, calculated by hashing each module separately, sorting the resulting hashes, and then rehashing them to form a stable digest).
3. Add a command to the programmable transaction to invoke the upgrade policy, which holds the UpgradeCap, requesting permission to perform the upgrade. The precise interface exposed by the policy is implementation-specific, but you will at least need to supply the digest calculated in the previous step. If the request is successful, this command will return an UpgradeTicket as a result.
4. Add the Upgrade command next, taking the ticket as the first input, and the bytes for the upgrade as the following input. If the upgrade is successful, it will produce an UpgradeReceipt as a result.
5. The final command in the programmable transaction is a move call, to finalize the upgrade by updating the UpgradeCap with the UpgradeReceipt by calling another function on the upgrade policy which accepts the receipt (again which function is specific to the policy).
6. Execute the programmable transaction on the network.

Assess risk from package upgrades

It's important that a package's upgrade policy is easily auditable, so that potential stakeholders can verify that the package they are agreeing to will not change in a way that they did not expect, after they lock value in it.

Because a package's upgrade policy is controlled by its UpgradeCap, auditability will be provided by tracking the relationship between different versions of the same package and their UpgradeCap, and display this information in the Explorer. When viewing a package it will be possible to identify whether its UpgradeCap is still available, and if so, where it is:

• If the UpgradeCap has been deleted, the package is not upgradeable.
• If it is not deleted, and is owned by an individual address, trust in the package is based on trust in the entity that controls that address, because they could modify the package.
• If it is not deleted, and is wrapped in another object that is part of an upgrade policy, trust in the package depends on trust in the code that defines the upgrade policy (and in turn whether that is upgradeable and if so what its upgrade policy is, and so on).

Package use limited to purely objects that you own confers minimal risk because you will always be able to continue using the old package. But if your use of a package involves shared objects, or handing ownership of your objects to other parties, then care should be taken when it comes to the relevant packages' upgrade policies:

As a general rule of thumb, packages used in production by a large number of third parties should have very restrictive upgrade policies. I.e. they are either not upgradable at all, or the right to upgrade is controlled by more than one individual address (e.g. a K-of-N policy). As an extreme example, actively used upgrade policies should not be upgradeable.

In comparison it is much less risky for packages that act as entry points to individual apps, or that are in the early stages of development to have permissive upgrade policies to allow for bugfixes.

Choose what can be upgraded

The owner of an UpgradeCap can restrict future upgrades to one of the following policies:

• COMPATIBLE: any change as long as it maintains link and layout compatibility.
• ADDITIVE: only new types, functions and modules, as well as changes to dependencies.
• DEP_ONLY: only changes to dependent packages.

Packages start off with upgrade caps that permit all compatible upgrades, and can be restricted using the following functions in UpgradeCap:

module sui::package {
    /// Restrict upgrades through this upgrade `cap` to just add code, or
    /// change dependencies.
    public entry fun only_additive_upgrades(cap: &mut UpgradeCap) {
        restrict(cap, ADDITIVE)
    }

    /// Restrict upgrades through this upgrade `cap` to just change
    /// dependencies.
    public entry fun only_dep_upgrades(cap: &mut UpgradeCap) {
        restrict(cap, DEP_ONLY)
    }
}

Once an UpgradeCap has been restricted in this way, it can only become more restrictive with time, not less.

Implement a custom upgrade policy

Below is an example of a custom upgrade policy that requires K-of-N votes to authorize a particular upgrade, built on types in the sui::package module:

module example::k_of_n {
    use sui::object::{Self, ID, UID};
    use sui::package::{Self, UpgradeCap, UpgradeTicket, UpgradeReceipt};
    use sui::tx_context::{Self, TxContext};
    use sui::vec_set::{Self, VecSet};

    /// An upgrade policy where upgrades through `cap` are controlled by 
    /// voting, with `required_votes` needed from the addresses in 
    /// `possible_votes`.
    struct KofNUpgradeCap has key, store {
      id: UID,
      cap: UpgradeCap,
      required_votes: u64,
      possible_votes: VecSet<address>,
    }

    /// An in-progress vote on whether `signer` should be allowed to issue an 
    /// upgrade with digest `digest`.
    struct Vote has key, store {
        id: UID,
        /// The ID of the `KofNUpgradeCap` that this vote was initiated from.
        cap: ID,
        /// The address requesting permission to perform the upgrade.
        signer: address,
        /// The digest of the bytecode that the package will be upgraded to.
        digest: vector<u8>,
        /// The voters who have already agreed to this upgrade.
        voters: VecSet<address>,
    }

    /// This address cannot participate in votes for this upgrade cap.
    const EInvalidVoter: u64 = 0;

    /// Not enough votes accrued to issue a ticket.
    const ENotEnoughVotes: u64 = 1;

    /// An upgrade ticket has already been issued from this ticket.
    const EAlreadyIssued: u64 = 2;

    /// The address requesting the ticket does not match the `signer` in the
    /// vote.
    const ESignerMismatch: u64 = 3;

    /// Protect `cap` in a `KofNUpgradeCap`.
    public fun new(
        cap: UpgradeCap,
        required_votes: u64,
        possible_votes: VecSet<address>,
        ctx: &mut TxContext,
    ): KofNUpgradeCap {
        KofNUpgradeCap {
            id: object::new(ctx),
            cap,
            required_votes,
            possible_votes,
        }
    }

    /// Start a new vote for `signer` to upgrade package in `cap` so its
    /// content's digest matches `digest`.
    public fun propose(
        cap: &KofNUpgradeCap,
        signer: address,
        digest: vector<u8>,
        ctx: &mut TxContext,
    ): Vote {
        Vote {
            id: object::new(ctx),
            cap: object::id(&cap),
            signer,
            digest,
            voters: vec_set::empty(),
        }
    }

    /// Discard a vote in progress.
    public fun burn_vote(vote: Vote) {
        let Vote { id, voters: _ } = vote;
        object::delete(id);
    }

    /// Vote in favour of an upgrade, aborts if the sender is not one of the
    /// possible voters for this cap.
    public fun vote(cap: &KofNUpgradeCap, vote: &mut Vote, ctx: &TxContext) {
        let voter = tx_context::sender(ctx);
        assert!(vec_set::contains(&cap.possible_voters, voter), EInvalidVoter);
        vec_set::insert(&mut vote.voters, voter);
    }

    /// Issue an `UpgradeTicket` for the upgrade being voted on.  Aborts if 
    /// the vote has not accrued enough votes yet, or has already issued a
    /// ticket, or the sender of this request is not the proposed signer of
    /// the upgrade transaction.
    public fun authorize_upgrade(
        cap: &mut KofNUpgradeCap,
        vote: &mut Vote,
        ctx: &TxContext,
    ): UpgradeTicket {
        assert!(
            vec_set::size(&vote.voters) >= cap.required_votes, 
            ENotEnoughVotes,
        );

        let signer = tx_context::sender(ctx);
        assert!(vote.signer != @0x0, EAlreadyIssued);
        assert!(vote.signer == signer, ESignerMismatch);

        vote.signer = @0x0;
        let policy = package::upgrade_policy(&cap.cap);
        package::authorize_upgrade(
            &mut cap.cap,
            policy,
            vote.digest,
        )
    }

    /// Finalize the upgrade that ran to produce the given `receipt`.
    public fun commit_upgrade(
        cap: &mut KofNUpgradeCap, 
        receipt: UpgradeReceipt,
    ) {
        package::commit_upgrade(&mut cap.cap, receipt)
    }
}

Deprecate a type, function, or package

Extreme cases may call for the prevention of future access to functions or types in a buggy package. This will not be possible for function calls involving purely owned objects, but will be if shared objects are involved.

Deprecation will use the "Struct Version Constraints" extension (see below), to prevent access to the shared object by older versions of the package, without losing data: Doing so forces any access to that shared object to go through the newer version of the package, so even though the older version exists on-chain, it cannot be used.

From the user perspective, this poses an additional risk, to watch out for when assessing exposure to risk due to package upgrades in the presence of shared objects.

Gotchas

Diamond Problem

The diamond dependency problem manifests with a dependency graph as follows:

+-- A --+
|       |
v       v
B       C
|       |
+-> D <-+

Where all packages are published by different parties who are unaware of each other. Although the graph appears consistent, the version of D that B depends on is different from the version that C depends on (W.l.o.g. C depends on the newer version):

+-- A --+
|       |
v       v
B       C
|       |
v       v
D   <   D'

This was not an issue for B or C separately, but becomes an issue when A enters the picture, because it is not clear which version of D it should depend on. This will be addressed by supporting dependency overrides: A will be able to specify its own version of D which subsumes the version specified by B and C. In general, a package can override the dependencies of any package that it dominates in the dependency graph.

When A is published, there will be a further check that the version of D it picks is at least as new as the newest version picked by its dependent packages. This is to ensure that A does not pick a version that is link incompatible with one of its dependencies.

Module Initializers

Module Initializers are commonly used to perform operations that developers rely on happening exactly once per package (such as creating one-time witnesses). As such, they will not be re-run when a package is upgraded.

Invariant Violation

One consequence of packages being immutable is that they cannot be deleted, even when they are superseded by a later version and as mentioned above, older versions of a package will still be callable, potentially accessing objects that are being used by newer version of that package.

This set-up can introduce bugs when the new version of the package is maintaining invariants that the old version is completely unaware of:

module 0xA0::counter {
    use sui::object::{Self, UID};
    use sui::tx_context::TxContext;
    use sui::transfer;

    struct Counter has key {
        id: UID,
        value: u64,
    }

    fun init(ctx: &mut TxContext) {
        transfer::share_object(Counter {
          id: object::new(ctx),
          value: 0,
        });
    }

    public entry fun increment(c: &mut Counter) {
      c.value = c.value + 1;
    }
}

module 0xA1::counter {
    use sui::object::{Self, UID}
    use sui::tx_context::TxContext;
    use sui::event;

    struct Counter has key {
        id: UID,
        value: u64,
    }

    struct Progress has copy, drop {
        reached: u64
    }

    public entry fun increment(c: &mut Counter) {
        c.value = c.value + 1;

        if (c.value % 100 == 0) {
            event::emit(Progress { reached: c.value });
        }
    }
}

In the example above, the new version of A::counter::increment emits a Progress event every 100 increments, whereas the old version does not. If there are a mix of callers for this function and some still use the old version of the package, this invariant will be broken (maintaining dynamic fields that need to remain in sync with a struct's original fields is another source of bugs).

While this issue could affect any object, it's particularly problematic for shared objects who do not have an owner to keep track of a common thread and make sure the object is not sent to packages at different versions. The "Struct Version Constraints" extension will help alleviate this risk in some cases, but for the most part package library developers need to be aware of the potential interplay between versions of their package.

Package Rug Pulls

This is a pattern where a user hands over control of their assets (e.g. by putting them in a shared object or through some other means) based on their understanding of what a package allows, only to have that change through an upgrade:

module 0xB0::safe {
    use sui::coin::Coin;
    use sui::object::{Self, ID, UID};
    use sui::sui::SUI;
    use sui::table::{Self, Table};
    use sui::tx_context::{Self, TxContext};


    struct Safe has key {
        id: UID,
        accounts: Table<ID, Coin<SUI>>,
    }

    fun init(ctx: &mut TxContext) {
        transfer::share_object(Safe {
            id: object::new(ctx),
            accounts: table::new<ID, Coin<SUI>>(),
        });
    }

    public entry fun deposit(
        safe: &mut Safe, 
        coin: Coin<SUI>, 
        ctx: &TxContext,
    ) {
        table::add(&mut safe.accounts, tx_context::sender(ctx), coin);
    }

    public entry fun withdraw(safe: &mut Safe, ctx: &TxContext) {
        transfer::transfer(
            table::remove(&mut safe.accounts, tx_context::sender(ctx)),
            tx_context::sender(ctx),
        );
    }
}

module OxB1::safe {
    /* ... as before ... */

    public entry fun steal(safe: &mut Safe, account: address, ctx: &TxContext) {
        transfer::transfer(
            table::remove(&mut safe.accounts, account),
            tx_context::sender(ctx),
        );
    }
}

This is undesirable, but stems from flexibility that is required to allow developers to fix bugs or add functionality -- core value propositions of package upgrades. This is countered by providing a clear audit trail so users can know who controls the packages they are relying on, and decide whether to trust them or not (informed consent, see "Assess risk from package upgrades", above).

Extensions

Automated address management

Managing on-chain package addresses is already cumbersome -- developers need to remember to update [addresses] entries after they publish, after wipes, when switching between devnet, testnet and mainnet etc. Package upgrades introduce another address to manage -- the ID of the latest version of the package.

The proposal introduces the published-at manifest field as a bare minimum form of support, but in the long-term, developers will not have to worry about updating addresses themselves. Address information will be moved to the lock file, and operations like publish and upgrade will update them automatically.

Struct Version Constraints

Given a struct S that was introduced at version V of a package P, if it appears in a later version U with a #[min_version(U)] annotation, it implies that transactions that use P at version U or above will write back instances of S with the package ID of P at version U (not P). From that point packages at versions below U will not be able to read that instance of S, because they will treat it as a different type.

This functionality allows us to perform one-way migrations on specific types where previously a type would have been forward and backward compatible, which enabled patterns to "Deprecate a type, function, or package" (see above).

Upgrade Hooks

Like the module initializer, it may be helpful to have a function that is run as part of a successful upgrade, e.g. to migrate objects (e.g. shared objects managed by a package), however this only becomes useful when this kind of call (i.e. module initializers) supports accepting parameters (which is not currently true).

Appendix: End-to-end Example

Consider packages A, depending on B, depending on C, where B depends on C at version 0, and A depends on C at version 1. Assume for convenience that the on-chain version of package P at version X is found at 0xPX, its upgrade cap is found at 0xCA4P and the source is found at:

P = { git = "https://github.com/MystenLabs/example", subdir = "P", rev = "4a540X" }

e.g. Version 1 of C is found at 0xC1 on-chain, sub-directory C of revision 4a5401 of https://github.com/MystenLabs/example off-chain, and its UpgradeCap is found at 0xCA4C.

Their manifests would be set-up as follows:

# MystenLabs/example/A/Move.toml @4a5400
[package]
name = "A"
version = "0.0.0"
published-at = "0xA0"

[addresses]
A = "0xA0"
B = "0xB0"
C = "0xC0"

[dependencies]
B = { git = "https://github.com/MystenLabs/example", subdir = "B", rev = "4a5400" }
C = { git = "https://github.com/MystenLabs/example", subdir = "C", rev = "4a5401" }

# MystenLabs/example/B/Move.toml @4a5400
[package]
name = "B"
version = "0.0.0"
published-at = "0xB0"

[addresses]
B = "0xB0"
C = "0xC0"

[dependencies]
C = { git = "https://github.com/MystenLabs/example", subdir = "C", rev = "4a5400" }

# MystenLabs/example/C/Move.toml @4a5400
[package]
name = "C"
version = "0.0.0"
published-at = "0xC0"

[addresses]
C = "0xC0"

# MystenLabs/example/C/Move.toml @4a5401
[package]
name = "C"
version = "0.0.0"
published-at = "0xC1"

[addresses]
C = "0xC0"

The packages would be published/upgraded with the following commands, and assuming that at the time of the transaction, its self-address is set to 0x0:

C (4a5400)$ sui client publish --gas-budget 10000
# Txn: linkage = {}, package = "..."

C (4a5401)$ sui client upgrade --gas-budget 10000 --cap 0xCA4C
# Txn: linkage = {}, package = "..."

B (4a5400)$ sui client publish --gas-budget 10000
# Txn: linkage = {0xC0}, package = "..."

A (4a5400)$ sui client publish --gas-budget 10000
# Txn: linkage = {0xB0, 0xC1}, package = "..."

Which is represented on-chain as:

0xA0.linkage = {
    0xB0: (0xB0, 0),
    0xC0: (0xC1, 1)
}

module 0xA0::A {
    use 0xB0::B::b;
    use 0xC0::C::{c, update, S, T};

    public fun foo(s: S) {
        bar(update(s))
    }

    public fun bar(t: T) {
        /* ... */
    }
}

0xB0.linkage = {
    0xC0: (0xC0, 0)
}

module 0xB0::B {
    use 0xC0::C;
}

0xC1.linkage = {}

module 0xC1::C {
    struct S { a: u64 }
    struct T { a: u64, b: bool }

    public fun update(s: S): T {
        let S { a } = s;
        T { a, b: false }
    }

    public fun c(): u64 { 43 }
}

0xC0.linkage = {}

module 0xC0::C {
    struct S { a: u64 }
    public fun c(): u64 { 42 }
}

• There is a clear distinction between the on-chain and off-chain representation of the package, for example:
  • version in the manifest has no bearing on the on-chain package, and
  • the only thing linking the source of a package at version N+1 with the package at version N is the call to sui client upgrade.
• When a package is published, it references other packages at their original IDs, not their upgraded IDs, relying on the linkage table to translate that back to the actual Package IDs during resolution.
• The linkage information is provided in the transaction as a set of Package IDs (which is automatically gathered from the published-at fields of dependency packages), but is stored on-chain as a mapping. This translation is done by the Upgrade transaction, after verifying that the upgrade is valid, to speed up loading in future.

[FrankC01]

Two things:

1. Immutable (flagging it so it can not be updated) is valuable in cases of one-time capability or time based subscription oriented contracts
2. Can these insightful info subjects be made into discussions or GitHub pages so as not to be hidden in the issues bucket?

[amnn]

Hey @FrankC01, thanks for raising. Immutable packages are indeed something that we're looking to support and let me talk to the team about what's a good place to surface these kinds of topics (this topic, and Time are the two examples I'm aware of).

One benefit of issues is that other projects can link to them and we get to see that, but yes, they do get kind of buried among all the other day-to-day issues. Maybe we can introduce a tag for larger features, so they can be filtered out better?

[FrankC01]

Tagging, discussions (which can generate issues from the discussions) are two good ones.

[JackyWYX]

Can the upgradeable permission (in description on publisher of the module) can be changed to another account? Is PackageUpgradeCap an object that can be transferred?

[amnn]

@JackyWYX, yes it will. Sorry it's taking a little while for me to publish the proposal for how upgrades will work -- our current testnet wave is taking all of our focus, but it will be out soon!

[ericEnjoy]

This feature will be very convenient for contract development, and I see it's in the Wave 3 milestone, so this feature will come out before mainnet, am I right?

[amnn]

Hi @ericEnjoy, it will be there for mainnet, but not before Wave 3. Our tags are a little stale on this issue, unfortunately. Thanks for spotting!

[amnn]

Heads up: This issue has now been updated with the latest proposal for how upgrades will work.

[d-moos]

Hey, happy to see that there is progress on this topic. Are package upgrades allowed to modify the friend declarations as well?

[amnn]

Hi @d-moos, yes, you can change your friend declarations, as well as the signatures and implementations of friend visibility functions, because friend modules all have to be in the same package (i.e. they behave like package-private visibility), so they do not get exposed in the package's public interface.

[WGB5445]

Thanks for providing such interesting features Is the #[min_version] function available now and how should I use it ?

[amnn]

Hi @WGB5445, version constraints on structs are not part of the initial upgrades release, they are still to come!

[WGB5445]

Thank you for your answer, Will there be before the mainnet goes live?

[amnn]

While it or a similar feature will be added, we haven't put it on the roadmap yet!

[nguyenhoaibao]

According to the Package Upgrade documentation, it said we can add abilities for existing structs. I tested with a struct without the store ability, that means it cannot be transferred:

struct Test has key { id: UID };

then I published that package to the testnet.

Later on I added the store ability for that struct:

struct Test has key, store { id: UID };

I upgraded the package successfully, however when I create a new instance of that new Test struct, I still cannot transfer that new instance. What can I do to make newly objects of that Test struct be able to be transferred?

Another thing to notice is if I create an entry transfer function for that Test struct in its module, like:

public entry fun transfer(object: Test, recipient: address) {
    transfer::transfer(object, recipient)
}

then I can use that entry function to transfer Test objects.

[amnn]

@nguyenhoaibao, thanks for reporting this, this does look like an issue we should address. I've started a discussion internally around how best to do that.

The technical context here is that when we upgrade a package, we don't change the identities of existing types in that package (i.e. if your type Test was introduced first in a package at address 0x123 then it will always be referred to as 0x123::Test on-chain), this means that when a transaction block tries to Transfer it, it's going to look at the abilities of 0x123::Test, even if 0x456::Test exists with a later version of the package (and type).

The reason why your example in Move works is that in Move, we are allowed to "re-link" packages and their dependencies, as long as they remain compatible. So if you were to call transfer::transfer in the upgraded package, or even from a package that depends on your upgraded package (i.e. outside the module, or even the package that defines Test), that would work, because it understands the expanded ability set of Test.

[Alivers]

After I upgraded my contract, I added a new dynamic object field to an existing parent object. But I could not get the new child object data of this dynamic object field. getObject(include explorer and cli) api returns with error: Failure serializing object in the requested format: "Could not find module". here is my original object id: 0x78673aae6463c14dab82feb636d7cf2b89a72a255be8375e22b47c71749d903b , on testnet https://explorer.sui.io/object/0x78673aae6463c14dab82feb636d7cf2b89a72a255be8375e22b47c71749d903b?network=testnet

The LendingMarket Object is created using old package entry. I added a new entry func to add dynamic field for LendingMarket in new package. The child object added to LendingMarket is a new struct defined in new module. Is there anything I can do to fix this error?

[amnn]

@Alivers, thanks for the report, we'll look into it!

[nguyenhoaibao]

@amnn I see the latest v0.33.0 release does not allow to add abilities to existing struct while upgrading the package anymore. So is there any other way to make existing struct can be transferred, if it cannot at the first publishing, like the case I described above https://github.com/MystenLabs/sui/issues/2045#issuecomment-1519596272?

By the way, after upgrading to v0.33.0, I cannot publish the contract anymore:

Multiple source verification errors found:

- Local dependency did not match its on-chain version at 0000000000000000000000000000000000000000000000000000000000000002::Sui::kiosk
- Local dependency did not match its on-chain version at 0000000000000000000000000000000000000000000000000000000000000002::Sui::transfer_policy

Do you know what it is and how to fix it?

[vivekascoder]

@nguyenhoaibao I'm having the same issue.

[amnn]

@nguyenhoaibao, @vivekascoder, the issue you are facing with source verification is happening because the network will undergo a framework upgrade. The branch that you are depending on contains the system packages that will be used after the next protocol upgrade, so if there are any changes, source verification will flag them. You have a couple of choices for how to get around this:

• Use --skip-dependency-verification to ignore the check temporarily.
• Depend on the specific revision that was running on testnet at the time (this can be difficult to determine though).
• Wait until the protocol upgrade happens (usually at the end of the epoch).

@nguyenhoaibao, regarding adding store (or other abilities) to types during upgrades. When investigating your report, we realised that there is an issue with how this feature is currently exposed, so we had to disable it (thanks again for the report!) We do plan on re-enabling this feature, but when we do, it will probably be along with "struct version constraints" (i.e. to change a type, you would also need to apply a version constraint to prevent old versions of the package from being able interact with it). Sorry we don't have a solution for you to do this today, but we're working on it.

@Alivers, we've figured out what is causing the issue you're facing, and we're working on fixing it!

[nguyenhoaibao]

@amnn have you found any solution yet? Sui is going to release mainnet today - within next 8 hours to be precise - so we also have to finalize our package to prepare for the deployment asap 😅

[amnn]

Hi @nguyenhoaibao, struct version constraints, and therefore the ability to add abilities to existing types was not on the roadmap before mainnet, but is high on the list of features we will tackle post-launch.

[10xhunter]

@sblackshear @amnn could you please add store ability to UpgradeTicket struct?

My use case: dao users can propose to issue an upgrade ticket to a third party to upgrade an external package.

[amnn]

Hi @10xhunter, no we cannot add store to UpgradeTicket, because doing so would allow someone to stall an upgrade half way through by issuing a ticket but then never using it.

You should still be able to achieve what you want by using a custom upgrade policy -- this is why UpgradeCap has store.

To allow someone to propose an upgrade, the package being upgraded should use an upgrade policy that allows such proposals. Take a look at the K-of-N upgrade policy example in this issue, or alternatively this PR, which is creating a productionised version of the same:

#14879

The details of the custom upgrade policy for your DAO example would be different (we can discuss this more if you share some more details), but these examples show how to add custom logic around upgrades.

[10xhunter]

@amnn I've also considered the issue you mentioned, in that case, we can propose to issue another ticket with different digest to another party to skip the stalled upgrade.

[10xhunter]

My use case is that, for example, group A published a package AP, and gave the AP UpgradeCap to shared object DAO_A, which is managed by dao3.ai contracts, when group A wants to upgrade package AP, they will propose in DAO_A, and issue an UpgradeTicket to whoever the receiver is to upgrade package AP.

[amnn]

It sounds like what you want to do can be achieved by using a custom upgrade policy, in this case, one that authorizes a specific address, or the holder of some capability, the ability to upgrade a package whose UpgradeCap is owned by a DAO.

Have you had a look at our documentation on custom upgrade policies, and if so, does it seem like you could create an upgrade policy that fits your purpose, if not, what are the blockers to doing that?