`Invoked an instruction with data that is too large` error

[acheroncrypto]

Problem

Some programs raise Invoked an instruction with data that is too large error with 1.16.0 release.

I wasn't able to reproduce this error in a smaller program but I can share 3 different programs where this happens.

• auction-house
• token-metadata
• openbook-dex

Logs from this instruction with 1.16.0(platform-tools v1.37):

'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk invoke [1]',
'Program log: Instruction: CreateAuctionHouse',
'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk consumed 11789 of 200000 compute units',
'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk failed: Invoked an instruction with data that is too large (12884933012 > 10240)'

Pre 1.16.0(platform-tools v1.29):

'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk invoke [1]',
'Program log: Instruction: CreateAuctionHouse',
'Program 11111111111111111111111111111111 invoke [2]',
'Program 11111111111111111111111111111111 success',
'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk consumed 29292 of 200000 compute units',
'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk success'

All examples work pre 1.16.0 and don't work with 1.16.0 and seems like others who have used the 1.16.0 version have run into this issue as well. See https://github.com/solana-labs/solana/issues/31337#issue-1682750647, https://github.com/solana-labs/solana/issues/30271#issuecomment-1518740240.

[buffalojoec]

Do you have some details on which specific instructions are throwing it? And is it always the same instructions on each program that errors?

[KirillLykov]

For auction-house example, did you reproduce the issue with the client https://github.com/armaniferrante/auction-house/blob/armani/pda/tests/auction-house.ts ? I wonder if I can get it just using this repo as starting point

[acheroncrypto]

For auction-house example, did you reproduce the issue with the client https://github.com/armaniferrante/auction-house/blob/armani/pda/tests/auction-house.ts ? I wonder if I can get it just using this repo as starting point

Yes. Switching back to 1.14.18 makes it work without any modifications to program/client code https://github.com/coral-xyz/anchor/blob/383e44078811fdf26dc73a4ad834b812c7b5636e/.github/workflows/reusable-tests.yaml#L419.

I've run into this issue in Anchor CI and the easiest way to reproduce it would be to clone Anchor and run the mentioned tests(they are submodules). Test workflow is useful to check the commands to run.

[KirillLykov]

Will try to check this out this week

[effeaucarre]

@acheroncrypto here is a simple program to reproduce it:

use anchor_lang::prelude::*;
use anchor_spl::token::{self, Mint, TokenAccount};

declare_id!("xxxx");

const METADATA_ACCOUNT_PREFIX: &str = "xxxx";

#[program]
pub mod real_program {
    use super::*;

    pub fn create_metadata_account(ctx: Context<CreateMetadataAccount>, ) -> Result<()> {
      
        Ok(())
    }
}

#[derive(Accounts)]
pub struct CreateMetadataAccount<'info> {
	
	/// CHECK
	#[account(mut, signer)]
  pub owner: AccountInfo<'info>,
  
  pub mint: Account<'info, Mint>,
	
  #[account(
    init,
    payer = owner,
		seeds = [METADATA_ACCOUNT_PREFIX.as_bytes(), mint.key().as_ref()],
    token::mint = mint,
    token::authority = owner,
    bump,
    )]
  pub real_metadata_account: Box<Account<'info, TokenAccount>>,
  
  /// CHECK
  #[account(address = anchor_spl::token::ID)]
  pub token_program: AccountInfo<'info>,
  
	/// CHECK
	pub system_program: AccountInfo<'info>,
  pub rent: Sysvar<'info, Rent>,
}

[KirillLykov]

@acheroncrypto could you provide details about reproducing this run wtih auction-house? I've tried anchor test (never used anchor before), it prints:

Warning: cargo-build-bpf is deprecated. Please, use cargo-build-sbf
cargo-build-bpf child: /home/klykov/.local/share/solana/install/active_release/bin/cargo-build-sbf --arch bpf

But it in the validator log program is not mentioned.

[acheroncrypto]

@KirillLykov you can see this run. It shows exactly what commands to run to reproduce it.

From the same CI logs:

 1) auction-house
       Creates an NFT mint:
     Error: failed to send transaction: Transaction simulation failed: Error processing Instruction 0: Program failed to complete
      at Connection.sendEncodedTransaction (/home/runner/work/anchor/anchor/ts/node_modules/@solana/web3.js/src/connection.ts:5646:13)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at Connection.sendRawTransaction (/home/runner/work/anchor/anchor/ts/node_modules/@solana/web3.js/src/connection.ts:5605:20)
      at sendAndConfirmRawTransaction (/home/runner/work/anchor/anchor/ts/packages/anchor/src/provider.ts:370:21)
      at AnchorProvider.sendAndConfirm (/home/runner/work/anchor/anchor/ts/packages/anchor/src/provider.ts:160:14)

This error message is not very helpful but if you get the transaction logs, you will see Invoked an instruction with data that is too large error.

Note that the first test is a setup step that doesn't even use the auction-house program.It uses this program which doesn't use Anchor. If you build that program with an older Solana version, the first test(Creates an NFT mint) passes but this time, Creates an auction house test fails. Both of the tests fail with Invoked an instruction with data that is too large error.

[KirillLykov]

@effeaucarre could you provide also a client to execute this program?

[dhruvja]

I was getting the above error so when i downgraded to 1.14.18, i am getting the following error. ( anchor version: 0.27.0 )

Warning: cargo-build-bpf is deprecated. Please, use cargo-build-sbf
cargo-build-bpf child: /Users/dhruvjain/.local/share/solana/install/active_release/bin/cargo-build-sbf --arch bpf
error: package `toml_edit v0.19.10` cannot be built because it requires rustc 1.64.0 or newer, while the currently active rustc version is 1.62.0-dev

I cannot upgrade to anchor version 0.28.0 since it is breaking for switchboard-v2 crate.

[acheroncrypto]

@KirillLykov here is a script to reproduce the issue:

#!/usr/bin/env bash

set -e

# Set solana version
solana-install init 1.16.0

# Create a temp directory
mkdir data-too-large-error
pushd data-too-large-error

# Clone and build token-metadata program
git clone https://github.com/metaplex-foundation/metaplex.git
pushd metaplex
git checkout cbb590d5
pushd rust/token-metadata/program
cargo build-bpf
popd
popd

# Create an Anchor project(only for testing)
anchor init reproduce
cd reproduce

# Add the program to genesis
echo '[[test.genesis]]
address = "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s"
program = "../metaplex/rust/target/deploy/metaplex_token_metadata.so"' >> Anchor.toml

# Setup tests
yarn add @metaplex/js@=4.4.1
echo 'import * as anchor from "@coral-xyz/anchor";
import { Program } from "@coral-xyz/anchor";
import { programs } from "@metaplex/js";
import { Token, TOKEN_PROGRAM_ID } from "@solana/spl-token";
import { PublicKey } from "@solana/web3.js";
import { Reproduce } from "../target/types/reproduce";

const TOKEN_METADATA_PROGRAM_ID = new PublicKey(
  "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s"
);

const { CreateMetadata, MetadataDataData } = programs.metadata;

describe("reproduce", () => {
  anchor.setProvider(anchor.AnchorProvider.env());

  const program = anchor.workspace.Reproduce as Program<Reproduce>;
  const payer = program.provider.publicKey;

  it("reproduces", async () => {
    const nftMintClient = await Token.createMint(
      program.provider.connection,
      // @ts-ignore
      program.provider.wallet.payer,
      payer,
      null,
      6,
      TOKEN_PROGRAM_ID
    );

    // Create the metadata
    const [metadata] = await PublicKey.findProgramAddress(
      [
        Buffer.from("metadata"),
        TOKEN_METADATA_PROGRAM_ID.toBuffer(),
        nftMintClient.publicKey.toBuffer(),
      ],
      TOKEN_METADATA_PROGRAM_ID
    );

    const tx = new CreateMetadata(
      { feePayer: payer },
      {
        metadata,
        metadataData: new MetadataDataData({
          name: "test-nft",
          symbol: "TEST",
          uri: "https://nothing.com",
          sellerFeeBasisPoints: 1,
          creators: null,
        }),
        updateAuthority: payer,
        mint: nftMintClient.publicKey,
        mintAuthority: payer,
      }
    );

    try {
      await program.provider.sendAndConfirm(tx);
    } catch (e) {
      console.log(e);
      return;
    }

    throw new Error("Did not reproduce");
  });
});
' > tests/reproduce.ts

# Run tests
anchor test

# Return back to the initial directory
popd

The script uses Anchor only to test from TypeScript and the program that returns this error is not an Anchor program.

Compiling the program with 1.14.18 CLI solves the problem

Seems like there is something wrong with the program binary. Running the tests with 1.16.0 works if the program was compiled with 1.14.18.

solana-install init 1.14.18
pushd data-too-large-error/metaplex/rust/token-metadata/program
cargo build-bpf
popd
pushd data-too-large-error/reproduce
solana-install-init 1.16.0
anchor test
popd

[KirillLykov]

Reproduced the problem with script above, note that works only with anchor 0.28.0

[KirillLykov]

Based on the following runs, looks like compiler toolchain (not sdk) is the source of problem:

# OK
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/sdk/bpf"
# OK
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/sdk/sbf"
# Fails
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/sdk/sbf"
# Fails
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/sdk/bpf"

[mvines]

@dmakarov might be able to assist if this indeed is a toolchain issue

[danmt]

I was facing the same problem here, I fixed it by downgrading Solana Tool Suite from 1.16.1 to 1.14.18 without additional changes.

[SebastianBor]

I get the same problem with GPL, tests work with 1.14.18 but fail with 1.16.1

[dmakarov]

I get the same problem with GPL, tests work with 1.14.18 but fail with 1.16.1

How to run GPL tests? Is running cargo test sufficient to reproduce the problem?

[SebastianBor]

How to run GPL tests? Is running cargo test sufficient to reproduce the problem? Yes, cargo test-sbf for the integrations tests

[KirillLykov]

@acheroncrypto From discussion with @dmakarov, looks like the problem is with the inconsistency of the dependencies version used by metaplex program, solana installation and anchor.

I've created a branch https://github.com/KirillLykov/metaplex-program-library/tree/klykov/sol-issue-31960 based on the commit cbb590d5 provided in the script above. This branch updates solana-sdk to 1.16.2 so it is aligned with solana installation, it updates anchor to 0.28 and updates borsh. This resolves the problem.

[acheroncrypto]

@KirillLykov thanks but I'll note that those versions hasn't changed in ~2 years and they were also tested with every Solana release within the last ~2 years(without having to change versions).

This will hurt composability quite a bit because not only you'd have to upgrade your own dependencies but you'd also potentially need to wait for all of the subdependencies to upgrade as well.

[KirillLykov]

@KirillLykov thanks but I'll note that those versions hasn't changed in ~2 years and they were also tested with every Solana release within the last ~2 years(without having to change versions).

This will hurt composability quite a bit because not only you'd have to upgrade your own dependencies but you'd also potentially need to wait for all of the subdependencies to upgrade as well.

It looks like it is not on our side directly, we suspect that the problem is related to the borsh update (https://github.com/solana-labs/solana/pull/30975#issuecomment-1594253543)

In this particular case, it was enough to update only direct dependencies of MPL. It looks like the master of this repo already uses quite recent version of solana / anchor so updating is hopefully not a problem (if it is only about mpl) https://github.com/metaplex-foundation/metaplex-program-library/issues/1137

[DogLooksGood]

After downgrading to 1.14.20, the version of cargo used in cargo-build-sbf is 1.62-dev. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than 1.64, how to solve this?

[KirillLykov]

After downgrading to 1.14.20, the version of cargo used in cargo-build-sbf is 1.62-dev. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than 1.64, how to solve this?

is this an opensource project? I want to check the cargo tree to see what do you mean. I guess your project depends on some crates that are requiring 1.64 but you solana installation is on 1.62, this should not be a problem I guess? Not sure

[acheroncrypto]

After downgrading to 1.14.20, the version of cargo used in cargo-build-sbf is 1.62-dev. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than 1.64, how to solve this?

is this an opensource project? I want to check the cargo tree to see what do you mean. I guess your project depends on some crates that are requiring 1.64 but you solana installation is on 1.62, this should not be a problem I guess? Not sure

It's one of the most reported problems and the error message is not very helpful(https://github.com/solana-labs/solana/issues/31428).

Here are some examples from the StackExchange:

• https://solana.stackexchange.com/questions/6526/error-package-winnow-v0-4-4-cannot-be-built-because-it-requires-rustc-1-64-0
• https://solana.stackexchange.com/questions/7074/error-when-anchor-build-package-constant-time-eq-v0-3-0-cannot-be-built
• https://solana.stackexchange.com/questions/6722/error-when-anchor-build-package-toml-datetime-v0-6-2-cannot-be-built

It's a miracle if you can compile a program with 1.14 CLI without a lock file(or downgrading all of the crates that bumped their MSRV past 1.62).

[KirillLykov]

After downgrading to 1.14.20, the version of cargo used in cargo-build-sbf is 1.62-dev. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than 1.64, how to solve this?

is this an opensource project? I want to check the cargo tree to see what do you mean. I guess your project depends on some crates that are requiring 1.64 but you solana installation is on 1.62, this should not be a problem I guess? Not sure

It's one of the most reported problems and the error message is not very helpful(#31428).

Here are some examples from the StackExchange:

* https://solana.stackexchange.com/questions/6526/error-package-winnow-v0-4-4-cannot-be-built-because-it-requires-rustc-1-64-0

* https://solana.stackexchange.com/questions/7074/error-when-anchor-build-package-constant-time-eq-v0-3-0-cannot-be-built

* https://solana.stackexchange.com/questions/6722/error-when-anchor-build-package-toml-datetime-v0-6-2-cannot-be-built

It's a miracle if you can compile a program with 1.14 CLI without a lock file(or downgrading all of the crates that bumped their MSRV past 1.62).

I was confused with the original message, I thought it is about solana and borsh update which lead to the current problem. Now it looks to me that this is problem with cargo version https://github.com/solana-labs/solana/issues/31428 and, hence, is unrelated to the Invoked an instruction with data ... problem, please correct me if I'm wrong

[acheroncrypto]

I was confused with the original message, I thought it is about solana and borsh update which lead to the current problem. Now it looks to me that this is problem with cargo version #31428 and, hence, is unrelated to the Invoked an instruction with data ... problem, please correct me if I'm wrong

The problem in #31428 is not directly related to the current issue but it's still related in a sense that when people upgrade to 1.16, their program compiles but they get a runtime error with Invoked an instruction with data... and when they try to downgrade to 1.14(which solves the runtime problem) they get a compile error due to MSRV, hence the comment https://github.com/solana-labs/solana/issues/31960#issuecomment-1644450572.

[DogLooksGood]

@KirillLykov I finally have my problem solved by following #31775. I used to think the solution can either be upgrading to 1.16 or downgrading to 1.14.

However, for upgrading. mpl-token-metadata still uses the old borsh, so it cannot be compiled with solana-program. So it doesn't work in my case.

For downgrading and using stable toolchain. Due to underlying libraries updated their dependencies, now they conflict with old cargo(1.62).

[KirillLykov]

Short summary of the above

Upgrade of solana-cli from 1.14 to 1.16 leads to problems with compiled on-chain program. What we know:

• No problem: if on-chain program has been compiled with toolchain 1.16 and it depends on anchor 0.28, solana-program/sdk >= 1.16
• Problem: program compiled with toolchain 1.16 but it depends on previous versions of anchor 0.28 and solana crates. In this case, we get runtime error Invoked an instruction with data that is too large (12884933012 > 10240)
• Most probably cause: borsh upgrade from 0.9 to 0.10. Introduced in https://github.com/solana-labs/solana/pull/30975#issuecomment-1594253543

My understanding is that it happens because the client program which triggers execution of the on-chain program serializes instruction data using borsh 0.9, while on-chain program uses newer version to deserialize it. (correct me if I'm wrong).

Do we know if borsh serialization was used to write serialized data to on-chain accounts? Like program with borsh 0.9 writes data in one format, while a program with borsh 0.10 reads this data and misinterprets it?

How it can be solved

• Downgrading borsh to 0.9 in solana => projects which upgraded to 0.10 will stop working, so we will not go this way
• Adding borsh 0.9 support to sdk types (together with 0.10) , partially done in https://github.com/solana-labs/solana/pull/32511

Other issues

There is another issue explained in The error message for MSRV mentioned here. My understanding is that it happens when the user updated their installation to the latest but later returned back to 1.14 so that some crates use versions that cannot be compiled with the rustc version used internally by cargo-build-sbf. (Correct me if I'm wrong)

@acheroncrypto could you confirm that this summary is correct?

[acheroncrypto]

@acheroncrypto could you confirm that this summary is correct?

Yes, it's a good summary overall but I'm not fully convinced of the analysis for the following reasons.

• Problem: program compiled with toolchain 1.16 but it depends on previous versions of anchor 0.28 and solana crates. In this case, we get runtime error Invoked an instruction with data that is too large (12884933012 > 10240)

Anchor version is irrelevant here, as I mentioned in my script post, Anchor is only being used for the client side tests and not as a program crate. Here are the dependencies that the program in the reproducable script example uses: https://github.com/metaplex-foundation/metaplex/blob/cbb590d53a3a9718ecf2b526580952fc6d10baaf/rust/token-metadata/program/Cargo.toml#L15-L23

• Most probably cause: borsh upgrade from 0.9 to 0.10. Introduced in Bump borsh to 0.10.3 #30975 (comment)

My understanding is that it happens because the client program which triggers execution of the on-chain program serializes instruction data using borsh 0.9, while on-chain program uses newer version to deserialize it. (correct me if I'm wrong).

Even if that's the case, why does the on-chain program use 0.10? The program in the reproducable script has a lock file so the version of solana-program and borsh crates don't actually change when you change the toolchain/CLI version. Why would changing the build-tools/CLI version have an effect on borsh de/serialization when the crate versions hasn't changed at all?

Do we know if borsh serialization was used to write serialized data to on-chain accounts? Like program with borsh 0.9 writes data in one format, while a program with borsh 0.10 reads this data and misinterprets it?

To my knowledge, borsh serialization is still the same and the breaking change comes from a new required deserialization method in the API. How the data is serialized hasn't changed and it's not like we have a new borsh that interprets the data differently after many years. That would totally break everything so it would most likely never happen(🤞).

Solana build tools is not opinionated on which de/serialization was used in the program, so if there is a program with a lock file, that used to work, should continue to work independent of the build tools/CLI version used. Why? Because the program doesn't even have a dependency to the new borsh version. (https://github.com/metaplex-foundation/metaplex/blob/cbb590d53a3a9718ecf2b526580952fc6d10baaf/rust/Cargo.lock#L392-L435)

[KirillLykov]

@acheroncrypto thanks for the comments, I somehow missed the part of why borsh has been upgraded after cli upgrade. Should be lock file which controls that, need to dig a bit more into that

[joncinque]

I figured this out finally! I was able to reproduce this reliably by upgrading my tools to 1.16, and then building and running a program that uses the 1.14 crates.

The program just does a token transfer in a CPI: https://github.com/solana-labs/solana-program-library/tree/master/examples/rust/transfer-tokens

If you build the program with 1.16 tools, 1.14 crates, and dump the raw byte representation of the transfer instruction, ie:

    let p = &ix as *const _ as *const u8;
    let bytes = unsafe { core::slice::from_raw_parts(p, 30) };
    msg!("{:?}", &bytes);

You'll get:

[2023-08-07T22:57:59.720273041Z DEBUG solana_runtime::message_processor::stable_log] Program log: [6, 221, 246, 225, 215, 101, 161, 147, 217, 203, 225, 70, 206, 235, 121, 172, 28, 180, 133, 237, 95, 91, 55, 145, 58, 140, 245, 133, 126, 255]

And with 1.14 tools, 1.14 crates, you get:

[2023-08-07T22:57:38.652318633Z DEBUG solana_runtime::message_processor::stable_log] Program log: [208, 124, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 88, 125, 0, 0, 3, 0]

What's going on here? This time, we can blame the Rust language! (But it's not their fault, because they explicitly say that they can change the internals)

The Solana tools v1.16 use a Rust compiler v1.68. In Rust v1.66, the byte layout of Vec changed, so we introduced some new types that maintain the old byte layout of important types such as Vec in https://github.com/solana-labs/solana/pull/30192

To preserve ABI compatibility, when you call invoke or invoke_signed, the 1.16 crates have an extra step where they transform from the normal Rust representation to the StableInstruction variant, which maintains the old ABI for Vec: https://github.com/solana-labs/solana/blob/b7c2ad5b67990161d9972d1fa6e81510a023a20d/sdk/program/src/program.rs#L296

But in our case, we're using the 1.16 compiler, which breaks Vec ABI, with the 1.14 crates, which have no correction for Vec, so things break!

Until we can backport those fixes to 1.14, the workaround for this issue is to be sure to always use the 1.16 crates with the 1.16 compiler, otherwise your program will break Vec ABI and fail runtime checks.