cli: Generate accounts with funded lamports and SPL tokens

[armaniferrante]

We should use https://github.com/project-serum/anchor/issues/1260 to setup solana-test-validator for testing by automatically generating a bunch of accounts funded with lamports and SPL tokens--including all the most popular mints like USDC.

[paul-schaaf]

some initial ideas:

instead of defining test settings in the main Anchor.toml, we have the following folder structure

.
src
tests
     test-one
         test-one.toml
         test-one-genesis.yaml/toml
         test-one.ts/js
     test-two
         ....

the test-one.toml file is where all the test.validator.<x> settings go. the test-one.js/ts file is where the test lives the test-one-genesis.yaml/toml is where the genesis accounts are defined

these would be defined exactly like they are in the rust code so if you have an account like this

#[account]
pub struct Data {
    pub val: u64,
    pub str: String,
    pub numbers: [u8; 10]
}

u define an account of that type in the genesis config like

accounts:
    - owner: Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS
      data:
              val: 1337
              str: "a cool string"
              numbers: [1,2,3,4,5,6,7,8,9,10]

the account in this config would then get processed and turned into a file that can be uploaded via the --account option of solana-test-validator

might make sense to allow filepaths to genesis files in the test-one.toml so they can be reused e.g. you could define a set of token accounts that you want to use in every test. alternatively, there could be shared-genesis folder instead of filepaths which is more straightforward but less flexible

[paul-schaaf]

some open questions:

• how to deal with non-anchor programs and non-anchor/non-borsh serialization formats
• ways to convert the genesis config into raw data. Maybe:
  • parse the data field in the yaml and create an instance from the program rust struct from it that is then simply AnchorSerialized
  • parse the data field in the yaml and directly serialize it into the data field by field (probably simpler than trying to create rust structs, especially when these come from different crates)

[paul-schaaf]

Some more ideas/open questions:

• provide a account-type enum option which can be anchor or non-anchor. anchor accounts are serialized with borsh. what about non-anchor accounts?
• provide a prefix_data option so that the anchor discriminator can be set. defaults to none for non-anchor accounts and to the INITIALIZED discriminator for anchor accounts (this would mean we'd also need a 'account type name' option because thats how the discriminator is calculated

[tomlinton]

What is the motivation behind changing the configuration structure? At first glance it looks too complex to me, but I think there is some need to provide specific validator setups for different test suites (https://github.com/project-serum/anchor/issues/339 is related). Is that the reasoning here?

In general few things should need to change in the test validator settings between test suites. The loaded accounts, the genesis programs, and perhaps clone options, so I like keeping everything in the root Anchor.toml and making the CLI transparently load accounts from tests/accounts/<address>.json files. Using the subdirectory structure with test-one, test-two that you have outlined could be used to let the CLI know it should tear down and reset the validator in between tests.

[asktree]

This would be extremely helpful for me. Our program interacts with user mSol, but I'm a noob so it would probably take a few hours (or help) for me to figure out how to use existing clone options in Anchor.toml to figure out how to do it. Something like

[[test.validator.spl]]
recipient = "9s3RbNan8nLetHoGFVgCJ4fABF3f5ha9dxcGqSdFrD9e"
mint = "mSoLzYCxHdYgdzU16g5QSh3i5K3z3KZK7ytfqcJm7So"

would be really convenient

[paul-schaaf]

@tomlinton

yes. independent test suites are one motivation.

but also (and it would probably make sense to do this in a separate PR) more configuration options for local environments.

Im not a fan of having all the test settings in the main Anchor.toml. Id like to at least have a Test.toml in the root of the tests directory. It just seems cleaner to have that info where it's used. But Im not sure In general few things should need to change in the test validator settings between test suites. is true. I have two ideas: 1. let test suites overwrite the Test.toml file 2. let test suites import shared test config toml. I like the 2nd more because Im not sure how to do the overwriting elegantly.

and while I agree that we should support json files as well, being able to configure accounts via yaml/toml is a big improvement to the DX. It's just a hassle to create accounts on some cluster and then download them, especially if creating these accounts requires multiple instructions of a program to get to the desired state

[paul-schaaf]

This would be extremely helpful for me. Our program interacts with user mSol, but I'm a noob so it would probably take a few hours (or help) for me to figure out how to use existing clone options in Anchor.toml to figure out how to do it. Something like

[[test.validator.spl]]
recipient = "9s3RbNan8nLetHoGFVgCJ4fABF3f5ha9dxcGqSdFrD9e"
mint = "mSoLzYCxHdYgdzU16g5QSh3i5K3z3KZK7ytfqcJm7So"

would be really convenient

@asktree using this particular config you mentioned, what would you like to happen?

[asktree]

@paul-schaaf

the idea is that this would use the genesis block to airdrop to 9s3RbNan8nLetHoGFVgCJ4fABF3f5ha9dxcGqSdFrD9e an spl token with mint addr mSoLzYCxHdYgdzU16g5QSh3i5K3z3KZK7ytfqcJm7So. does that make sense?

I suppose missing from my example is an amount to airdrop.

my understanding is that this is already possible in userland, you just have to know how to make the ATAs and you can use [[test.validator.account]]?

[asktree]

@paul-schaaf

I think the ability to have test.genesis.toml for each test makes a lot of sense for obvious reasons. I'd still like to be able to use Anchor.toml or similar to set genesis accounts, outside of tests, because we use anchor localnet for stuff (mostly because we don't use anchor test, but its also neat to be able to develop UI against localnet)

[paul-schaaf]

@asktree

the idea is that this would use the genesis block to airdrop to 9s3RbNan8nLetHoGFVgCJ4fABF3f5ha9dxcGqSdFrD9e an spl token with mint addr mSoLzYCxHdYgdzU16g5QSh3i5K3z3KZK7ytfqcJm7So. does that make sense?

yes that makes sense! just wanted to clarify because like you said the amount was missing.

my understanding is that this is already possible in userland, you just have to know how to make the ATAs and you can use [[test.validator.account]]?

yes, either you make the accounts and then use solana account to download them or you clone them from a different cluster into localnet.

mostly because we don't use anchor test

why not?

[asktree]

why not?

I dont remember because it's been a while since we made this choice. I think early on we realized we'd need the metaplex programs on our test validator but couldn't figure out how to do this within anchor test. In the end we never actually did this integration until [[test.validator.clone]] worked, so anchor ended up helping instead of hindering. In general we didn't feel confident in our ability to customize anchor. Sorry this is not more helpful.

[tomlinton]

But Im not sure In general few things should need to change in the test validator settings between test suites. is true

The list of test validator settings is here. I don't really see anything there that needs to change per test outside of account, clone, and maybe the slots_per_epoch and warp_slot option (and even then, you can probably get away with just loading the data you need under different addresses). The CLI should control all the port, path, etc options to allow for test/ledger isolation and parallelism.

and while I agree that we should support json files as well

This issue is about providing prefunded accounts in the format that solana account <address> --output json generates, and being able to load them via the solana-test-validator --account <address> <file> flag (which was done in #1260). I propose we split this discussion up into separate issues for config restructure, rust/toml/yaml <-> json accounts, etc.

[paul-schaaf]

we split this discussion up

agreed

lets get this done first https://github.com/project-serum/anchor/issues/1658

[SC4RECOIN]

this would be cool to have