foundry `forge test` hangs on `invariant` testing

Component

Forge

Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?

forge 0.2.0 (97f070f 2023-03-17T09:30:30.350509Z)

What command(s) is the bug in?

No response

Operating System

macOS (Intel)

Describe the bug

When running any type of invariant testing, the terminal hangs after compiling the contracts:

This is the function I'm calling:

function invariant_getHello() public {
    assertTrue(true);
}

It hangs with either forge test --match-test $FUNC --fork-url $ARB --fork-block-number $BLOCK -vvv where $FUNC is invariant_getHello or with forge test --match-test invariant --fork-url $ARB --fork-block-number $BLOCK -vvv.

If I change the function name to test_getHello, the test runs and passes without issues, so the problem is in the invariant keyword:

Mar 27 '23 19:03 trojanMcAfee

The bug is when passing the --fork-url $ARB flag. If I remove it, the test runs and passes.

Mar 27 '23 19:03 trojanMcAfee

Selecting the fork directly in the contract with createSelectFork doesn't solve the issue.

Mar 27 '23 21:03 trojanMcAfee

If you run with RUST_LOG=forge=trace,foundry_evm=trace,ethers=trace forge test <rest of command> you'll see this is just because it's making a lot of RPC requests since it's a fork test. If you pin to a block and use a fuzz seed, subsequent invocations will be much fast since responses will be cached. You can read more here: https://book.getfoundry.sh/forge/fork-testing

Mar 28 '23 18:03 mds1

It didn't work @mds1

As you can see on my screenshots, I was passing the flag --fork-block-number $BLOCK with 69254399 as BLOCK.

I also tried removing the flag and setting the block directly in the contract with vm.createSelectFork(vm.rpcUrl('arbitrum'), 69254399); while having my .toml file like:

[profile.default]
src = 'contracts'
out = 'out'
libs = ['node_modules', 'lib']
test = 'test/foundry'
cache_path = 'forge-cache'

[rpc_endpoints]
arbitrum = "${ARBITRUM}"

[fuzz]
seed = 10

As you can see there, I specified a seed also.

Mar 29 '23 09:03 trojanMcAfee

I created #4662 since the problem is unresolved and I can't re-open this issue.

Once the situation is resolved, I can close that other issue.

Mar 29 '23 09:03 trojanMcAfee

Don't think that it's a cache issue, which would explain why your solution didn't work @mds1

I removed cache_path = 'forge-cache' from my .toml file so the cache would default back to ~/.foundry/cache/rpc/arbitrum/69254399 (which it did), but the hangs still remains only for invariant tests.

EDIT:

Foundry is catching properly to ~/.foundry/cache.. (there's data there), but when I run forge cache ls, I still get an output of 0.0B

Mar 29 '23 11:03 trojanMcAfee

One way you can try to verify that tests are not hanging, and are simply slow due to the RPC queries is as follows:

Modify your foundry config as follows:

[profile.default.fuzz]
seed = 1

[profile.default.invariant]
runs = 1
depth = 1

This specifies a seed for the random number generator, and makes the invariant tests only run a single run of depth 1. Now, run forge test --fork-url $ARBITRUM_RPC_URL . I ran forge init, and modified Counter.t.sol as follows:

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

import "forge-std/Test.sol";
import "../src/Counter.sol";

contract CounterTest is Test {
    Counter public counter;

    function setUp() public {
        counter = new Counter();
        counter.setNumber(0);
    }

    function invariant_example() public {
        assertTrue(true);
    }
}

With this configuration, it took 2.5s to run the test. Remember that you can modify the test command to be RUST_LOG=forge=trace,foundry_evm=trace,ethers=trace forge test --fork-url $ARBITRUM_RPC_URL to see what's happening under the hood so you can see if things are hanging vs. if there's just a lot of RPC requests happening.

However, it does seem that RPC caching is not being leveraged, as it does take ~2.5s each invocation, even if I specify a fuzz seed

Mar 29 '23 14:03 mds1

Thanks for the reply :) @mds1

And yeah, setting runs = 1 makes the test run, and like you mentioned, with RUST_LOC=...., it shows the multiple RPC calls being made under the hood, but in the end, the test just sits there (aka it hangs), which is what matters since I'm trying to run an invariant test, and having only one run defeats the purpose.

I'm not sure I fully understood your last sentence, but it seems like you were able to reproduce my issue (?).

In the end, what I'm trying to do is to run a invariant fuzzing campaign on an Arbitrum fork with standard parameters. Is this a bug or do I have to make a change in my config in order to achieve this outcome?

Mar 29 '23 14:03 trojanMcAfee

but in the end, the test just sits there (aka it hangs)

Can you clarify what you mean here? From what I can see it is running and making RPC requests, it's not stuck/hanging

In the end, what I'm trying to do is to run a invariant fuzzing campaign on an Arbitrum fork with standard parameters. Is this a bug or do I have to make a change in my config in order to achieve this outcome?

Are you running your own node or using a service like e.g. Alchecmy/Infura? If the latter, it's very likely you'll get rate limited due to the number of RPC requests, which will make the tests execute very slowly. Unfortunately there's not much we can do here on the forge side. One solution is to allow forge to be initialized with a list of provider URLs and have it rotate providers with each call, but that might be a nontrivial change. If this is something you need I'd suggest opening a separate issue for this

I'm not sure I fully understood your last sentence, but it seems like you were able to reproduce my issue (?).

I just meant that it seems the cache is not being used for invariant tests, because every run took ~2.5s, but I'd have expected subsequent runs to be only a few milliseconds thanks to using the cache. However, I had forgotten to pin a block, and with pinning + seed, RPC caching is working as expected (~2.5s for first run, a few millseconds for subsequent runs)

Mar 29 '23 14:03 mds1

Can you clarify what you mean here? From what I can see it is running and making RPC requests, it's not stuck/hanging

I can see that the RPC requests are being made when I run forge test with RUST_LOG=...., but if I run it without it, so I don't see under hood, which is what matters for me, the completion of the test per se (not the behaviour under the hood), it just sits in this screen forever. It doesn't complete the test. That's what I mean by "it hangs".

Do you manage to actually complete a test with standard invariant-fuzzing parameters on a fork?

Are you running your own node or using a service like e.g. Alchecmy/Infura? If the latter, it's very likely you'll get rate limited due to the number of RPC requests, which will make the tests execute very slowly

I'm using RPC services (both Alchemy and Infura), but with other fuzzers like Echidna, running tests with forks work without issue, so it should work with Foundry also, no? Especially if you're mentioning that the RPC catching is not being leveraged, that I can't successfully run an invariant test on a fork, and have the evidence of the issue. @mds1

Isn't this enough to qualify the issue as a bug?

Mar 29 '23 15:03 trojanMcAfee

I understand what you mean now. @mds1

The "hang" that I'm experiencing is just forge doing the RPC requests. If I have set runs to like 100, it's going to feel like forever.

But still, there's a catching issue that foundry is not doing. That's the bug I'm mentioning should be considered no?

Mar 29 '23 15:03 trojanMcAfee

it just sits in this screen forever. It doesn't complete the test. That's what I mean by "it hangs".

Got it. So it will complete eventually, just slowly, due to RPC queries. Something like https://github.com/foundry-rs/foundry/issues/585 would help make it clear that things are progressing and not actually hanging

Do you manage to actually complete a test with standard invariant-fuzzing parameters on a fork?

If I run using alchemy on arbitrum, I have not waited long enough for it to complete. However if I use my local mainnet erigon node I get through an invariant run with the default settings (256 runs, depth of 15) in ~20 seconds

Especially if you're mentioning that the RPC catching is not being leveraged,

Caching is being leveraged if you use a seed and pin to a block, I just forgot to pin to a block my initial test test

I'm using RPC services (both Alchemy and Infura), but with other fuzzers like Echidna, running tests with forks work without issue, so it should work with Foundry also, no?

Maybe they just make fewer RPC requests somehow? They would have the same issue about being rate-limited, so I'm not sure what they'd do different. Maybe @mattsse has some insight here

Mar 29 '23 15:03 mds1

Caching is being leveraged if you use a seed and pin to a block, I just forgot to pin to a block my initial test test

I have a pinned block and seed also, but the behaviour is the same.

vm.createSelectFork(vm.rpcUrl('arbitrum'), 69254399)

But don't you think that the fact that I can't successfully complete one invariant test on a fork can be considered a bug?

I have runs = 100 (which is not that much in what fuzzing concerns), and I still have the same test that I started 10 minutes ago running without producing any output. @mds1

Mar 29 '23 15:03 trojanMcAfee

this has to do with how the sender fuzzing in combination with forking mode works

because we're using random callers which all need to be fetched via rpc first if the test is started in forking mode.

But I'm not sure if this is really necessary. manually entering forking mode should get around this though, so

vm.createSelectFork(vm.rpcUrl('arbitrum'), 69254399)

should only fetch everything after entering the fork, if the test is not launched in forking mode

Mar 29 '23 19:03 mattsse

What does that mean exactly if I may ask? @mattsse

I tried both ways, with vm.createSelectFork(vm.rpcUrl('arbitrum'), 69254399) in the code directly, and with --fork-url $ARB --fork-block-number $BLOCK as flags on the CLI, but with both the issue is still present (both as separate attempts and not mixed with one another)

Do I have to do something extra in order to get invariant tests to work?

Mar 29 '23 19:03 trojanMcAfee

if launched in forking mode (--fork-url) then the executor that is used to run a test is already in forking mode. So every time the test is called the sender account is fetched via rpc, because it is already in forking mode.

entering manually will use a random, empty account for the sender (not in forking mode yet)

    function invariant_example() public {
       vm.createSelectFork("https://eth-mainnet.alchemyapi.io/v2/<>");
       assert(true);
    }

forge t
[PASS] invariant_example() (runs: 256, calls: 3840, reverts: 108)
Test result: ok. 1 passed; 0 failed; finished in 1.82s

I guess technically it is correct when running in forking mode that the sender account should be fetched via rpc, but I'm not sure if this is useful here or the "right" behaviour

Mar 29 '23 20:03 mattsse

I guess technically it is correct when running in forking mode that the sender account should be fetched via rpc, but I'm not sure if this is useful here.

What data are we actually fetching from the sender up front? vs. just fetching e.g. nonce or something lazily as needed

Mar 29 '23 20:03 mds1

the account consists of:

code
balance
nonce

that's what we need to create the Account object the evm needs.

this is actually loaded during evm execution, evm needs to check balance, nonce

Mar 29 '23 20:03 mattsse

This example doesn't work for me unfortunately @mattsse

I need to use the fork in setUp() since I'm doing something with arbitrum's state there.

I tried creating several forks in the setup, end that function up with a different fork, and then select arbitrum fork in my invariant test, but the test still hangs

Mar 29 '23 20:03 trojanMcAfee

Hmm interesting. It does feel unnecessary, as in practice I'd suspect most bugs surfaced by invariant testing are not due to differences in the sender. But I'm also not sure what the best fix is here.

Current behavior:

/// Strategy to select a sender address:
/// * If `senders` is empty, then it's either a random address (10%) or from the dictionary (90%).
/// * If `senders` is not empty, a random address is chosen from the list of senders.

Perhaps instead, if senders is empty, we change it to 5-10 default random senders to use for invariant testing?

That way we know, code, balance, and nonce will always be zero
Can derive the hardcoded addresses as address(uint160(uint256(keccak256("invariant sender i")))) for i in the range of 1..=10 or something
Can have a flag to switch back to the current behavior of generating senders if we feel current behavior is useful

cc @joshieDo @lucas-manuel curious to hear thoughts

Mar 29 '23 20:03 mds1

Any other possible solution that I could try?

I tried also creating a fork in the setup and one in the invariant test...same result

Mar 29 '23 20:03 trojanMcAfee

yeh if entered in setUp then it is also already in forking mode when the random sender for the test function is fetched.

this does not seem very useful imo especially since we don't enforce balance/nonce/code constraints for the sender.

so I suggest we insert an empty sender account into the database first?

Mar 29 '23 20:03 mattsse

How do I do that? Sorry for the ignorant question.

Mar 29 '23 20:03 trojanMcAfee

so I suggest we insert an empty sender account into the database first?

agreed it's not too useful, I think it'd be better to just change the default sender approach to the one suggested in https://github.com/foundry-rs/foundry/issues/4656#issuecomment-1489249915, I don't see much value in pulling it from the dict / generating a purely random address

Mar 29 '23 20:03 mds1

Don't forget about me please.

I really really want to get invariant testing to work, and I've been struggling for like a week trying to solve it.

Or is it a bug and there's no workaround I can do to get it to work and a patch from you guys is the only solution? @mds1 @mattsse

Mar 29 '23 20:03 trojanMcAfee

this we'd need to add to the forge rust code.

The way the Evm impl works is like this


// This is configured with the transaction (test call) and a block environment (block number etc.)
struct Evm {tx,block_env}

// This interface is used by the evm to fetch all the data like: `Account{balance, nonce, code}`
// There can be multiple implementations, like a `Database` impl that fetches `Account` via rpc (forking mode DB)
trait Database {
  fn load(Address) -> Account;
}

so because we select a random caller Database::load(caller) is called on each test run and the Account is fetched via RPC because the evm is configured with the forking mode Database implementation

Mar 29 '23 20:03 mattsse

Don't forget about me please.

I really really want to get invariant testing to work, and I've been struggling for like a week trying to solve it.

Or is it a bug and there's no workaround I can do to get it to work and a patch from you guys is the only solution? @mds1 @mattsse

sorry about the inconvenience

I guess we just haven't tested this with forking mode properly. Thanks for flagging this.

@mds1 I think both fixes would be pretty simple

Perhaps instead, if senders is empty, we change it to 5-10 default random senders to use for invariant testing?

this makes sense, this way we still have the actual accounts

Mar 29 '23 20:03 mattsse

@cdgmachado0 If you want a workaround in the meantime just use targetSenders() to hardcode a list of 1 or more senders, that way random senders are not generated. You can read more in the docs: https://book.getfoundry.sh/forge/invariant-testing#invariant-test-helper-functions

Mar 29 '23 20:03 mds1

@mds1 I think both fixes would be pretty simple

Sorry just to clarify, this is just a single solution that would behave as follows:

Define 5-10 default random senders to use for invariant testing?
Derive those hardcoded addresses as address(uint160(uint256(keccak256("invariant sender i")))) for i in the range of 1..=10 (or however many senders we generate)
Since no one will have the private key for these addresses, we know, code, balance, and nonce will always be zero
We can consider also adding a flag to switch back to the current behavior of "If senders is empty, then it's either a random address (10%) or from the dictionary (90%)." IMO we can leave this out for now and only add it back in if requested, or if @joshieDo has a reason for the current approach that we're not thinking of. I know @lucas-manuel usually uses targetSenders so I think he'd be onboard with this change and not need a flag to bring back old behavior

Mar 29 '23 20:03 mds1

Just to let you know guys, targetSender didn't work as a workaround. It still hangs.

Or were you able to get a test going? If you did, let me know and I'll check if the problem is me.

Mar 30 '23 18:03 trojanMcAfee