Invariant Testing: Reverts with assertion failed but nothing was asserted during depth calls inside a run

[0xTimepunk]

Component

Forge

Have you ensured that all of these are up to date?

• [X] Foundry
• [X] Foundryup

What version of Foundry are you on?

forge 0.2.0 (fa6b39c 2023-11-04T00:17:54.146286000Z)

What command(s) is the bug in?

forge test

Operating System

macOS (Apple Silicon)

Describe the bug

While doing invariant testing in a multi-fork protocol setup with a handler (with targetContract being the handler), after some depth in a given run the test fails with "Assertion Failed"

However, nothing is asserted as per the -vvvvv logs, but it can be seen that what is going to be asserted would pass the assertion (assertEq(vaultSharesStore.superPositionsSum(), vaultSharesStore.vaultShares())

This is the last portion of the error log

    ├─ [5442] VaultSharesStore::setSuperPositions(1100178655211191658495 [1.1e21])
    │   └─ ← ()
    ├─ [5398] VaultSharesStore::setVaultShares(1100178655211191658495 [1.1e21])
    │   └─ ← ()
    └─ ← ()

Test result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 4.72s

Ran 1 test suites: 0 tests passed, 1 failed, 0 skipped (1 total tests)

Failing tests:
Encountered 1 failing test in test/invariant/VaultShares.invariant.t.sol:VaultSharesInvariantTest
[FAIL. Reason: Assertion failed.]
	[Sequence]
		sender=0x00000000000000000000000000000000000059F5 addr=[test/invariant/handlers/VaultSharesHandler.sol:VaultSharesHandler]0xF62849F9A0B5Bf2913b396098F7c7019b51A820a calldata=singleXChainSingleVaultDeposit(uint256,uint256,uint256,uint256,uint256,uint64,uint64,uint256,uint256), args=[1276, 39548127599650612562675395263511828807659227779522933251885709603164229992493 [3.954e76], 4259, 12050 [1.205e4], 1950, 39, 44, 2925, 8273]
		sender=0x0000000000000000000000000000000000000B3B addr=[test/invariant/handlers/VaultSharesHandler.sol:VaultSharesHandler]0xF62849F9A0B5Bf2913b396098F7c7019b51A820a calldata=singleDirectSingleVaultDeposit(uint256,uint256,uint256,uint256,uint256,uint64,uint256,uint256), args=[31832160851906571763120612042207763562075912756566557833510525053824624 [3.183e70], 1198419075554122319554971631503617215527304075928 [1.198e48], 161173968795298143063021478275479671275860903514403122 [1.611e53], 115792089237316195423570985008687907853269984665640564039457584007913129639932 [1.157e77], 692185473334003511218357122244830392308939670117385315266785921795 [6.921e65], 0, 2029848068874465088568 [2.029e21], 16684452001925767596015656742902671239935658315607813 [1.668e52]]
		sender=0x000000000000000000000000000000008de833aa addr=[test/invariant/handlers/VaultSharesHandler.sol:VaultSharesHandler]0xF62849F9A0B5Bf2913b396098F7c7019b51A820a calldata=singleXChainSingleVaultDeposit(uint256,uint256,uint256,uint256,uint256,uint64,uint64,uint256,uint256), args=[5641276831496337050631457663379 [5.641e30], 2, 115792089237316195423570985008687907853269984665640564039457584007913129639933 [1.157e77], 0, 3, 2154224951080008 [2.154e15], 202498 [2.024e5], 115792089237316195423570985008687907853269984665640564039457584007913129639933 [1.157e77], 115792089237316195423570985008687907853269984665640564039457584007913129639933 [1.157e77]]
		sender=0x0000000000000000000000000000000000000001 addr=[test/invariant/handlers/VaultSharesHandler.sol:VaultSharesHandler]0xF62849F9A0B5Bf2913b396098F7c7019b51A820a calldata=singleXChainSingleVaultDeposit(uint256,uint256,uint256,uint256,uint256,uint64,uint64,uint256,uint256), args=[20409 [2.04e4], 8291, 4004, 6571, 18560 [1.856e4], 288230376151711744 [2.882e17], 55010946 [5.501e7], 10056 [1.005e4], 252]

 invariant_vaultShares() (runs: 1, calls: 4, reverts: 1)

And this is the code of the Invariant test

/// SPDX-License-Identifier: Apache-2.0
pragma solidity ^0.8.21;

import { VaultSharesHandler } from "./handlers/VaultSharesHandler.sol";
import { VaultSharesStore } from "./stores/VaultSharesStore.sol";
import { BaseInvariantTest } from "./Base.invariant.t.sol";

contract VaultSharesInvariantTest is BaseInvariantTest {
    VaultSharesStore internal vaultSharesStore;
    VaultSharesHandler internal vaultSharesHandler;

    function setUp() public override {
        super.setUp();
        (
            address[][] memory coreAddresses,
            address[][] memory underlyingAddresses,
            address[][][] memory vaultAddresses,
            address[][][] memory sAddresses,
            uint256[] memory forksArray
        ) = _grabStateForHandler();

        /// @dev set fork back to id 0 to create a store and a handler (which will be shared by all forks)
        vm.selectFork(FORKS[0]);
        vaultSharesStore = new VaultSharesStore();

        vaultSharesHandler =
        new VaultSharesHandler(chainIds, contractNames, coreAddresses, underlyingAddresses, vaultAddresses,
        sAddresses, forksArray, vaultSharesStore, timestampStore);

        vm.label({ account: address(vaultSharesStore), newLabel: "VaultSharesStore" });
        vm.label({ account: address(vaultSharesHandler), newLabel: "VaultSharesHandler" });

        /// @dev Note: disable some of the selectors to test a bunch of them only
        bytes4[] memory selectors = new bytes4[](2);
        selectors[0] = VaultSharesHandler.singleXChainSingleVaultDeposit.selector;
        selectors[1] = VaultSharesHandler.singleDirectSingleVaultDeposit.selector;
        targetSelector(FuzzSelector({ addr: address(vaultSharesHandler), selectors: selectors }));
        targetContract(address(vaultSharesHandler));

        // Prevent these contracts from being fuzzed as `msg.sender`.
        excludeSender(address(vaultSharesStore));
        excludeSender(address(vaultSharesHandler));
    }

    /*///////////////////////////////////////////////////////////////
                    INVARIANT TESTS
    //////////////////////////////////////////////////////////////*/

    function invariant_vaultShares() public useCurrentTimestamp {
        assertEq(vaultSharesStore.superPositionsSum(), vaultSharesStore.vaultShares());
    }
}

EDIT:

You can now inspect a showcase of this error in our now public repository in https://github.com/superform-xyz/superform-core/actions/runs/7018510375/job/19094554230?pr=413 and check how we are conducting the invariant test https://github.com/superform-xyz/superform-core/blob/invariant-ci/test/invariant/VaultShares.invariant.t.sol

[0xTimepunk]

@Evalir @gakonst @mattsse

An update to this issue

Using latest forge (forge 0.2.0 (7b45265 2023-11-21T00:18:30.267248000Z))

The sequence (when it reverts randomly), now says [FAIL. Reason: <no data>]

Encountered 1 failing test in test/invariant/VaultShares.invariant.t.sol:VaultSharesInvariantTest
[FAIL. Reason: <no data>]
	[Sequence]
....

[0xTimepunk]

@Evalir @gakonst @mattsse

You can now inspect a showcase of this error in our now public repository in https://github.com/superform-xyz/superform-core/actions/runs/7018510375/job/19094554230?pr=413 and check how we are conducting the invariant test in

https://github.com/superform-xyz/superform-core/blob/invariant-ci/test/invariant/VaultShares.invariant.t.sol

Could the issue be related with multiple deployments going on on each depth call during a run? This error doesn't show up when depth is = 1

[0xTimepunk]

Could be related to? https://github.com/foundry-rs/foundry/pull/6200

[0xTimepunk]

Closed related PR, but still looking forward to know why do these tests fail

[grandizzy]

Could be related to? #6200

@0xTimepunk pretty sure they're related, could you please recheck now since #6200 was fixed? thanks

[zerosnacks]

Ping @0xTimepunk, would be great if you could recheck this

[grandizzy]

@0xTimepunk please reopen if still an issue. Thank you!