Test cases that do not fail test (and should as indicated in test name testFailTransferFromNotAuthorized)

[sambacha]

I opened this issue earlier in foundry-rs regarding an issue running solmate's tests in the benchmarks

https://github.com/foundry-rs/foundry/issues/7531

I think this should be addressed in solmate tests too, thing is that there are cases that do not fail test (and should as indicated in test name testFailTransferFromNotAuthorized), like using counterexample above in an unit test https://github.com/foundry-rs/foundry/issues/7531#issuecomment-2029209895

    function testFailTransferFromNotAuthorized_counterexample() public {
        address sender = address(0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496);
        address receiver = address(0x000000000000000000000000000000000000089a);
        uint256 id = 31;
        uint256 amount = 3642;
        amount = bound(amount, 1, type(uint256).max);

        token.mint(sender, id, amount);

        token.transferFrom(sender, receiver, id, amount);
    }

yields failure because test passes

Ran 1 test for src/test/ERC6909.t.sol:ERC6909Test
[FAIL. Reason: assertion failed] testFailTransferFromNotAuthorized_counterexample() (gas: 43212)
Logs:
  Bound Result: 3643

Traces:
  [559113] ERC6909Test::setUp()
    ├─ [504137] → new MockERC6909@0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f
    │   └─ ← [Return] 2518 bytes of code
    └─ ← [Stop] 

  [47424] ERC6909Test::testFailTransferFromNotAuthorized_counterexample()
    ├─ emit log_named_uint(key: "Bound Result", val: 3643)
    ├─ [25249] MockERC6909::mint(ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], 31, 3643)
    │   ├─ emit Transfer(caller: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], from: 0x0000000000000000000000000000000000000000, to: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], id: 31, amount: 3643)
    │   └─ ← [Stop] 
    ├─ [25892] MockERC6909::transferFrom(ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], 0x000000000000000000000000000000000000089a, 31, 3643)
    │   ├─ emit Transfer(caller: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], from: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], to: 0x000000000000000000000000000000000000089a, id: 31, amount: 3643)
    │   └─ ← [Return] true
    └─ ← [Stop] 

Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 994.36µs (159.48µs CPU time)

Ran 1 test suite in 767.02ms (994.36µs CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)

Failing tests:
Encountered 1 failing test in src/test/ERC6909.t.sol:ERC6909Test
[FAIL. Reason: assertion failed] testFailTransferFromNotAuthorized_counterexample() (gas: 43212)

whereas if rename test to testTransferFromNotAuthorized_counterexample it is passing

To prevent against panics when running forge benches probably here we should not assume execution is always success but just execute https://github.com/foundry-rs/foundry/blob/d94e3c631e2da7756af46c70f8f58b75563b7013/crates/forge/benches/test.rs#L17-L19

The cache/fuzz/failures cannot be found because bench clones and runs project in a temp dir which is wiped out after execution

Originally posted by @grandizzy in https://github.com/foundry-rs/foundry/issues/7531#issuecomment-2029209895

[grandizzy]

failure can be also reproduced by placing a failures file in cache/fuzz/ dir with following content

# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 22e4b1983d70b8b5cd9d0de968c12d8781238594f65b39328f157cdcadca620a # shrinks to 0xc5b21e8f0000000000000000000000007fa9385be102ac3eac297483dd6233d62b3e14960000000000000000000000000000000000000000000000000000000000000899000000000000000000000000000000000000000000000000000000000000001f0000000000000000000000000000000000000000000000000000000000000e3c