foundry
foundry copied to clipboard
foundry-rs
Fuzz testing only catches console.log in `setUp()` and in failing test functions, not passing functions
Component
Forge
Have you ensured that all of these are up to date?
- [X] Foundry
- [X] Foundryup
What version of Foundry are you on?
forge 0.2.0 (2e99f7f 2022-12-03T00:08:28.619060629Z)
What command(s) is the bug in?
forge test
Operating System
Linux
Describe the bug
Similar to issue #655, which was closed as stale (creating a new issue since there have been changes since that issue, and mine may be subtly different).
Only console logs in failing functions and setUp() are showing in the Logs section of the results, above the traces, but not the logs in passing functions. I can see the other calls to console::log in the traces, but they're not getting formatted.
Running 1 test for src/test/compound/Comptroller-fork-diff.sol:TestComptroller
[PASS] test_claimComp_diff_before_after(uint8) (runs: 1, μ: 2924860, ~: 2924860)
Logs:
Number of addresses: 74
Traces:
[4852548] TestComptroller::setUp()
├─ [0] console::log(Number of addresses: %s, 74) [staticcall]
│ └─ ← ()
...
[2924860] TestComptroller::test_claimComp_diff_before_after(5)
├─ [0] console::log(Address %s, 0x000008599eB841e58b3e13305Da3dA5df5Ed8fFD) [staticcall]
│ └─ ← ()
This would be fine if I were only trying to do one run, but since I'm fuzzing, I'd like to see all of the logs without printing all of the traces.
I have added a minimized test contract in my repo for reproducing this bug: https://github.com/webthethird/solidity-diff-fuzz-upgrades/blob/main/src/test/test_logging.sol
Simple way to reproduce:
import "forge-std/Test.sol";
contract Test5 is Test {
function setUp() public {
console2.log("Test5.setUp()");
}
function test_FuzzVals(uint x) external {
console2.log("Test5.test_FuzzVals()");
}
}
As shown in the screenshot below, forge test with any verbosity shows the Test5.setUp() log, but never shows the Test5.test_FuzzVals() log. However, in the trace, both are shown.
When all runs of a fuzz test are successful, the trace shows the values from the last run. The console.logs should do the same
running into the same issue. tried both console and console2 but its not outputting . only prints if it is in setUp()
cc @Evalir, I just verified this bug is still occurring on the latest nightly using the repro in https://github.com/foundry-rs/foundry/issues/3843#issuecomment-1402311583. Guessing this is a simple fix
So, I can confirm this is an issue—even though just fixing it could make everyone's CI or local runs extremely noisy as a default. @mds1 this is somewhat easy fix, but I wonder if we should put this behind some kind of flag or increase the verbosity level needed for printing fuzz logs?
even though just fixing it could make everyone's CI or local runs extremely noisy as a default
We already would show the logs if it was a regular test and not a fuzz test, so I think this is ok? Unclear why we'd want to handle the console.logs differently between the two, it seems unintuitive. To hide the logs you'd just run with -v as the verbosity
Any update on this issue? I really need to show logs in fuzz tests for both failed and success results.
Bump to this, also looking to get output on each fuzz test, not just the failing ones
having this feature would be great.
I think the only readability issue will be there because fuzzing runs the function too many times and logging every time will spam the terminal
I think we could track it woth https://github.com/foundry-rs/foundry/issues/3844
I think we could track it woth https://github.com/foundry-rs/foundry/issues/3844
how should this feature be implemented? i didn't take every situation into consideration in my last try. i'm still using locally built foundry when fuzz logs are needed 😅
