Gas report bug

[gigamesh]

Component

Forge

Have you ensured that all of these are up to date?

• [X] Foundry
• [X] Foundryup

What version of Foundry are you on?

forge 0.2.0

What command(s) is the bug in?

forge test --gas-report

Operating System

macOS (Intel)

Describe the bug

forge test --gas-report outputs reports for a seemingly random subset of my contracts, and the contracts I'm testing aren't included. If I specify the list of contracts I care about as instructed in the docs, it doesn't output any report. 🤔

reproduction example: https://github.com/gigamesh/forge-gas-bug

following up from here (cc @mds1 🙏 ) https://t.me/foundry_support/17808

[mds1]

I've also noticed this. Last week myself and someone else ran forge test --gas-report and got different gas report contract outputs on each of our machines. We ensured we were using the same version, both ran forge clean, and basically made sure the conditions were identical but could not figure out where the difference was coming from.

I also suspect this may be related to https://github.com/foundry-rs/foundry/issues/1815

[onbjerg]

Seems like the contracts just aren't identified. Are the traces decoded?

[onbjerg]

Ping @gigamesh is this still present?

[gigamesh]

@onbjerg yes unfortunately. I just tried running foundryup, then forge test --gas-report in this repo, and got the same result. If gas_reports = ["*"], it only prints reports on a subset of contracts, but if I specify gas_reports = ["ArtistV5"] (for example), nothing prints.

[mds1]

Ok, I think there's two (maybe related?) gas report issues here:

1. Gas reports seem to be printed in order based on when tests finish. But tests are run in parallel and therefore this is not deterministic. I'd propose printing the table sorted alphabetically by path.
2. It seems sometimes not all contracts are shown? Here are three gists from myself, @apbendi, and @garyghayrat. We checked out this commit of our repo, cd contracts-periphery; forge install; make snapshot-gas. This just dumps the output of the forge test --gas-report to a file.
   • https://gist.github.com/mds1/07636301420e1019c85953b2c5c18625
   • https://gist.github.com/apbendi/e44f122e3fe28c89fc2015f5e430cbd5
   • https://gist.github.com/garyghayrat/b403a10b4e896cf33a5cc178c3297889

We got (un)lucky in the sense that all orders are correct in these gists, but we did observe differing orders between runs. In these gists you'll notice:

• Differing contract sizes in the gas reports (no one changed anything from that commit)
• Differing gas numbers
• Gary's gist is missing a lot of gas report tables, he noticed the same behavior no both linux and macOS and we're all using the same foundry version (Ben and I are both on macOS)

[0xadrii]

Is there any news on this? Unfortunately I'm facing the exact same issue described by @gigamesh. @onbjerg