optimism
optimism copied to clipboard
ethereum-optimism
CI can't generate coverage
CI seems unable to generate coverage as part of the step contracts-bedrock-coverage. This is important because the step is responsible for generating and uploading the coverage to Codecov, which means that changes, such as new tests, may not make it to Codecov, preventing certain PRs from passing all CI checks.
To validate whether this was due to an error from changes in my branch, I created a fresh branch from develop. There, I slightly modified two comments as to make the system not skip generating coverage and made a PR. Again, CI failed, precisely at the point of running forge coverage --report lcov, as can be seen here: https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/37495/workflows/[%E2%80%A6]4-8d5e-c36cba321263/jobs/1687717/parallel-runs/0/steps/0-105.
There's an issue posted in Foundry's repo that outlines a similar problem, which, as it says in the description, belongs to tests of Optimism.
The fact that I cannot replicate this error locally (macOS Sonoma 14.2.1, M3 Max) seems unlikely to be due to a mismatch in Forge versions, since my local computer and CI are using the same version: forge 0.2.0 (53b15e6 2023-12-14T00:17:09.405850859Z).
@tynes pointed out that the issue could originate here: https://github.com/ethereum-optimism/optimism/blob/355bdf337a9331418f1f41d8451a1a4cd56fdd94/packages/contracts-bedrock/test/setup/Setup.sol#L80 which gets wonky when 2 diff compiler versions are used. More specifically, the error could arise from the code not being able to find the artifact (where the error msg is coming from).
Hi, @0xfuturistic! Can I begin to work on this issue? First, by trying to reproduce it on my system (I'm using Sonoma 14.1, but can also test this out on Ubuntu 22.04).
I've been going through this repo and OP stacks docs, and this issue doesn't seem to have anyone else working on this.
@AryanGodara unfortunately it seems to be widely unreported, but if you look at other recent PRs in the optimism monorepo you'll see that CI fails at the same spot!
I'll go through the recent PRs to see the point of failure and understand the problem better. Do you know any specific place to begin looking apart from this, or do I start from the different workflow directories and try to look things up?
I also want to understand the monorepo structure better😅, maybe that happens as I go through this PR
for sure! it's a tricky problem. @tynes could you provide some direction?
for sure! it's a tricky problem. @tynes could you provide some direction?
Hi, is there any update on this?
I'll go through the recent PRs to see the point of failure and understand the problem better. Do you know any specific place to begin looking apart from this, or do I start from the different workflow directories and try to look things up?
I also want to understand the monorepo structure better😅, maybe that happens as I go through this PR
The tricky thing here is that it only appears to happen in CI. We use a pinned version of foundry that can be found in .foundryrc. In CI, I am not sure if we are using the same version. That could be the first place to check, it could be a versioning thing. See if you can repro the failure on the latest forge. If not, see if you can repro the failure on the pinned version
I'll go through the recent PRs to see the point of failure and understand the problem better. Do you know any specific place to begin looking apart from this, or do I start from the different workflow directories and try to look things up? I also want to understand the monorepo structure better😅, maybe that happens as I go through this PR
The tricky thing here is that it only appears to happen in CI. We use a pinned version of foundry that can be found in
.foundryrc. In CI, I am not sure if we are using the same version. That could be the first place to check, it could be a versioning thing. See if you can repro the failure on the latest forge. If not, see if you can repro the failure on the pinned version
Oh okay, I'll start from there then. I'll keep you updated on this, it may take a day, but I'm working on it in the meantime.
Hey, when I opened that issue on the foundry repo I was actually running an older version of the Optimism tests pre-foundry migration. I actually ran an updated version recently and I'm encountering the same setup error occasionally again. I'm fairly sure the last one was due to some cheat code race condition and I wouldn't be surprised if that's also the case here, since the full foundry setup uses considerably more cheat codes.
A small side note: Am I the only one where the actual coverage report is missing major parts of the actual coverage?
For a quick repro, when running forge coverage --match-path test/L1/L2OutputOracle.t.sol, the coverage is completely empty except for the file test/mocks/NextImpl.sol.
I can create a separate issue for this though if it is not isolated to my setup.
Hey, when I opened that issue on the foundry repo I was actually running an older version of the Optimism tests pre-foundry migration. I actually ran an updated version recently and I'm encountering the same setup error occasionally again. I'm fairly sure the last one was due to some cheat code race condition and I wouldn't be surprised if that's also the case here, since the full foundry setup uses considerably more cheat codes.
can you try running pnpm clean in packages/contracts-bedrock and in the root of the monorepo make nuke && make clean (in addition to make devnet-clean if you are using a Docker devnet). then run pnpm install && make op-node op-proposer op-batcher && pnpm build. then you can run the tests packages/contracts-bedrock pnpm test.
@0xfuturistic I can run the tests without issues and they all pass. It's just that the coverage report shows a lot of them as empty. The example for the L2OutputOracle tests is just a minimal repro that should show coverage but it's empty.
Does the invocation I shared actually report coverage on your end?
This has been hacked around for now, the problem is that the contracts don't compile without the optimizer on