facebook
[Compiler Bug]: Coverage report shows missing branch coverage
What kind of issue is this?
- [ ] React Compiler core (the JS output is incorrect, or your app works incorrectly after optimization)
- [x] babel-plugin-react-compiler (build issue installing or using the Babel plugin)
- [ ] eslint-plugin-react-compiler (build issue installing or using the eslint plugin)
- [ ] react-compiler-healthcheck (build issue installing or using the healthcheck script)
Link to repro
https://github.com/ValentinGurkov/vitest-react-compiler-missing-coverage-repro
Repro steps
Describe the bug
Hello,
I've been testing out the new React compiler and have noticed that the code coverage report becomes incorrect with the compiler turned it. I believe it may be related to the way it changes the output react component code.
I've created a minimal reproduction repository to demonstrate the issue: 👉 https://github.com/ValentinGurkov/vitest-react-compiler-missing-coverage-repro, but I also want to share my findings here:
The button we are going to test is a simple one:
export const Button = () => {
return <button>Click me</button>;
};
As well as its test:
import { page } from '@vitest/browser/context';
import { Button } from '@repo/components/ui/button.js';
import { describe, expect, it} from 'vitest';
import { render } from 'vitest-browser-react';
describe(Button, () => {
it('renders with default variants', async () => {
render(<Button />);
const button = page.getByRole('button', { name: 'Click me' });
await expect.element(button).toBeInTheDocument();
});
});
The coverage report with the React compiler turned on looks like:
✓ browser (chromium) test/components/ui/button.test.tsx (1 test) 9ms
✓ Button > renders with default variants 9ms
Test Files 1 passed (1)
Tests 1 passed (1)
Start at 20:34:46
Duration 632ms (transform 0ms, setup 133ms, collect 28ms, tests 9ms, environment 0ms, prepare 93ms)
% Coverage report from v8
------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
------------|---------|----------|---------|---------|-------------------
All files | 100 | 50 | 100 | 100 |
button.tsx | 100 | 50 | 100 | 100 | 2
------------|---------|----------|---------|---------|-------------------
The component has no branching logic while the report shows are the are missing some. I believe I've managed to set up the vitest configuration to also log the transformed component and we have:
import { jsxDEV } from "react/jsx-dev-runtime";
import { c as _c } from "react/compiler-runtime";
export const Button = () => {
const $ = _c(1);
let t0;
if ($[0] === Symbol.for("react.memo_cache_sentinel")) {
t0 = /* @__PURE__ */ jsxDEV("button", { children: "Click me" }, void 0, false, {
fileName: "/Users/<my-user>/Projects/vitest-react-compiler-coverage/src/components/ui/button.tsx",
lineNumber: 6,
columnNumber: 10
}, this);
$[0] = t0;
} else {
t0 = $[0];
}
return t0;
};
It looks like the React compiler introduces a conditional for memoization purposes. That may explain the coverage issue, though it raises the question: is it even possible to get 100% branch coverage for components compiled like this?
The test without the React compiler looks like:
- react({
- babel: {
- plugins: [
- ["babel-plugin-react-compiler", {}]
- ],
- },
- }),
+ react(),
✓ browser (chromium) test/components/ui/button.test.tsx (1 test) 10ms
✓ Button > renders with default variants 10ms
Test Files 1 passed (1)
Tests 1 passed (1)
Start at 20:36:42
Duration 490ms (transform 0ms, setup 36ms, collect 6ms, tests 10ms, environment 0ms, prepare 92ms)
% Coverage report from v8
------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
------------|---------|----------|---------|---------|-------------------
All files | 100 | 100 | 100 | 100 |
button.tsx | 100 | 100 | 100 | 100 |
------------|---------|----------|---------|---------|-------------------
The code coverage is 100% as expected. The output code also has no conditions:
import { jsxDEV } from "react/jsx-dev-runtime";
export const Button = () => {
return /* @__PURE__ */ jsxDEV("button", { children: "Click me" }, void 0, false, {
fileName: "/Users/<my-user>/Projects/vitest-react-compiler-coverage/src/components/ui/button.tsx",
lineNumber: 2,
columnNumber: 12
}, this);
};
Coming from https://github.com/vitest-dev/vitest/issues/7843#issuecomment-2812107855, it seems that babel's auxiliaryCommentBefore also does not have any effect on this.
How do we see test coverage working once the React Compiler becomes standard?
How often does this bug happen?
Every time
What version of React are you using?
19.1.0
What version of React Compiler are you using?
19.1.0-rc.1
Understanding and Fixing Missing Coverage from React Compiler Memoization in Vitest
❗️Issue Type Classification
Thanks for posting. React Compiler automatically memoizes components to make updates (re-rendering) faster. This includes memoizing JSX elements, such as the <button> in your example. Memoization uses if/else statements plus a cache, though the exact output may very well change in the future.
The 50% branch coverage you're seeing reflects the fact that the test doesn't exercise the update (re-render) case. It looks like you're using Vitest's default option for code coverage, backed by V8. It looks like there is another option, to use Istanbul, which may be able to take advantage of source maps to provide more precise coverage of the original, pre-transformed code (i'm not sure, you may need to experiment). https://vitest.dev/guide/coverage.html#coverage
V8 coverage uses source maps too.
If a compiler adds code during transform, and includes it in source maps, coverage tools will show it in the report. Generated code should not be added in source maps.
This affects all JS coverage tools, not just Vitest. Either remove it from source maps, or add coverage exclusion hints during transform.
@AriPerkkio Interesting, thanks for the tip. I'll experiment with trying to omit nodes from source maps, it has not been easy to find docs on this stuff (and how it interplays with Babel). We had naively assumed that if we skipped a loc property that the node would be emitted from source maps.
Agree with AriPerkkio – the core issue feels like the generated memoization branch showing up in source maps. Ideally, coverage should reflect the original source, not the compiler's internal optimizations.
Yes we agree, and this is why we intentionally don’t add source location to nodes for generated memoization logic. The problem is that Babel appears to synthesize incorrect source map information for those nodes, rather than excluding them as you’d expect.
I see that the related PR was closed, was the attempted fix unsuccessful?
I see that the latest attempt to fix this was, unfortunately, unsuccessful. What do you think are the next steps? Should we try to reach out to someone from the Babel team, and see whether they are keen on providing some insight?
Yeah, unfortunately I couldn't figure out how to get Babel to do the right thing here. Maybe it's possible somehow.
If you'd like to help, reaching out to the Babel team makes sense as the next step. All the information should be in the linked PR (#33051)
I just ran into this today. As a workaround (until this can be fixed), I just excluded the compiler when running the unit tests:
vite.config.js
export default defineConfig(({ command, mode }) => {
const baseConfig = {
plugins: [tailwindcss()],
test: {
globals: false,
environment: 'jsdom',
coverage: {
provider: 'v8',
...
},
...
},
...
};
let finalConfig = { ...baseConfig };
// if we are in test mode, we do not want the react compiler included
// it causes issues with coverage: https://github.com/facebook/react/issues/32950
// otherwise, we want to include the compiler
if (mode === 'test') {
finalConfig.plugins.push(react());
} else {
finalConfig.plugins.push(react({ babel: { plugins: ['babel-plugin-react-compiler'] } }));
}
... // other config setup
return finalConfig;
});
Sample of coverage when the react compiler is included:
Sample of coverage when the react compiler is excluded:
I just ran into this today. As a workaround (until this can be fixed), I just excluded the compiler when running the unit tests:
vite.config.js
export default defineConfig(({ command, mode }) => { const baseConfig = { plugins: [tailwindcss()], test: { globals: false, environment: 'jsdom', coverage: { provider: 'v8', ... }, ... }, ... };
let finalConfig = { ...baseConfig };
// if we are in test mode, we do not want the react compiler included // it causes issues with coverage: https://github.com/facebook/react/issues/32950 // otherwise, we want to include the compiler if (mode === 'test') { finalConfig.plugins.push(react()); } else { finalConfig.plugins.push(react({ babel: { plugins: ['babel-plugin-react-compiler'] } })); }
... // other config setup
return finalConfig; }); Sample of coverage when the react compiler is included:
Sample of coverage when the react compiler is excluded:
I am doing the same, but I worry that if there are places where we rely on the compiler optimisations, we might end up, for example, with constant re-renders in the tested component.
@obryckim We don't recommend using different configurations for development/tests and production — as a general rule, that's a recipe for extremely hard to diagnose bugs. (bikeshed alert) I have not personally found code coverage metrics to be especially meaningful — you can have full coverage and still miss cases — but if you find the coverage metrics more important than performance gains then I'd recommend disabling the compiler everywhere. I just wouldn't want you to risk gnarly bugs in the off chance you see different behavior. This is a subjective opinion, just my cents. It's your app, your call.
We aren't actively investigating this. See #33051 for more context on what we tried and why it didn't work with Babel.
Is there any mitigation for the moment? This is blocking me from adopting the compiler since it means we can't rely on branch coverage being correct. Just need to wait for a non-babel compiler integration?
Coming from https://github.com/vitest-dev/vitest/issues/7843#issuecomment-2812107855, it seems that babel's auxiliaryCommentBefore also does not have any effect on this.
Documentation on auxiliaryCommentBefore is extremely sparse, if anyone has context on how this is supposed to work we could explore this route.
Oh Man, there really needs to be a fix for this otherwise I can see adoption of the react compiler being lowered as people crash into this issue and then deciding NOT to use it. Like I am considering...
Coming from vitest-dev/vitest#7843 (comment), it seems that babel's auxiliaryCommentBefore also does not have any effect on this.
Documentation on
auxiliaryCommentBeforeis extremely sparse, if anyone has context on how this is supposed to work we could explore this route.
Has anyone posted a question or created an issue requesting better documentation for auxiliaryCommentBefore with the Babel team?