chore: upgrade to nx 18

[getlarge]

Continue #595 Fixes #594 Supersedes #625

[nx-cloud[bot]]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 5b7c9bbdb6d56ff23669db39b08fc4d65b682728. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

🟥 Failed Commands
nx affected:e2e --parallel=3

✅ Successfully ran 5 targets

• nx code-pushup -- --persist.outputDir=.code-pushup --persist.filename=report --persist.format=json --persist.format=md
• nx affected -t unit-test --parallel=3 --coverage.enabled
• nx affected -t integration-test --parallel=3 --coverage.enabled
• nx affected:build --parallel=3
• nx affected:lint --parallel=3

---

Sent with 💌 from NxCloud.

[getlarge]

@BioPhoton @matejchalk It looks like there is a problem with the e2e tests setup:

• cli-e2e and nx-plugin-e2e declares the same globalSetup which is the relative path to global-setup.e2e.ts.
• in global-setup.e2e.ts the ephemeral package registry is created
• This means that when running the e2e tests in parallel, there are two registries on two different ports since Verdaccio increments the port number when one is busy.
• but the folder is the same for both...

Which leads to race conditions and conflicts with package publishing/installation.

[getlarge]

Also, the e2e tests implicitly depend on Chromium being available in the OS, and believe it or not, some people do not have the Chrome browser installed.

[matejchalk]

It looks like there is a problem with the e2e tests setup:

• cli-e2e and nx-plugin-e2e declares the same globalSetup which is the relative path to global-setup.e2e.ts.
• in global-setup.e2e.ts the ephemeral package registry is created
• This means that when running the e2e tests in parallel, there are two registries on two different ports since Verdaccio increments the port number when one is busy.
• but the folder is the same for both...

Which leads to race conditions and conflicts with package publishing/installation.

Ah yes, I believe this is another casualty of #618 being merged :sweat_smile:

Seems to be an easy fix for it, fortunately. In 2143dc5fa4910a0d8522efd39c43caa16e7c5938, I removed the Verdaccio setup from nx-plugin-e2e. It's actually not needed, nx g @code-pushup/nx-plugin:... works the same for local plugin as for NPM package. E2E tests pass in CI for Linux and MacOS, although Windows fails for some other reason (this was also introduced in #618 somehow, although still not sure why exactly).

[BioPhoton]

Also, the e2e tests implicitly depend on Chromium being available in the OS, and believe it or not, some people do not have the Chrome browser installed.

I have to set it as hard dependency it seems... thought lighthouse does it.

[matejchalk]

@getlarge We've just fixed the CI and Code PushUp workflows in main. So if you merge main into this branch, the workflows should pass and then we can merge this PR, hopefully :crossed_fingers:

[getlarge]

@getlarge We've just fixed the CI and Code PushUp workflows in main. So if you merge main into this branch, the workflows should pass and then we can merge this PR, hopefully 🤞

Just did and also updated the e2e setup process. I'll add comments in a second to explain...

[getlarge]

Apparently, the publishing is still flaky, I tried only once locally and the e2e tests were passing but not the second time. I got the same error as in the CI. I will fix it now.

[getlarge]

I am stuck in the CI hole with this PR. Do you happen to know why the e2e tests are timing out on Windows? It looks like they were already slow on the previous PR but are now stuck during the publishing phase.

Also, the Code PushUp workflow fails because a coverage file is missing, is it an expected behavior when some tests would not run because the lib is not affected?

[Error: ENOENT: no such file or directory, open 'coverage/plugin-js-packages/unit-tests/lcov.info'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'coverage/plugin-js-packages/unit-tests/lcov.info'
}

[BioPhoton]

Also, the Code PushUp workflow fails because a coverage file is missing, is it an expected behavior when some tests would not run because the lib is not affected?

It is an expected behaviour. If your tests fail the plugins fails.

I personally dont like it and would like to have it pass with the passed tests :) But I see the reasoning behind this decision.

[getlarge]

Apparently code coverage is not generated in my local environment. It concurs with the error in the CI

[getlarge]

Thanks @BioPhoton for joining the debugging, we found out that the coverage is not generated due to this issue.

[getlarge]

@BioPhoton apparently, enabling coverage for all the tests left them hanging. The CI jobs kept running forever. Yet another Vitest surprise!

[getlarge]

@BioPhoton @matejchalk I've got news regarding Nx Vitest executor. The executor now correctly handles the Vitest flags since this PR was merged and released. I upgraded Nx to 18.3.4 and confirmed it works. I also noticed that the tests are left hanging when coverage is enabled and the test suite is empty! We could remove those tests' configurations, right?

One bad thing with the upgrade though, is that many unit tests fail for the ESLint plugin.

[vmasek]

Amazing, these fails seem like some kind of cache access lock 🤔

Error: EACCES: permission denied, mkdir '/test/.nx/cache'
   ❯ mkdirSync ../../node:fs:1380:26
   ❯ Function.Module._load ../../node_modules/nx/src/native/index.js:61:7
   ❯ Object.<anonymous> ../../node_modules/nx/src/native/native-bindings.js:184:31
   ❯ Function.Module._load ../../node_modules/nx/src/native/index.js:67:25
  
  ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
  Serialized Error: { errno: -13, code: 'EACCES', syscall: 'mkdir', path: '/test/.nx/cache' }

[getlarge]

Amazing, these fails seem like some kind of cache access lock 🤔

Error: EACCES: permission denied, mkdir '/test/.nx/cache'
   ❯ mkdirSync ../../node:fs:1380:26
   ❯ Function.Module._load ../../node_modules/nx/src/native/index.js:61:7
   ❯ Object.<anonymous> ../../node_modules/nx/src/native/native-bindings.js:184:31
   ❯ Function.Module._load ../../node_modules/nx/src/native/index.js:67:25
  
  ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
  Serialized Error: { errno: -13, code: 'EACCES', syscall: 'mkdir', path: '/test/.nx/cache' }

locally I get another FS error code:

ENOENT: no such file or directory, mkdir '/test/.nx/cache'  

In this test suite, i noticed that process.cwd is mocked to return the /test path, which could maybe interfere with some internal Nx functions... I'll keep on digging.

[getlarge]

The other bad news is that the e2e setup still hangs for the Windows environment. Reminder, I might push a test commit that restores the previous process of publishing packages to the local registry.

[vmasek]

Amazing, these fails seem like some kind of cache access lock 🤔

Error: EACCES: permission denied, mkdir '/test/.nx/cache'
   ❯ mkdirSync ../../node:fs:1380:26
   ❯ Function.Module._load ../../node_modules/nx/src/native/index.js:61:7
   ❯ Object.<anonymous> ../../node_modules/nx/src/native/native-bindings.js:184:31
   ❯ Function.Module._load ../../node_modules/nx/src/native/index.js:67:25
  
  ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
  Serialized Error: { errno: -13, code: 'EACCES', syscall: 'mkdir', path: '/test/.nx/cache' }

locally I get another FS error code:

ENOENT: no such file or directory, mkdir '/test/.nx/cache'  

In this test suite, i noticed that process.cwd is mocked to return the /test path, which could maybe interfere with some internal Nx functions... I'll keep on digging.

I see mkdir

[vmasek]

Sorry for closing, that was miss click 😅

I am wondering if it is allowed to make directories or add files (mkdir) inside of a GH action runtime inside of a /tests/ folder

[github-actions[bot]]

Code PushUp

🤨 Code PushUp report has both improvements and regressions – compared target commit c67d39825fce898230d2ce77ddd2446cf77aec78 with source commit a7d6d1b1b7f74a245a542e70f722a96e1343849e.

🏷️ Categories

🏷️ Category	⭐ Current score	⭐ Previous score	🔄 Score change
Performance	🟡 88	🟡 60
Updates	🟡 84	🟡 70
Code coverage	🟡 73	🟡 70
Accessibility	🟢 98	🟢 100
Best Practices	🟢 100	🟢 100	–
SEO	🟡 86	🟡 86	–
PWA	🟡 63	🟡 63	–
Bug prevention	🟢 100	🟢 100	–
Code style	🟢 99	🟢 99	–
Security	🟢 94	🟢 94	–
Custom checks	🟡 67	🟡 67	–

🗃️ Groups

👍 3 groups improved, 👎 1 group regressed

🔌 Plugin	🗃️ Group	⭐ Current score	⭐ Previous score	🔄 Score change
Lighthouse	Performance	🟡 88	🟡 60
JS Packages	NPM outdated dependencies	🟡 84	🟡 70
Code coverage	Code coverage metrics	🟡 73	🟡 70
Lighthouse	Accessibility	🟢 98	🟢 100

17 other groups are unchanged.

🛡️ Audits

👍 13 audits improved, 👎 1 audit regressed, 8 audits changed without impacting score

🔌 Plugin	🛡️ Audit	📏 Current value	📏 Previous value	🔄 Value change
Lighthouse	Minimizes main-thread work	🟩 1.9 s	🟥 5.1 s
Lighthouse	Background and foreground colors have a sufficient contrast ratio	🟥 0	🟩 0
JS Packages	Outdated NPM optional dependencies.	🟨 3 outdated package versions (1 major, 2 minor)	🟥 1 major outdated package version
Lighthouse	Total Blocking Time	🟩 160 ms	🟥 950 ms
Lighthouse	First Contentful Paint	🟩 1.2 s	🟨 2.7 s
Lighthouse	Speed Index	🟩 1.9 s	🟨 4.1 s
Lighthouse	First Meaningful Paint	🟩 1.2 s	🟨 2.8 s
Lighthouse	Max Potential First Input Delay	🟥 370 ms	🟥 580 ms
Lighthouse	Largest Contentful Paint	🟨 2.8 s	🟨 3.3 s
Lighthouse	Time to Interactive	🟩 2.6 s	🟨 4.0 s
Code coverage	Line coverage	🟨 72 %	🟨 67 %
Code coverage	Function coverage	🟨 68 %	🟨 64 %
Code coverage	Branch coverage	🟨 84 %	🟨 82 %
JS Packages	Outdated NPM dev dependencies.	🟨 53 outdated package versions (26 major, 19 minor, 7 patch, 1 prerelease)	🟨 52 outdated package versions (26 major, 20 minor, 5 patch, 1 prerelease)
Lighthouse	Metrics	🟩 2576	🟩 4039
Lighthouse	Properly size images	🟥 Potential savings of 240 KiB	🟥 Potential savings of 240 KiB
Lighthouse	JavaScript execution time	🟩 0.1 s	🟩 0.1 s
Lighthouse	Server Backend Latencies	🟩 10 ms	🟩 60 ms
Lighthouse	Avoids enormous network payloads	🟩 Total size was 432 KiB	🟩 Total size was 432 KiB
Lighthouse	Uses efficient cache policy on static assets	🟨 11 resources found	🟨 11 resources found
JS Packages	Outdated NPM prod dependencies.	🟩 9 outdated package versions (4 minor, 5 patch)	🟩 7 outdated package versions (3 minor, 4 patch)
Lighthouse	Initial server response time was short	🟩 Root document took 0 ms	🟩 Root document took 10 ms

501 other audits are unchanged.

[getlarge]

The other bad news is that the e2e setup still hangs for the Windows environment. Reminder, I might push a test commit that restores the previous process of publishing packages to the local registry.

After reverting to using the previous publish target, it looks like the CI workflow is going to hang during the Windows e2e job. I looked at many past workflow runs to compare, and I could not find a single job run for E2E-test (windows-latest) that actually runs the build. They all seem to pick the build output from the cache.

Workflows:

• https://github.com/code-pushup/cli/actions/runs/9175838368/job/25229713877
• https://github.com/code-pushup/cli/actions/runs/9161552409/job/25186658890
• https://github.com/code-pushup/cli/actions/runs/9158674616/job/25177512068
• https://github.com/code-pushup/cli/actions/runs/9155531208/job/25168089689

Now, I wonder if the issue is related to cli:build task and not the registry setup.

[getlarge]

In the CI workflow, I saw the build job is not using a strategy matrix, maybe this should change?

[BioPhoton]

Hi @getlarge imo the scope of the PR was a bit too ambitious.

Just focusing on the update and not trying to refactor testing and use new APIs etc would have made it easier to go on. Furthermore, there is now another PR on Nx 19 open.

What should we do here? Keeping it hanging around is I guess counter productive.

Is there information inside that we can add as a comment to the new PR or anything that you think helps?

Let me know how to go on. And what you think. 🙏

[getlarge]

Hi @getlarge imo the scope of the PR was a bit too ambitious.

Just focusing on the update and not trying to refactor testing and use new APIs etc would have made it easier to go on. Furthermore, there is now another PR on Nx 19 open.

What should we do here? Keeping it hanging around is I guess counter productive.

Is there information inside that we can add as a comment to the new PR or anything that you think helps?

Let me know how to go on. And what you think. 🙏

Hey, I guess it’s time to close it then! I don’t think this was ambitious, at the time the real blocker was the windows e2e tests which were left hanging and I never found (the time to find) the reason. You encountered a similar situation in the Verdaccio setup repository right?

[BioPhoton]

Yes let's close it!

Also, I neither found time (or a window laptop) to debug it