test: run tests on multiple node versions

[dymart]

[jbedard]

Note the current test failure seems to be because the median test run is faster then expected? See poll_spec.ts

[alan-agius4]

Is this something that you still want to land?

[jbedard]

Yeah we'd still like to do this one. I had I think all but one test green but have been working on https://github.com/angular/angular-cli/pull/23074 instead lately.

[jbedard]

@alan-agius4 there is one TODO left here but I think this is worth landing as-is for now. WDYT?

[jbedard]

The flakyness of these tests is very odd, and I think a lot of it has nothing to do with this change. Things like @angular-devkit/core not being found errors 🤔

[alan-agius4]

The flakyness of these tests is very odd, and I think a lot of it has nothing to do with this change. Things like @angular-devkit/core not being found errors 🤔

Strange as I haven’t seen any similar flakes on the main branch recently.

[jbedard]

@alan-agius4 we'd like to merge this as-is without actually enabling the multiple versions and just setting up the toolchains + BUILDs.

It seems like these tests are very flaky and doubling the number of them makes it fail what seems like 100% of the time. Specifically the packages/angular_devkit/build_angular tests which are put under the LARGE_SPECS variable name, the rest actually seem ok if we want to enable the rest (14/15 of them). This test even fails sometimes with no changes at all (like the PR currently is - only adding the toolchains config).

[alan-agius4]

@jbedard can you please rebase? As I am unable to do that for you as edits by maintainers is not allowed for this PR.

[alan-agius4]

Since this change went in, test flakiness have increased.

One of the main errors is

java.lang.OutOfMemoryError: Java heap space

[jbedard]

@alan-agius4 can we increase the circleci resource_class?

[jbedard]

Or it might actually be better as a separate circleci step per node version? Then we're doubling the number of steps instead of number of concurrent tests. That is probably required for the e2e tests so might be a good prefactor that we can do first...

[alan-agius4]

We have discussed this during our meeting today and for the time being until we find out the cause of the increase in memory usage. We will like to disable testing on multiple node versions.

The main reasons for this is that we don’t have many targets which justice usage of 14Gb memory considering we also use RBE. An increase the resource class would also double the credit usage if we go with 2 X-large.

I am curious to know which additional “steps” you thing we can share.

[jbedard]

The increase in memory usage is from doubling the number of tests running concurrently in the test circleci job.

Today we're running the different node versions concurrently in one circleci step within the test job. We could do the same as the e2e-tests and use matrix params to create a separate job per node version, or create a separate step per node version within the test job so they run sequentially within a single job.

[alan-agius4]

The increase in memory usage is from doubling the number of tests running concurrently in the test circleci job.

But we are using RBE so they are not actually run on circle. Also I wouldn’t expect 14GB of memory to be used just to run these tests considering we don’t have many targets.

[jbedard]

But we are using RBE so they are not actually run on circle.

Right... you're sure bazel reporting those means it is the circleci machine and not the remote one?

[jbedard]

But we are using RBE so they are not actually run on circle.

Looking at the bazelrc... isn't that only configuring bazel build (not test) to run remotely?

[alan-agius4]

test inherits from build.

https://docs.bazel.build/versions/main/guide.html#option-defaults

[jbedard]

If it is running remotely why would browser-tools/install-chrome be required? Is there any way to verify it's actually running remote? I'm not too familiar with circleci or remote exec...

[alan-agius4]

It does appear that there are some tests that run locally.

INFO: 679 processes: 215 remote cache hit, 288 internal, 40 local, 4 processwrapper-sandbox, 85 remote, 47 worker.

[jbedard]

Right, there are a few tagged as no-remote I think due to protractor/webdriver issues: https://github.com/angular/angular-cli/blob/9171543f06641293cb97d5a801a626a889243db2/packages/angular_devkit/build_angular/BUILD.bazel#L298-L299 (and another one down a bit)

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity. Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}