Allow _test targets to specify it cannot be run with itself in parallel

[AdamSvetec]

Description of the problem / feature request:

Currently using the feature --runs_per_test arg to run all tests multiple times. Many tests are not safe to be run concurrently. The tag "exclusive" guarantees a given test runs with no other tests but this causes testing times to be really slow. Really I just want to specify that a given test can't be run with itself concurrently. It can run at the same time as other tests, just not at the same time as itself.

Feature requests: what underlying problem are you trying to solve with this feature?

Could another tag be added "self_exclusive" or something that stops single test from being executed concurrently with itself but still concurrently with other tests

What operating system are you running Bazel on?

Ubuntu 16.04

What's the output of bazel info release?

release 0.23.2

Have you found anything relevant by searching the web?

Something similarish here: https://github.com/bazelbuild/bazel/issues/2525 . This fixes bug that would've affected us but doesn't quite address our use case.

[sdtwigg]

Wouldn't the reason a test cannot run with itself (some sort of non-hermiticity or resource contention issue) make it highly likely another (potentially future) test also cannot run with this thus leading to eventually just adding the exclusive tag?

(For implementers reference, this almost assuredly involves changing skyframe/TestCompletionFunction.java and will conflict with a pending refactoring I am working on.)

[AdamSvetec]

Potentially, in the case of shared socket or something yes... Most of the one's I'm investigating are file based which would not conflict with each other...

We have a pretty good scheme for ensuring tests don't share resources as they all run in parallel currently, we didn't really factor in the parallel with itself. Certainly fixable on our side, but seems theoretically much easier to enforce in bazel. Although your internal comment makes me think this will be headache for you as well. I'm surprised this hasn't been addressed already tbh. Are you aware of this breaking anyone else?

[sdtwigg]

For implementation, it actually wouldn't be that bad (just noted that in case someone else tries it to start there and talk to me first because there will be a merge conflict in some direction). Really just a matter of figuring out how to thread the 'self exclusive' tag into TestCompletionFunction.

My main fear is the feature as-is seems a bit too particular of a behavior and I am afraid it would encourages bad testing practices. (The parallel runs of the test 'should' be occurring in different sandboxes so touching files via absolute paths outside the sandbox would seem to be a hermiticity violation.) I'm definitely interested in second opinions though.

[AdamSvetec]

ah I see. I probably need to read more about bazel sandboxing. File wise I'm sure we're not using best practices and those should be easily fixable. socket wise I assume that's not something that's handled by bazel sandbox?

[aiuto]

Some hardware types gave me another real world use case a few weeks ago.

• Some tests need to use an actual piece of hardware linked to the machine
• The tests obviously must linearly take access to the hardware
• They want to be a Bazel user running tests locally on their desktops (with the attached test hardware)

One way to think of this might be the Action groups stuff @juliexxia worked on. We could say that the test executor platform needs "hardware-X", so that tests are only scheduled to build machines with the hardware, and then add some way to serialize tests around that resource requirement "hardware-X"

OTOH, that idea may be crap. I'm just throwing it out there.

[sdtwigg]

The custom resource group is a little more tempting as it allows us to solve other variants of the problem (e.g. being able to run exactly two of the test at a time or, as I noted earlier, bridging the gap for two tests that must run exclusive to each other but not necessary to everything else.).

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 2.5 years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[fmeum]

@sdtwigg Has this been solved by https://bazel.build/reference/command-line-reference#flag--local_extra_resources?

[brentleyjones]

I assume that only applies to local execution. Ideally we would have a way that to specify custom resource requirements and have platform executors see those when scheduling. For example, for iOS we want to limit the number of simulator tests run on a single machine, but with RBE we want it to be based on simulators available on each executor.

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.