Ideas about splitting slow `l3build check/doc` into smaller even runs

[muzimuzhi]

When using l3build in a CI, users may want to split a slow job running l3build check and/or l3build doc into several smaller even jobs run in parallel, to speed up the CI. The "even" means those smaller jobs are expected to take time as equal as possible.

Splitting by bundles and by config files are natural ways, but they usually result in uneven jobs, and they can't help in splitting a single config further.

To split l3build check, assuming each test takes equal times to check,

• LaTeX2e already used --first <name> and --last <name>, which is far from ideal. See commit https://github.com/latex3/latex2e/commit/46f836ed22c7a83066fe26df8fbc39319e2b5dbc (Split LuaTeX main test executions in GH Actions, 2023-08-07) and latex3/latex2e#1073.
• Some new options like --first-num <number> and --last-num <number> which denotes the number of first and last tests to run seems a little better than --first/--last.
• Options that allow user to run the first third, second third, and last third (in general kth nth where 1 <= k <= n) is even better.

To split l3build doc, assuming user knowns which names are specifically slow, for example the source3 in l3kernel bundle, latex3/latex3 repository,

• A new option --skip <names> would help a lot. It will opens the possibility of running l3build doc source3 and l3build doc --skip source3 in parallel jobs.
• My comment in https://github.com/latex3/latex3/pull/1295#issuecomment-1776060520 suggested adding support for glob pattern !, so l3build doc '!source3' would be the same as l3build doc --skip source3.

To split l3build ctan, one possible way is to provide a pair of new options --no-check and --no-doc.

How do other testing tools for general-purpose programming languages handle this task?

[josephwright]

To split l3build ctan, one possible way is to provide a pair of new options --no-check and --no-doc.

This part I think is a definite no: the entire reason that the ctan target is set up the way it is is that we want to ensure that a release build does 'everything' before getting to the .zip file creation.

[josephwright]

* Some new options like `--first-num <number>` and `--last-num <number>` which denotes the number of first and last tests to run seems a little better than `--first/--last`.

* Options that allow user to run the first third, second third, and last third (in general kth nth where 1 <= k <= n) is even better.

Sounds like --test-block=<m>:<n> or similar - 'test the <m>th block when dividing the tests into <n>' blocks?

[josephwright]

Could just be --block, so with a short version -b?

[josephwright]

To split l3build doc, assuming user knowns which names are specifically slow, for example the source3 in l3kernel bundle, latex3/latex3 repository,

* A new option `--skip <names>` would help a lot.
  It will opens the possibility of running `l3build doc source3` and `l3build doc --skip source3` in parallel jobs.

I'd imagine --exclude or --exclude-files, based on other tools: I guess one would likely go for just --exclude as we don't use file names, and this would need to apply to tests and to docs.

[davidcarlisle]

I just came to open an issue here to suggest a new way to filter check runs and saw it would mostly duplicate this, so adding here.

The check runs at https://github.com/latex3/tagging-project/actions are very slow. At one point earlier configs were split on first/last but more than most test suites, tests are renamed and.or moved between the test config directories as their status changes (eg a test moves from testfiles-partial to testfiles-compatible ). That meant the names used in the github actions matrix to split runs up with first and last became wrong leading to duplication and/or tests being omitted completely.

A safer way of splitting the runs for parallelisation that doesn't rely on specific test names at the split points would be good. Numeric would be possible, or a more general Lua patterns match filter might be good so you could run all tests with name containing BAD or all tests starting a-m then all tests starting n-z I think this would be simple to implement, just guard the insert(names,name) with if name:match(pattern) where pattern is the passed in filter pattern?

[josephwright]

@davidcarlisle Stlll comes down to what interface we want: thoughts?

[davidcarlisle]

@josephwright I thought l3build --pattern '^[a-m]' which would just insert an if(name:match('^[a-m]) ooh more or less exactly where you have excludetests /`includetests hm maybe just need to be able to set them via an argument rather than via a separate config file.

[josephwright]

@josephwright I thought l3build --pattern '^[a-m]' which would just insert an if(name:match('^[a-m]) ooh more or less exactly where you have excludetests /`includetests hm maybe just need to be able to set them via an argument rather than via a separate config file.

We can do that, cetainly - I guess then --exclude and --include would be enough?

[davidcarlisle]

I guess then --exclude and --include would be enough?

probably depends exactly how they work, presumably

--include=^a --exclude=BAD

should process tests starting with a that don't have BAD in their name, and exclude everything else.

[zauguin]

I see an issue with using name patterns because that makes it rather inflexible about what the separation is. Something like ^[a-m] is nice until you reach the point that many slow tests are similar and end up having similar names. Of course you can make the patterns more complicated, but then making sure that you are actually catching everything gets tricky again. (actually even just filtering on the first letter gets annoying once someone adds a test not starting with a letter)

Maybe we could just have a small variant of --last which is an exclusive bound instead of an inclusive bound. Then you can reliably ensure that all tests are run by ensuring that the --last-exclusive from the previous test run matches the --first of the next run. Also we could implement these a bit differently than we do at the moment to compare names with first and last instead of assuming that the first and last are actually existing names, then something like a-m and n-z could still be done by running with --first a --last-exclusive n and --first n.

[FrankMittelbach]

Maybe we could just have a small variant of --last which is an exclusive bound instead of an inclusive bound. Then you can reliably ensure that all tests are run by ensuring that the --last-exclusive from the previous test run matches the --first

seems like a pragmatic solution to me

[josephwright]

That's something like --next?

[davidcarlisle]

@zauguin I had wondered about asking for start from next, but actually the main breakage here was that packages were moved from testfiles-unknown to testfiles-compatible or testfiles-incompatible so the test runs on testfiles-unknown with start=xxx didn't match at all. Of course moving packages around without updating the CI matrix is user error but easily done as normally you can change a test status with no issue, unless you happen to change the status of a test that had been used as a boundary in the test matrix.

That's why I was looking for a way of splitting the runs that did not depend on existing test names at all.

But actually the main reason things are slow here is the file copying, locally here (with a non-removable virus checker in operation) it is unusable to use

l3build check -c config-partial somepackage

as it copies all tests for all status partial packages before starting and this takes far longer (several minutes) than running the test, I always copy the test directory to an empty configuration such as testfiles-unknown before doing anything.

For tests such as these the copying isn't really needed (and complicates packages that assume sub directories) and an option to run tests in-place would greatly speed things up.

[zauguin]

@zauguin I had wondered about asking for start from next, but actually the main breakage here was that packages were moved from testfiles-unknown to testfiles-compatible or testfiles-incompatible so the test runs on testfiles-unknown with start=xxx didn't match at all. Of course moving packages around without updating the CI matrix is user error but easily done as normally you can change a test status with no issue, unless you happen to change the status of a test that had been used as a boundary in the test matrix.

Ok, but that should still be fixed if we adjust the --first / --last logic to not rely on the specified tests existing. If we just check that the current test compares higher than --first and lower than --last, then it doesn't matter anymore that --first and --last might refer to tests which have already been moved away.