Support nox sessions running in parallel

[FollowTheProcess]

How would this feature be useful?

This has brought up in other issues a few times (see #198 #88), on the latter one it seemed like there was some conflation of two different issues (supporting session parallelism and allowing sessions to launch background tasks). So I've broken this out into it's own issue.

#198 Brings up the idea of running tasks in the background from within a session (i.e. spinning up a webserver with something like session.background)

This issue is to track the idea of allowing Nox sessions to run in parallel

Describe the solution you'd like

The parallelism would be opt-in with a flag or some parameter to the session decorator (I have a preference for the former)

Such that you could run e.g. nox --parallel or nox --jobs 4 or similar so that the selected sessions would run in parallel.

My only real ideas for implementation could be a threadpool or wrapping each nox session in it's own subprocess as suggested in https://github.com/theacodes/nox/issues/88#issuecomment-383637043. I'm not sure which is best here.

You could also spin up an asyncio event loop but I suspect this would require a lot of code changes as async and await tend to spread everywhere as soon as you introduce them. There's also the issue that asyncio is never actually parallel so it's speedup factor would be heavily dependent on the work done by the sessions (CPU vs IO bound).

I think this would be reasonably complex to implement, especially coordinating and handling the output from the sessions (see https://github.com/theacodes/nox/issues/88#issuecomment-383637043 also) but I think, if done correctly would be hugely powerful

Describe alternatives you've considered

There's no real one I can think of, other than some sort of xargs hackery?

[FollowTheProcess]

Just some further thoughts on this (mainly brainstorming). There are two major sticking points that I can see:

• Synchronising or gathering up the console output from concurrently executing sessions and showing it in a sensible non-garbled way
• The fact that any given session can mutate the manifest (session.notify) and this is only known at runtime AFAIK?

Potential solutions I can see:

Output:

• Maybe if --parallel is used we restrict the output to only a single "this session passed" or "this session failed" type message?
• Not sure whether this exists in python but in rust I know you can take a lock on stdout

Mutable manifest:

• Easiest solution (but probably the worst): Document that session.notify is problematic when used with --parallel and raise a warning in any sessions that do this. Maybe this is something to do in the short term and then improve in subsequent PRs?
• Better (but harder): Rather than looping over the manifest and executing each session, eagerly gather up the manifest and evaluating the session.notify calls (somehow?) then launch all the sessions in a thread/process pool e.g. concurrent.futures

Obviously it would be possible for noxfile authors to accidentally create race conditions or other problems but I think the only way of dealing with this would be good documentation on the parallelism and what sorts of things to avoid doing in sessions if you want to use parallelism e.g. all actions that directly modify source code should be contained in a single session to avoid things like black and isort racing with each other.

If anyone has any ideas or input I'm keen to hear it 👍🏻

I'll try and play around with some things and see where I end up.

[anselor]

I don't know the internals of how nox behaves, but a couple ideas to mind for me that could address some of the above concerns. My biggest concern is I don't think I'd always want a noxfile to parallelize all the things. I'd want control over which things are run sequentially and which things are run in parallel.

First, I'll describe my mental model of what nox is doing and the basis for some of my thoughts. If this is wrong then some of my ideas may not quite fit. Seems like each nox session is run serially in definition order in the noxfile. If you use the -s option to restrict to specific session(s). If a session requires a subsequent session to also run, the session.notify() call is how you tag another session to also run. It occurs to me that I've never tried to use notify for a session defined earlier than the session being run - I don't know if that's possible.

So, I'm going to think of each thing running in series as a 'stage' going forward. So, that means that by default each session is a stand-alone stage containing session.

I'd suggest that for parallel execution we could have a way to tag multiple sessions as belonging to the same stage. Could be as simple as a decorator with a name, like: nox.stage('my parallel tests'). At the completion of all sessions in a stage, aggregate all of the session notifies and evaluate what to do from there.

In this scheme, parallel execution is baked into the noxfile so the command line switch would make more sense as --no-parallel or --force-sequential

This gives the developer control over what makes sense to run in parallel and what makes sense to run serially.

May make sense to either:

• Disallow notify between sessions contained within the same stage
• Allow it by waiting for the stage to complete and running the stage again as needed.

Other possible scenarios:

Explode out tests in parallel python versions:

@nox.stage("parallel tests")
@nox.session(python=['3.8', '3.9', '3.10'])
def stuff():
   ...

Explode out parallel executions with parameters:

@nox.stage('parallel tests')
@nox.parameterize('foo', ['b', 'a', 'r'])