Implement fine-grained parallelization

[ctheune]

The current state of parallelisation can only really leverage this over multiple hosts (because the execnet connection will block). Fine-grained parallelization would mean we can also trigger multiple components on the same host at the same time.

I have drafted a way to do this with execnet over here: https://gist.github.com/ctheune/eda98d8667f65c962c03c0ce2b92a11a

However, the existing code is quite involved and requires detailed cleanup of both the remote core and the controller and will benefit from revamping the output management (and maybe switch over to structlog).

• [ ] switch output to structlog, separate output data collection from terminal/log writing
• [ ] implement pattern from the gist
• [ ] upgrade remote_core protocol to multiplex commands

[gforcada]

A bit unrelated, but around the parallelism topic: it would be great/needed to have a way to limit the amount of parallelism.

Scenario: I'm deploying some new changes to our Plone based website where we have 9 workers. If all 9 workers at the same time run buildout and finally the process, via supervisor, is restarted, it might very well be that some requests will be refused due to the lack of servers available, as HAProxy will not see any worker available roughly at the same time.

If instead, we could limit the parallelism to, say 3 workers, then the deployment will still be 3x faster than batou 1.x already, but still allow the website the be responsive.

[ctheune]

Yes, thanks for that! I forgot to add it as a separate ticket. I have a plan in my head already:

• for the moment I want to get the mechanism for paralellization ready
• i'm expecting to collect 2-3 projects that want/need to use it, and then
• gather the specific requirements how we want to limit things based on the actual requirements from the projects

I have a couple of things in my head how to provide syntax for those limits but I really want to base it on actual use cases ...

[gforcada]

@ctheune where is the form to sign up? :raising_hand_man: :smile:

[ctheune]

He, sorry @gforcada that I missed your signup ... there's an incoming PR from @wosc's colleagues that will allow setting parallelization limits similar to '-j' on make. That can be turned to "1" for the old behaviour. I'm also finishing up the general Python 3 migration and then I can take on more projects ... that want to migrate and help those :)