nuke
nuke copied to clipboard
nuke-build
Parallel execution of targets
I see a ton of value in this and would like to help move it along if possible. I haven't dug into the code around target execution much yet, as I'm sure you have some thoughts on this already. I assume https://github.com/nuke-build/common/tree/feature/parallel2 has the latest WIP for this.
As far as the user-facing side of this, to me the minimum requirement would be:
- Allowing the user to opt in to parallel target execution. This could be via parameter, overridden NukeBuild property (could also simply be overriding the default parameter value), and/or environment variable.
Some things I could see users maybe wanting to do:
- Mark specific targets to not be parallelized (CPU-heavy, prone to async issues, or just user preference).
- Configure the max number of parallel targets.
Aside from setting up the execution plan and actually invoking targets in parallel, a couple interesting challenges might be:
- Logging/IO - printing output from async logs in a readable manner (e.g., buffered and flushed per-target for standard console, printed in realtime with flowId when running in TeamCity, etc.)
- Target execution times - I don't know how precise this feasibly can or should be, but showing the actual times spent executing each target—i.e., excluding any time an async task spends waiting for a thread on which to continue. Maybe this isn't really an issue depending on the number of parallel tasks.
If you have any suggestions on where I can pitch in or pick up on work, please let me know! I have limited time to spend on this every couple weeks, but it seems like a fun problem and I think it would be very beneficial feature to have.
Allowing the user to opt in to parallel target execution. This could be via parameter, overridden NukeBuild property (could also simply be overriding the default parameter value), and/or environment variable.
Agreed. Parameters are already aware of environment variables.
Mark specific targets to not be parallelized (CPU-heavy, prone to async issues, or just user preference).
This would be solely defined via the dependency model. If two targets do not depend on each other (execution/order/trigger dependency), they can run in parallel.
Configure the max number of parallel targets.
Not needed imo.
Logging/IO - printing output from async logs in a readable manner (e.g., buffered and flushed per-target for standard console, printed in realtime with flowId when running in TeamCity, etc.)
Flow ids would be the easiest for a start. My preferred option would be to flush only when a target has finished. https://github.com/nuke-build/nuke/issues/107 would be great for that too, to show some kind of progress.
Target execution times - I don't know how precise this feasibly can or should be, but showing the actual times spent executing each target—i.e., excluding any time an async task spends waiting for a thread on which to continue. Maybe this isn't really an issue depending on the number of parallel tasks.
I wouldn't change anything around that. Let's wait for an actual request.
The first next step would be to create a appropriate data structure from which we can tell which targets can be run in parallel. I've stubbed some classes for that in https://github.com/nuke-build/common/commit/b7b836643af0ea37454b37bb32c132fadc855cbd#diff-8c68ce1c46e703ab9fd73f4d53581127R224. However, I've never really managed to invest some time implementing the Compose method https://github.com/nuke-build/common/commit/b7b836643af0ea37454b37bb32c132fadc855cbd#diff-8c68ce1c46e703ab9fd73f4d53581127R27.
@matkoch I see 500 stars and still no parallel execution?!
https://nukebuildnet.slack.com/archives/C9Q99MFU0/p1550410859094700
Is this still something that is planned to be included? I can see a PR exist for this functionality. This would be utterly amazing to have included 😄
I see a PR but aside from @matkoch keeping it up to date with develop nothing seems to be happening there? I assume this means its still a planned feature but something is blocking this. Is there any help needed in completing this?
Hi @matkoch, what's happening with this feature? Is there a plan to release it soon?
Would it be possible to get a little bit more information about this please? The PR is still being kept up to date but no plans to merge and release. Just helps to avoid confusion for the community. Kind regards!
Not sure why this is confusing. I already mentioned that I have no plans and that there's nothing the community can help with. I also don't know anything about a branch being kept up-to-date. Where does that come from?
Hi @matkoch the branch and open PR can be found here: https://github.com/nuke-build/nuke/pull/526
It looked like the PR has had a bunch of commits to it recently, keeping it up to date with develop. From my naïve perspective, it looked like community members had submitted a contribution and it was a potential candidate for the community to help with to get this feature across the line. Thanks for your time and responses.
He might be referring to my comment but that comment is now 8 months old.
A bummer this is not going to make it in NUKE though. Would be a killer feature as that really would put NUKE even further ahead compared to other build automation tools.
I would love to see this feature in Nuke, because at the moment we still have to depend on a particular provider to build more complex pipelines and leverage parallelism
I would love to see this feature in Nuke, because at the moment we still have to depend on a particular provider to build more complex pipelines and leverage parallelism
We have Cake scripts. I was looking into Cake and Nuke for a long time to introduce parallelism... To none result.
That is why I'm looking into Bullseye which can parallelize tasks.
Writing everything in the C# script and just running it on any platform with one command is I think better than making complex pipelines within that platform.
That is what I learned after this. :D Btw, handling artifacts between jobs (which runs in dockers) is the worst!
