feat(core): support long-running tasks as dependencies

[luxaritas]

Current Behavior

Running a task where dependencies do not exit causes the task to stall on the dependency

Expected Behavior

If a task is long-running and not the primary task, it will wait until any declared outputs are present, but otherwise it will not block. If any dependencies are noted as long-running, task output will be prefixed with the package name

Open Questions

• Is there a good way to add automated tests for this?
• Is the output format appropriate?
• Is using the current definition of a "long running task" reasonable?
• Where should this behavior be documented?

Related Issue(s)

Fixes #5570

[nx-cloud[bot]]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit ef1351b2387ecaccc3145af3745ae713ab99f307. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this branch

---

✅ Successfully ran 12 targets

• nx affected --target=build --base=6132ad747248eb6c456ac9f5bcd08fba6fc877c9 --head=ef1351b2387ecaccc3145af3745ae713ab99f307 --parallel=3
• nx affected --target=e2e --base=6132ad747248eb6c456ac9f5bcd08fba6fc877c9 --head=ef1351b2387ecaccc3145af3745ae713ab99f307 --exclude=e2e-detox,e2e-js,e2e-next,e2e-workspace-create,e2e-workspace-integrations,e2e-workspace-core,e2e-react,e2e-web,e2e-angular-extensions,e2e-angular-core,e2e-cli,e2e-nx-plugin,e2e-cypress,e2e-node,e2e-linter,e2e-jest,e2e-a...
• nx affected --target=e2e --base=6132ad747248eb6c456ac9f5bcd08fba6fc877c9 --head=ef1351b2387ecaccc3145af3745ae713ab99f307 --exclude=e2e-storybook,e2e-react-native,e2e-detox --parallel=1
• nx affected --target=lint --base=6132ad747248eb6c456ac9f5bcd08fba6fc877c9 --head=ef1351b2387ecaccc3145af3745ae713ab99f307 --parallel=3
• nx affected --target=test --base=6132ad747248eb6c456ac9f5bcd08fba6fc877c9 --head=ef1351b2387ecaccc3145af3745ae713ab99f307 --parallel=1
• nx build typedoc-theme
• nx-cloud record -- yarn check-commit
• nx-cloud record -- yarn check-imports
• nx-cloud record -- yarn depcheck
• nx-cloud record -- yarn documentation
• nx-cloud record -- yarn nx format:check --base=6132ad747248eb6c456ac9f5bcd08fba6fc877c9 --head=ef1351b2387ecaccc3145af3745ae713ab99f307
• nx-cloud record -- yarn nx workspace-lint

---

Sent with 💌 from NxCloud.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
nx-dev	✅ Ready (Inspect)	Visit Preview	Jun 11, 2022 at 5:27AM (UTC)

[luxaritas]

Just realized this may not work properly with run-many. The way I implemented this, it blocks if the task is for the initiating project, but what if there are multiple initiating projects? I suppose it should still work as long as the max parallel processes is high enough to cover all roots - is that sufficient, or should blocking be initiated somewhere else?

[GYatesIII]

@luxaritas Are you saying that the total number of running watchers cannot exceed the --parallel option/default? That makes sense, but to me the ideal would be that there would be no cap on watchers? Except for a an irrationally large max to protect against accidental loops.

For what it's worth, this feature would help my team greatly if it can be merged into the library. I've been following #5570 and this PR for the last few months.

[luxaritas]

No - I'm saying the total number of root tasks would be limited. Let's say you had 3 apps, each with 20 libs. If you run-many across all 3 libs and your max parallel is 3, no problems. But if your max parallel is 2, only 2/3 of those apps (and their dependencies) would be kicked off

[luxaritas]

That being said, this PR has been sitting for over a month without any feedback from the Nx team - is there anything I can do to get this queued for review? It's not a particularly large patch

[GYatesIII]

Understood. IMO, doesn't seem like a blocker for this PR as it's easily user-configurable to adjust the --parallel option. A nice-to-have would be to detect this issue at the start by analyzing the dep-graph and exit with a helpful error.

That being said, this PR has been sitting for over a month without any feedback from the Nx team - is there anything I can do to get this queued for review? It's not a particularly large patch

^^ this is mainly why I'm commenting. Hopefully to help add some urgency to get this in.

[juristr]

Hey, thanks a lot for the PR 🙇 .

The team is looking into it, but we think we might need some more thought around this and make sure it is designed properly. While watching for outputs might work in most situations there might be a more elegant and robust approach.

Sorry for the delay in responding but there has been lots of stuff going on 😅.

[luxaritas]

Thanks for the response! I'm happy to make adjustments, and would certainly be interested in how this could be made more robust and usable.

[FrozenPandaz]

Hi, let's keep discussing this in the issue while we design this properly. I'm going to close the PR for now until we settle on a design.

We can reference this solution in the issue. :+1: Thank you for working on it!

[github-actions[bot]]

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.