[bug] `runDepsInParallel: false` does not seem to work correctly when the dependent task has its own dependencies

[niieani]

Describe the bug

I have a clean task that should run before any building starts. In order to facilitate that, I have created a "build" task, that should run serially - clean first, then execute remaining build-tasks in parallel. Unfortunately for some reason the first thing that runs is cli:build-library-bundle-cli and clean in parallel. Not sure why this is happening, but it looks like a bug perhaps?

Steps to reproduce

Here's my moon.yml:

$schema: https://moonrepo.dev/schemas/project.json
tasks:
  tsc-project:
    command: tsc --project tsconfig.json
  prepare-package:
    command: yarn run prepare
    deps:
      - ~:tsc-project
  build-tasks:
    deps:
      - cli:build-library-bundle-cli
      - ~:tsc-project
      - ~:prepare-package
    inputs: []
  clean:
    command: rm -rf build && echo done
    options:
      cache: false
  build:
    deps:
      - ~:clean
      - ~:build-tasks
    inputs: []
    options:
      runDepsInParallel: false

Expected behavior

When I run build, I would expect the following order:

1. clean
2. cli:build-library-bundle-cli + tsc-project (in parallel)
3. prepare-package (because it depends on tsc-project)

instead, I'm seeing the following:

1. cli:build-library-bundle-cli
2. clean
3. tsc-project
4. prepare-package

Environment

node v20.12.1
@moonrepo/cli 1.23.3

[milesj]

Yeah this has been a bug? for a while. I'm not entirely sure if it is a bug: https://github.com/moonrepo/moon/issues/744

That setting only applies to direct dependencies, and not grandchildren dependencies. I'm not sure if it's even possible to make the entire hierarchy serial in a DAG, especially when other tasks get pulled in.

[niieani]

From my perspective it is definitely a bug and might just force me to switch away from moon to another task runner if you confirm that you don't plan to address this. I like moon so far, so I would rather not :(

The main requirement for me is that the clean step runs before any build-related tasks, which should run in parallel. In other way: a higher order sequence of internally parallelized tasks.

How would you solve that problem using moon with its current limitations?

[milesj]

From my perspective it is definitely a bug and might just force me to switch away from moon to another task runner if you confirm that you don't plan to address this. I like moon so far, so I would rather not :(

How would you achieve this in other runners, because they parallelize transient deps as well.

How would you solve that problem using moon with its current limitations?

Well you couldn't. You can't have some deps be serial, and others be parallel. You would need to make them all explicitly serial:

  build:
    deps:
      - ~:clean
      - cli:build-library-bundle-cli
      - ~:tsc-project
      - ~:prepare-package
    inputs: []
    options:
      runDepsInParallel: false

The current problem is that when building the DAG, the edges between nodes happens in isolation, so we don't have access to grandchildren deps. I honestly don't know how to fix it (I've tried a few times). I would love some insight on this from someone who knows graphs very well.

Another thing is that the shape of the DAG also becomes unwieldy, especially in CI when there's more than 1 task being ran, as a serial deps chain by 1 task will affect everything else in the DAG, really disrupting parallelization.

[niieani]

Would you suggest not modeling this dependency using the DAG and instead splitting the commands run on the CI into two serial commands? Like this:

$ moon run :clean
$ moon run :build

Now that I think of it, building this complexity into the DAG seems like an overkill. Would it make sense to add a different type of task into moon that just runs all the specified tasks, as if they were run from the CLI, one by one? Literally all I wanna do is just have moon call clean first, and then build once its finished cleaning. Saying that build "depends on" clean is even semantically incorrect. It doesn't - I just want it to happen before the build.

[milesj]

Yeah right now everything is kind of powered by the task graph. I would like to support other ways to handle this, as forcing the DAG to support all these edge cases is really difficult.