[rush] Expose project graph.

[dfee]

Please prefix the issue title with the project name i.e. [rush], [api-extractor] etc.

Is this a feature or a bug?

• [x] Feature
• [ ] Bug

Please describe the actual behavior. rush change along with rush build and other commands build and navigate a DAG. This DAG would be very useful for CI and analysis.

I'm unsure of the best format for exposing this information, but it could be either via command-line or in a file managed by Rush.

If this DAG is flattened (which is fine) there are two highly relevant features that would be helpful:

1. upstream dependencies of the package
2. downstream dependents of the package

here is a yaml example (though it doesn't /need/ to be in yaml):

---
packages:
  "@mono/package-a":
    dependencies: []
    dependents:
      - "@mono/package-b"
      - "@mono/package-c"
  "@mono/package-b":
    dependencies:
      - "@mono/package-b"
    dependents:
      - "@mono/package-c"
  "@mono/package-c":
    dependencies:
      - "@mono/package-a"
      - "@mono/package-b"
    dependents:
      - "@mono/package-d"
  "@mono/package-d":
    dependencies:
      - "@mono/package-c"
    dependents: []

Notice in the case of @mono/package-d:

• we're not requiring that @mono/package-a report grandchildren as dependencies
• we're not requiring that it reports grandparent dependents.

This should be enough for us to "walk" the dependencies programmatically.

In addition, if there was an option to list those packages that have changed (e.g. rush change --list) then that would be helpful for informing the CI of which packages need to be tested when branches are pushed. Note, this is in addition to the DAG description above, but is the most immediate use-case.

[iclanton]

This is already exposed via Rush's API (in a way). You need to do some manual calculation to determine if each project dependency is inside the monorepo, but the dependency list and the APIs to do that are there. This could definitely be improved, though.

Here's a little program that will print a JSON-formatted dependency graph:

import {
  RushConfiguration,
  PackageJsonDependency
} from '@microsoft/rush-lib';
import { IPackageJsonDependencyTable } from '@rushstack/node-core-library';

const rushConfiguration: RushConfiguration = RushConfiguration.loadFromDefaultLocation({
  startingFolder: process.cwd()
});

interface IDependencyGraphEntry {
  dependencies: string[];
  dependents: string[];
}

const dependencyGraph: { [projectName: string]: IDependencyGraphEntry} = {};

function ensureAndGetDependencyGraphEntry(projectName): IDependencyGraphEntry {
  if (!dependencyGraph[projectName]) {
    dependencyGraph[projectName] = { dependencies: [], dependents: [] };
  }

  return dependencyGraph[projectName];
}

for (const project of rushConfiguration.projects) {
  const thisProjectEntry: IDependencyGraphEntry = ensureAndGetDependencyGraphEntry(project.packageName);

  function addDependencies(dependencies: IPackageJsonDependencyTable): void {
    for (const dependencyName in dependencies) {
      if (
        rushConfiguration.projectsByName.has(dependencyName) &&
        !project.cyclicDependencyProjects.has(dependencyName)
      ) {
        thisProjectEntry.dependencies.push(dependencyName);

        const dependencyEntry: IDependencyGraphEntry = ensureAndGetDependencyGraphEntry(dependencyName);
        dependencyEntry.dependents.push(project.packageName);
      }
    }
  }

  addDependencies(project.packageJson.dependencies);
  addDependencies(project.packageJson.devDependencies);
}

console.log(JSON.stringify(dependencyGraph, undefined, 2));

[devin-fee-ah]

This is awesome. I unfactored(?) your approach (in many more lines)... as hopefully this is a good recipe for future comers.

Honestly, I'd not even though to use @microsoft/rush-lib or @rushstack/node-core-library. I had no idea that those exposed the internals (maybe that's a documentation opportunity... or I'm simply aloof).

Anyway, thank you so much for the foundational recipe. I think this could be useful as a built-in. However, for now this is good enough 😄 .

import {
  RushConfiguration,
  RushConfigurationProject,
} from "@microsoft/rush-lib";
import { IPackageJsonDependencyTable } from "@rushstack/node-core-library";

type DependencyGraphEntry = {
  dependencies: string[];
  dependents: string[];
};
type DependencyGraph = Record<string, DependencyGraphEntry>;

type EnsureAndGetDependencyGraphEntryOptions = {
  packageName: string;
  dependencyGraph: DependencyGraph;
};
function ensureAndGetDependencyGraphEntry({
  packageName,
  dependencyGraph,
}: EnsureAndGetDependencyGraphEntryOptions): DependencyGraphEntry {
  if (dependencyGraph[packageName] === undefined) {
    dependencyGraph[packageName] = { dependencies: [], dependents: [] };
  }

  return dependencyGraph[packageName];
}

type AddDependenciesOptions = {
  dependencies: IPackageJsonDependencyTable | undefined;
  dependencyGraph: DependencyGraph;
  project: RushConfigurationProject;
  rushConfiguration: RushConfiguration;
};
function addDependencies({
  dependencies = {},
  dependencyGraph,
  project,
  rushConfiguration,
}: AddDependenciesOptions): void {
  const dependencyGraphEntry = ensureAndGetDependencyGraphEntry({
    dependencyGraph,
    packageName: project.packageName,
  });
  Object.keys(dependencies).forEach((dependencyName) => {
    if (
      rushConfiguration.projectsByName.has(dependencyName) &&
      !project.cyclicDependencyProjects.has(dependencyName)
    ) {
      dependencyGraphEntry.dependencies.push(dependencyName);

      const dependencyEntry = ensureAndGetDependencyGraphEntry({
        dependencyGraph,
        packageName: dependencyName,
      });
      dependencyEntry.dependents.push(project.packageName);
    }
  });
}

type ProcessPackageOptions = {
  dependencyGraph: DependencyGraph;
  project: RushConfigurationProject;
  rushConfiguration: RushConfiguration;
};
function processPackage({
  dependencyGraph,
  project,
  rushConfiguration,
}: ProcessPackageOptions) {
  [
    project.packageJson.dependencies,
    project.packageJson.devDependencies,
    project.packageJson.peerDependencies,
  ].forEach((dependencies) =>
    addDependencies({
      dependencies,
      dependencyGraph,
      project,
      rushConfiguration,
    }),
  );
}

function makeDependencyGraph() {
  const dependencyGraph: DependencyGraph = {};
  const rushConfiguration = RushConfiguration.loadFromDefaultLocation({
    startingFolder: process.cwd(),
  });
  rushConfiguration.projects.forEach((project) =>
    processPackage({
      dependencyGraph,
      project,
      rushConfiguration,
    }),
  );
  return dependencyGraph;
}

// ok... so you just want to pretty print it...
const dg = makeDependencyGraph();
console.log(JSON.stringify(dg, undefined, 2));

[octogonz]

I had no idea that those exposed the internals (maybe that's a documentation opportunity... or I'm simply aloof).

I opened https://github.com/microsoft/rushjs.io-website/issues/63

I think this could be useful as a built-in.

Do you have any thoughts about how it would be designed? A CLI command? A richer API?

[dfee]

I also came up with this to show me dependency order:

/**
 * Returns a list of ordered dependencies based on dependency-order.
 * @param dependencyGraph the dependency graph
 * @param packageName the target packageName to find dependents for
 * @param visited the packages that have already been visisted
 * @param entries the ordered list of dependents
 */
export function postorderTraverseDependencyGraph(
  dependencyGraph: DependencyGraph,
  packageName: string,
  visited: Set<string> = new Set([]),
  entries: string[] = [],
): string[] {
  if (!visited.has(packageName)) {
    visited.add(packageName);
    dependencyGraph[packageName].dependencies.forEach((dependency) =>
      postorderTraverseDependencyGraph(
        dependencyGraph,
        dependency,
        visited,
        entries,
      ),
    );
    entries.push(packageName);
  }
  return entries;
}

Consider it a poor mans version of what rush-lib also provides... somewhere 😄

@octogonz ... now that I know that rush-lib is a thing... well, there are two things:

1. I'd love to see rushjs.io and rushstack.io merged. the power granted by this monorepo tooling is amazing (though sometimes there are issues / opportunities).
2. specific to the dependency graph situation... I think as much of the API that can be exposed should be exposed. With the help of @iclanton I exposed the DAG in a way where I could iterate dependencies. This is already done by rush-lib (somewhere) and I shouldn't need to re-invent the wheel (regardless of how much fun it was!).

In general I see the rushstack.io being the missing handbook for rush. And, outside a couple "gotchas" (i.e. things we've already discussed, like rush's authoritarian need for a re-definition of the CLI in command-line.json), I think this is hands down the most elegant tool for JS monorepos (or any monorepos that I've worked with). Sure, a coat of fresh paint, and a fresh push to the open-source community wouldn't be bad either... but I do (unfortunately) feel like we're a secondary audience.

[ujwal-setlur]

I personally would love to see this as a CLI command/output that I can call from my CircleCI helper bash script

[halfnibble]

What if we add a --dependencies parameter to rush list?

E.g. rush list --dependencies @microsoft/my-project Would list all the dependencies of @microsoft/my-project and support the other flags like --json, --version, and --path by displaying the equivalent for the dependencies?

I'd also like to alias rush ls to rush list

[bacebu4]

Might be a good idea to generate the output with mermaid, gonna be much easier than simple binary image

Also this seems like a good idea because both GitHub and GitLab have the ability to visualize this syntax

UPD: after making the graph by myself I realize it's not a good idea to add it directly into Rush since a monorepo easily can have tens of packages and the graph becomes non-informative at all so you have to do some filtering manipulation in order to get where you want

[joelzimmer]

I'll just throw out here that nx/lerna offer a nx graph command to view the dependency graph. It would be incredibly powerful to have an equivalent for rush