Build a set of projects with a single command

[filipesilva]

It would be useful to have a way to build a certain set of projects on a workspace.

Perhaps this can be achieved with global workspace targets, or via a filtering mechanism.

Followup to https://github.com/angular/angular-cli/pull/10397

[abhijeetkpawar]

Any plans for this feature?

[klemenoslaj]

Hi @filipesilva,

Every project in a workspace already contains package.json (obviously). We could use peerDependencies to declare the dependencies between projects of the workspace. Then build and test commands would be able to detect the dependencies and build the libraries in the correct sequence.

I have written myself a node executable that does exactly that. Follows the dependency tree and builds the libraries in the correct sequence.

If that is something you see as a valuable addition to Angular CLI I would be more than happy to contribute.

[filipesilva]

@klemenoslaj the approach we are looking at now would be able to do that implicitly, we hope. We're still finishing up the Architect API (this is what we use for build definitions).

[klemenoslaj]

@filipesilva, that sounds good :+1:

I however still like the idea of using peerDependencies, because it forces people to think about their dependencies (very important for published packages) and if that is done correctly, build sequence is automatically correct.

I can see how that can be a bit complex for someone, but I still think handling dependencies via package.json is a good way to go.

[RobinBomkampDv]

Are there any plans when it can be expected?

[jpleclerc]

Is it possible with the new CLI builders announced here : https://blog.angular.io/introducing-cli-builders-d012d4489f1b ?

If yes could we have an example ?

[meriturva]

Actually, the builder approach is just project related and not workspace related. We already implemented npm script that just analyzes all projects, builds dependency tree and calls sequentially build but still problems during watch mode: build is too slow!

During development time I guess only productive way is to change tsconfig path as described on https://github.com/angular/angular-cli/issues/10643#issuecomment-388389988

[siddharth1903]

@meriturva Is this possible for multiple libraries within a single project? I understand schematics exposes options to invoke external schematics and builder is just an extension of schematics!?

[meriturva]

@Goonersid our approach is just for a single project with multiple libraries. Right now we have all to test new build based on bazel before improving our internal solution.

[siddharth1903]

@meriturva I'm looking for a cli cmd that builds all libraries within a single project... Is it possible with v6 or the latest stable? Something like ng build-all

[meriturva]

@Goonersid you have to write you own task script and run as: npm run build-all

[probert94]

@Goonersid I was also looking for something like this and created a script, which reads the angular.json file and executes ng build for every project and every configuration. Also this script collects all failed builds and prints them to stdout at the end.

EDIT: I am using nrwl/nx, so I don't have to build libraries manually. So this script will probably not work for default Angular CLI.

The script looks like this:

import { ProjectType, WorkspaceSchema } from "@schematics/angular/utility/workspace-models";
import { execSync } from "child_process";
import { readFileSync } from "fs";

interface ExecParams {
  project: string;
  config: string;
}

interface ExecError extends ExecParams {
  message: string;
}

function buildAll() {
  const json: WorkspaceSchema = JSON.parse(readFileSync("./angular.json").toString());
  const errors: ExecError[] = Object.keys(json.projects)
    // Filter application-projects
    .filter(name => json.projects[name].projectType === ProjectType.Application)
    // Determine parameters needed for the build command
    .reduce<ExecParams[]>((arr, project) => {
      const proj = json.projects[project];
      let build = proj.architect && proj.architect.build;
      if (build) {
        arr = arr.concat(...Object.keys(build.configurations || {})
          .map(config => ({ project, config }))
        );
      }
      return arr;
    }, [])
    // Execute `ng build` and collect errors.
    .reduce<ExecError[]>((err, exec) => {
      try {
        console.log(`Building ${exec.project} (${exec.config}):`);
        execSync(`ng build --prod --project ${exec.project} --configuration ${exec.config}`, {stdio: "inherit"});
      }
      catch (error) {
        err.push({
          project: exec.project,
          config: exec.config,
          message: error.message
        });
      }
      console.log("\n");
      return err;
    }, []);

  // Conditionally log errors
  if (errors.length === 0)
    console.log("Completed");
  else {
    console.log("\n");
    errors.forEach(error => {
      console.error(`Building ${error.project} (${error.config}) failed:\n\t${error.message}`);
    });
  }
}

buildAll();

This script can be compiled to Javascript and then you can run it using NodeJs.

[klemenoslaj]

@Springrbua but you're not following any dependencies, therefore if some library depends on another this will break or work by accident.

[probert94]

@klemenoslaj You are right, I never used the normal Angular CLI for multi app projects and therefore forgot, that you need to build libraries yourself. I am using nrwl/nx, which automatically builds all needed libraries when executing ng build, so that's not a problem in my case.

[hiepxanh]

I think this is an important feature for enterprise project with many library

[otiai10]

I'm using too https://github.com/otiai10/too.js, hoping it would help

[DerHerrGammler]

Any plans actually?

[Tjommel1337]

A dirty hack (that ignores inner-dependencies) for windows would also be this BATCH-script:

DIR "projects\" /B /AD > "projects.txt"
FOR /f "usebackq delims=" %%f IN ("projects.txt") DO call ng build %%f
FOR /f "usebackq delims=" %%f IN ("projects.txt") DO call ng build %%f
DEL "projects.txt"

I do the FOR-loop twice because of an inner dependency in my projects. Also i use "call" to ignore errors in the first run.... dirty as hell but it's a workaround.

[krishnathota]

Any progress on this?

[hiepxanh]

you can use https://nx.dev/latest/angular/generators/workspace-generators to build set of project, I'm using it to generate 6 project and 20 lib support for it will 1 command, easy

[klemenoslaj]

As an exercise, I recently developed a little CLI tool that reads the dependencies from the TypeScript AST and executes the commands in the right sequence and parallel (if so specified).

I tried to use it on a quite large Angular CLI monorepo and the results were great. Dropping the sequential builds from 865s to 118s.

I did not publicly release this because a) It was an exercise b) Nx already does something like that. If there is however a need for that I'd be glad to push it to the wild.

Bottom line is, there is a lot of potential in this so if the community requires this I'm sure the Angular team will implement something similar. I have a feeling tho that this is a bit exotic request so it might never happen.

Expand to see CLI in action videos

https://user-images.githubusercontent.com/7548247/119224114-372b2a00-bafd-11eb-98c6-de2fa353364d.mov

https://user-images.githubusercontent.com/7548247/119224117-3abeb100-bafd-11eb-98bf-994a7d795a62.mov

https://user-images.githubusercontent.com/7548247/119224436-bc630e80-bafe-11eb-9554-00ca3e608398.mov

[hiepxanh]

interesting, that nice tool, I think you should share with people, it very good, your tool better than mine, I just using set of generator chain like this:

[krishnathota]

@klemenoslaj

This is pretty good having various options to build. I think you should publish it out too if that's okay for you though.

I also do not understand why would this be an exotic request. As we could create just one repo instead on multiple for multiple libraries and there is a high chance that a library depends on an other one. Doesn't it?

Either I don't understand why or I am completely doing things wrong. :thinking:

[klemenoslaj]

This is pretty good having various options to build. I think you should publish it out too if that's okay for you though.

@krishnathota I probably will, but need to think if I can cope with the maintenance that comes along with it.

I also do not understand why would this be an exotic request

It is just my feeling/assumption, not the fact. I think the majority of applications out there are built without libraries or they go for Nx. Again, that's just an assumption.

I'd be super happy seeying this in Angular CLI

[boeckMt]

We had the need too, to build multiple libraries depending on their dependency tree, in order to publish them with GitHub Actions to NPM.

For this we used depcheck and toposort to generate the execution order (based on peerDependencies) and then used a small script to run all ng build commands.

But we also would be happy to see "workspace commands" for ng test <project-list> and ng build <project-list> inside Angular CLI :)

[klemenoslaj]

@boeckMt this is almost the same solution we used to have before. 😂 nice!

We moved to the CLI I showcased above because it got very slow. since all builds are sequential, so it sometimes took very long (big monorepo).

I think if Angular CLI builds that in, they'd need to account for parallel builds.

Glad to see there is someone else with similar issues 😆

[boeckMt]

@klemenoslaj we only need this for for publishing our packages. At "runtime" we use the default ng serve or ng build with a path mapping for the Imports https://github.com/dlr-eoc/ukis-frontend-libraries/blob/master/tsconfig.json#L21 to our package scope.

e.g.

import { xyz } from '@scope/package';
...

// tsconfig.json
...
"paths": {
      "@scope/*": [
        "dist/*",
        "projects/*"
      ],
      ...
    }
...

[klemenoslaj]

We have this too :+1: We even generate it with out own schematics. But in pipelines we build libraries anyway, because sometimes ng-packagr can trip over some syntax that would work in directly linked application build.

---

I wonder if there is any chance of getting this into Angular CLI. @filipesilva WDYT?

[klemenoslaj]

@klemenoslaj the approach we are looking at now would be able to do that implicitly, we hope. We're still finishing up the Architect API (this is what we use for build definitions).

There is not much going on in this feature request. I would still like to offer my repository for possible inspiration but will leave it at that due to lack of interest.

@filipesilva feel free to have a peek at the following repository. I implemented an implicit approach, so no additional config is required whatsoever.

The repository: https://github.com/klemenoslaj/ong-cli

[vajahath]

This is indeed looks like a rough edge where the community seem to be struggling. I was able to stumble upon a lot of SO threads pointing to the need of this feature, for years.

To point out a few:

• https://stackoverflow.com/questions/50181181/angular-6-cli-how-to-make-ng-build-build-project-libraries
• https://stackoverflow.com/questions/66432674/ng-build-to-multiple-projects-angular
• https://stackoverflow.com/questions/39624749/how-to-build-multiple-applications-with-angular-cli
• https://stackoverflow.com/questions/56168702/how-to-build-multiple-apps-in-angular-7-project-in-a-single-build-command
• https://stackoverflow.com/questions/56884697/angular-build-process-with-multiple-projects-and-libraries
• https://stackoverflow.com/questions/52763631/how-to-make-ng-build-watching-multiple-apps
• https://stackoverflow.com/questions/65908672/how-to-make-angular-watch-multiple-libraries-for-changes-and-recompile-when-need
• https://stackoverflow.com/questions/61190511/angular-workpace-cant-build-project-without-having-to-build-other-projects-in
• https://stackoverflow.com/questions/56056362/angular-cli-doesnt-include-all-packages-on-ng-build-prod
• https://stackoverflow.com/questions/60473727/angular-build-watch-for-multiple-libraries-and-and-serve-to-another-app

and all of these threads seem to be very genuine.

It would be very great when we run serve/build, cli would automatically build its depended libraries as well. The solution should also be compatible when ng serving multiple apps.

As of now what I'm doing is open a terminal per library and run ng build --watch. Guess how many terminal tabs I'm keeping!

[hiepxanh]

I have to move nx to have this: https://nx.dev/generators/workspace-generators I can build a set of project easy, example

import { appAndLibSetting } from '../../new-tool/common-setting.tool';
import { Tree, formatFiles, readProjectConfiguration } from '@nrwl/devkit';
import { wrapAngularDevkitSchematic } from '@nrwl/devkit/ngcli-adapter';
import shellGenerator from '../shell';
import sharedGenerator from '../shared';

export default async function (tree: Tree, schema: any) {
  const domainName = `${schema.directory}-${schema.device}`;
  await createApplication(tree, schema);
  deleteUnecessaryFile(tree, schema, domainName);
  await createDependanceLibrary(tree, schema, domainName);

  await formatFiles(tree);
  return () => {};
}

function deleteUnecessaryFile(tree: Tree, schema: any, domainName: string) {
  const libraryRoot = readProjectConfiguration(tree, domainName).root;
  tree.delete(`${libraryRoot}/assets/.gitkeep`);
  tree.delete(`${libraryRoot}/environments/environment.prod.ts`);
  tree.delete(`${libraryRoot}/environments/environment.ts`);
}

async function createApplication(tree: Tree, schema: any) {
  const libraryGenerator = wrapAngularDevkitSchematic('@nrwl/angular', 'app');
  await libraryGenerator(tree, {
    ...schema,
    ...appAndLibSetting,
    name: schema.device,
    directory: schema.directory,
    routing: false,
    tags: `scope:shared,type:app`,
  });
}

async function createDependanceLibrary(tree: Tree, schema: any, domainName: string) {
  await shellGenerator(tree, {
    directory: `${schema.directory}/${schema.device}`,
  });
  await sharedGenerator(tree, {
    directory: `${schema.directory}/${schema.device}`,
  });
}

import { Tree, formatFiles, readProjectConfiguration, generateFiles, joinPathFragments, readJson } from '@nrwl/devkit';
import { wrapAngularDevkitSchematic } from '@nrwl/devkit/ngcli-adapter';
import { classify, dasherize } from '@nrwl/workspace/src/utils/strings';
import { appAndLibSetting } from '../../new-tool/common-setting.tool';

export default async function (tree: Tree, schema: any) {
  await createShellLibrary(tree, schema, 'shell');
  const projectName = dasherize(`${schema.directory}-shell`).replace(/\//g, '-');
  addShellModule(tree, schema, projectName, 'shell');
  await formatFiles(tree);
  return () => {};
}

export async function createShellLibrary(tree: Tree, schema: any, type) {
  const libraryGenerator = wrapAngularDevkitSchematic('@nrwl/angular', 'lib');
  await libraryGenerator(tree, {
    name: `${schema.directory}/${type}`,
    tags: `scope:${type},scope:shared,type:${type}`,
    ...appAndLibSetting,
  });
}

export function addShellModule(tree: Tree, schema: any, projectName, type) {
  const libraryRoot = readProjectConfiguration(tree, projectName).root;
  generateFiles(tree, joinPathFragments(__dirname, `./files`), `${libraryRoot}/src/lib/${dasherize(`${schema.directory}-${type}`)}`, {
    ...schema,
    sharedModule: classify(`${schema.directory}-shared-module`.replace(/\//g, '-')),
    shellModule: classify(`${schema.directory}-shell-module`.replace(/\//g, '-')),
    shellName: dasherize(`${schema.directory}-shell`.replace(/\//g, '-')),
    sharedName: dasherize(`${schema.directory}-shared`.replace(/\//g, '-')),
  });
}

[meriturva]

Our way (without using nx) is to analyze all projects from angular.json file and create a dependency tree, then run multiple build commands based on tree (parallelly) to speed up the process.

See how we read angular.json file: https://medium.com/we-code/angular-inspect-workspace-programmatically-3cdfeb9b93d4

[K3CharlieStuart]

Our way (without using nx) is to analyze all projects from angular.json file and create a dependency tree, then run multiple build commands based on tree (parallelly) to speed up the process.

See how we read angular.json file: https://medium.com/we-code/angular-inspect-workspace-programmatically-3cdfeb9b93d4

Hi,

We are facing the same problem as you and you're solution sounds perfect! Would you be willing to share how you create the dependency tree and run build commands based on it?

Thanks in advance

[boeckMt]

Our way (without using nx) is to analyze all projects from angular.json file and create a dependency tree, then run multiple build commands based on tree (parallelly) to speed up the process. See how we read angular.json file: https://medium.com/we-code/angular-inspect-workspace-programmatically-3cdfeb9b93d4

Hi,

We are facing the same problem as you and you're solution sounds perfect! Would you be willing to share how you create the dependancy tree and run build commands based on it?

Thanks in advance

@meriturva I would also be interested in how you did this because it sounds very similar to what we did :)

@K3CharlieStuart current solutions I know now are:

• either take: https://github.com/klemenoslaj/ong-cli // download and build the repo, than you can link it as npm dependency
• or check how a dependancy tree is created and build commands are run // but this is not working in parallelly
  • https://github.com/dlr-eoc/ukis-frontend-libraries/tree/main/scripts/library
  • https://github.com/dlr-eoc/ukis-frontend-libraries/blob/0e5949ed4040a605de40a899be18e04c7d84b73d/scripts/library/index.ts#L100
• use nx ...

[K3CharlieStuart]

@boeckMt thanks for the suggestions!

I'm just looking into your second suggestion but I cant seem to figure out how we would utilise this in our project. Would it be a case of copying the script folder into our application and somehow executing the quoted function?

[meriturva]

Here is just a code snippet of how we build our dependencies tree, it is really simple, the complex one is to run all scripts parallel and that script it is on our private repo right now and I cannot share it for now. May be I could arrange an article (or a few) about that. Would you be interested?

export function buildPeerDependenciesTree(filterArchitect: string): Record<string, PackageNode> {

  const angularJson = JSON.parse(fs.readFileSync('angular.json', 'utf8'));

  const nodes: Record<string, PackageNode> = {}; 

  Object.values(angularJson.projects).forEach((value) => {
    if ((value as any).projectType === 'library' && (value as any).architect[filterArchitect]) {
      const packageContent = JSON.parse(fs.readFileSync(path.join((value as any).root, 'package.json'), 'utf8'));
      const moduleName = packageContent.name;
      const peerDependencies = new Set<string>();

      // Look for peerDependencies also in package.json in subFolders
      const allPackageJson = glob.sync(`${(value as any).root}/**/package.json`);
      allPackageJson.forEach(filePath => {
        const fileContent = JSON.parse(fs.readFileSync(filePath, 'utf8'));
        if (fileContent && fileContent.peerDependencies) {
          Object.keys(fileContent?.peerDependencies).forEach(peerDependency => peerDependencies.add(peerDependency));
        }
      });

      if (!nodes[moduleName]) {
        nodes[moduleName] = new PackageNode(moduleName, (value as any).root, (value as any).sourceRoot);
      }

      nodes[moduleName].package = packageContent;

      peerDependencies.forEach((nodeDependency) => {
        // Check if dependency is present locally (to be compile)
        if (angularJson.projects[nodeDependency]?.architect[filterArchitect]) {
          if (!nodes[nodeDependency]) {
            nodes[nodeDependency] = new PackageNode(nodeDependency, (angularJson.projects[nodeDependency] as any).root, (angularJson.projects[nodeDependency] as any).sourceRoot);
          }

          nodes[moduleName].originalDependencies.push(nodeDependency);
          nodes[moduleName].dependencies.push(nodeDependency);
        }
      });
    }
  });

  return nodes;
}

[klemenoslaj]

@boeckMt thanks for the suggestions!

I'm just looking into your second suggestion but I cant seem to figure out how we would utilise this in our project. Would it be a case of copying the script folder into our application and somehow executing the quoted function?

Sorry for repeating the same thing, but I feel we are going back to basics here - reading angular.json etc.

Have a look at the https://github.com/angular/angular-cli/issues/11002#issuecomment-1075285169 where I made an entire repo public - it even supports parallel builds no matter the complexity of dependency tree.

Running example in the following comment: https://github.com/angular/angular-cli/issues/11002#issuecomment-846391752

Due to lack of interest I did not yet made this an npm package tho, but feel free to consume & contribute.