Support running an app from a subdirectory

[jmorrell]

trafficstars

There is currently no supported way to run a node app from a directory within your git repo. You can work around it most of the time by using git subtree push but that wouldn't work if you have shared code outside of your directory.

A super common situation is to have the following repo structure:

├── shared-code
├── server
└── frontend-app

Being able to direct the buildpack to look in /server instead of / would be helpful for cases like this.

[Arrow7000]

Exactly what I need. Thanks @jmorrell!

[jmike]

@jmorrell here's a workaround. It ain't perfect but works...

1. Make sure to add package.json under server and frontend-app. List dependencies (as you normally do) and specify a start script for the server.

2. Create a root package.json file as follows:

{
  "name": "foobar-app",
  "version": "0.1.0",
  "scripts": {
    "postinstall": "npm install --prefix server && npm install --prefix frontend-app"
  }
}

Notice the --prefix argument which tells npm which subfolder to work on.

3. Add the following to your procfile:

web: npm start --prefix server

Hope that helps,

[jmike]

OK, I just realized you are working for Heroku, so this is probably not a real question, but rather a feature request. Sorry for the misunderstanding.

In any case, I will leave my answer above in case someone finds it useful.

[Arrow7000]

@jmike thanks for weighing in. Is this something you've tried yourself and has worked? Because in the absence of an official solution I'll give that a try.

[jmike]

@Arrow7000 yeap, works like a charm. The only problem is caching - see issue #387 even though it's somewhat unrelated to the application structure. See, I am using local files as npm dependencies.

In any case if you face a similar issue you can always disable caching according to heroku instructions https://devcenter.heroku.com/articles/nodejs-support#cache-behavior.

[Arrow7000]

Thanks @jmike that worked like a treat! It's not the cleanest solution, because it requires an extra package.json with important options duplicated - eg engines.node - but until we get an official solution this seems like the best and simplest way to get it done!

[gablg1]

We had been using git subtree push for a while but started facing the exact issue described above, so we implemented the following custom buildpack https://github.com/Pagedraw/heroku-buildpack-select-subdir

which allows us to deploy multiple apps from the same Heroku repo. Then we just make each of the frontend-app and server require the shared-code as an npm local dependency.

One caveat is that we have to explicitly add the node_modules folder installed to the NODE_PATH so npm knows where to look for requires within shared-code.

It works for us. Let me know if it also works for you!

[Jan0707]

@jmike I tried your approach and it sadly does not work for me. Here is my package.json

{
  "name": "App",
  "engines": {
    "node": "8.1.x"
  },
  "scripts": {
    "postinstall": "npm install --prefix app/Resources && app/Resources/node_modules/.bin/gulp --gulpfile app/Resources/gulpfile.js"
  }
}

It fails though when trying to run gulp:

remote: -----> Building dependencies
remote:        Installing node modules (package.json)
remote:
remote:        > App@ postinstall /tmp/build_98b4546541f7050670cac9ef04e8ace1
remote:        > npm install --prefix app/Resources && app/Resources/node_modules/.bin/gulp --gulpfile app/Resources/gulpfile.js
remote:
remote:        added 14 packages in 2.164s
remote:        sh: 1: app/Resources/node_modules/.bin/gulp: not found
remote:        npm ERR! file sh
remote:        npm ERR! code ELIFECYCLE
remote:        npm ERR! errno ENOENT
remote:        npm ERR! syscall spawn
remote:        npm ERR! App@ postinstall: `npm install --prefix app/Resources && app/Resources/node_modules/.bin/gulp --gulpfile app/Resources/gulpfile.js`
remote:        npm ERR! spawn ENOENT
remote:        npm ERR!
remote:        npm ERR! Failed at the App@ postinstall script.
remote:        npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
remote:
remote:        npm ERR! A complete log of this run can be found in:
remote:        npm ERR!     /app/.npm/_logs/2017-07-14T10_49_30_508Z-debug.log

Any idea on what I am doing wrong here would be highly appreciated :)

[atomkirk]

It appears that if you run the buildpack from within a subdirectory, the commands it's suppose to install are not available after release. My buildpack runs the heroku/php buildpack in a sub directory, but when I heroku run bash, which php returns nothing. Same goes if I cd into the directory.

[jmorrell]

@atomkirk The buildpacks install their dependencies globally, and I'm not sure what you mean by running the "php buildpack in a sub directory". Could you open a support ticket at https://help.heroku.com/ so we can sort out what's going wrong?

[atomkirk]

So it looks like they don't install them globally. When I deploy a root directory php app with heroku/php buildpack, then heroku run bash, and which php, its found in /app/.heroku/php/bin/php and echo $PATH is /app/.heroku/php/bin:/app/.heroku/php/sbin:/app/.heroku/php/bin:/app/.heroku/php/sbin:/app/.heroku/php/bin:/app/.heroku/php/sbin:/app/.heroku/php/bin:/usr/local/bin:/usr/bin:/bin:/app/vendor/bin

So this is actually an easy fix, just need to fix PATH so it points to /app/subdir/… instead of /app/…

Would be nice if the buildpack system provided an option so we didn't have to make our own buildpacks to work around it

[jmorrell]

@atomkirk Sounds like you're trying to do something that's not supported cc @dzuelke

[jmike]

@Jan0707 have you tried running gulp on postinstall from within the internal package.json file, i.e. app/Resources/package.json in your case?

Another idea might be to install gulp with root package.json and update your configuration as such:

{
  "name": "App",
  "engines": {
    "node": "8.1.x"
  },
  "dependencies": {
    "gulp": "^3.9.1"
  },
  "scripts": {
    "postinstall": "npm install --prefix app/Resources && gulp --gulpfile app/Resources/gulpfile.js"
  }
}

Please note that you don't really need to provide the qualified file path for gulp - npm knows what to do.

[dzuelke]

Why a subdirectory, @atomkirk? Or, at least, why does the PHP runtime have to be in a subdirectory, and not just your application? A lot of moving parts are set up to look at $HOME/.heroku/… for buildpacks, and that includes how PHP is built and where it looks for its config files, where binaries look for other .sos, and so forth. It's not something that's easily changed, and so far I don't understand the use case for it.

[atomkirk]

Well, thanks for the response, but @jmorrell is right, I'm trying to do something unsupported. I kind of took us off topic. Sounds like if you want to support the original issue, the buildpack will probably have to run in the subdirectory and then PATH point at where it put the .heroku folder (or if the buildpack puts the .heroku folder in the repo root, then change nothing)

[gplusdotgr]

This feature is what I was looking for. I have a client and a server which have two independent package files, and I want them to be sibling directories and not nested. Therefore I need a way to instruct heroku to start from a specific package file which is not located at the root of my repository.

[ivanovivelin]

Hi, I seem to face the same issue when running under sub directory =>

"postinstall": "npm install --prefix server && npm install --prefix ./server/client",

The entire project is inside server folder , including the client ... any help

[jmorrell]

@IIvanov8888 could you open a support ticket at help.heroku.com?

[wmaciejak]

@gablg1 for me your solution doesn't work.

After valid deploy I see in console

app[client.1]: bash: npm: command not found. 

The same situation with node. Even If I change $PATH variable, the issue still occurs.

[ClayShentrup]

I have a similar issue. I want to run an Ember Fastboot app without having to make a separate repo for my API backend (which is Rails in my case). Ideally I'd be able to specify multiple web processes for a single app. Since I can't do that, I have to make two separate apps for the frontend and backend. But I don't want to create multiple repos for this. I just want to be able to tell one app to use one subdirectory and tell the other app to use the other subdirectory from the git project root.

[WeishiZ]

As mentioned in the ticket, one can use git subtree push --prefix {app_subfolder} heroku master to push a subfolder only. It would only work if the subfolder is self-contained

[Bnaya]

@WeishiZeng can you please elaborate? :)

[WeishiZ]

@Bnaya The git command is used to push a subfolder to remote. If the subfolder happens to be a self-contained app recognized by heroku, then it'll be deployed.

[javidjamae]

One note to add to the conversation is that the git subtree push solutions only work when you're directly pushing your app to Heroku, but it doesn't solve the problem when you're setting up a Heroku CI Pipeline. In that case, the filesystem is readonly and (AFAIK) there is no way to control what subdirectory of the code is used for the test execution, deployment, etc.

I encountered this issue when trying to use a monorepo with a node.js app as I describe here: https://stackoverflow.com/questions/51449750/how-do-i-get-heroku-ci-to-run-tests-when-using-a-monorepo

[kris-campos]

has there not been an official solution for this yet? is there a ticket somewhere that we can follow?

[jmorrell]

@kris-campos I haven't had a chance to loop back on this yet, so there is no official solution. In the meantime I would look at Yarn workspaces: https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/ which I believe npm is also in the process of implementing.

This will be the first thing I explore as a potential solution. If you try them please report back with how it worked for you.

[danielmahon]

@jmorrell I just starting switching to yarn workspaces & lerna. So far using https://github.com/timanovsky/subdir-heroku-buildpack I can get the package to run properly but I'm unable to access a shared module because it bascially wipes everything outside the sub directory so shared-code doesnt exist and is a local module, not published. Did you figure out a good way to solve this?

├── shared-code
├── server
└── frontend-app

[danielmahon]

@jmorrell

UPDATE: Well I tried to avoid it but looks like I need to use https://www.npmjs.com/package/yalc instead of yarn link. Since the symlinks don't seem to always work properly inside Heroku. I could drop yarn workspaces completely for yalc and it would definitely simplify/remove most of these steps but the point of this was to try to use yarn workspaces so I'll leave it for now and just use yalc inside Heroku.

This is my current working deployment setup using a local lerna / yarn workspaces monorepo:

root
├── buildpack-run.sh
├── lerna.json
├── package.json
├── packages
│   ├── app
│   │   ├── package.json
│   ├── graphql
│   │   ├── package.json
│   ├── native
│   │   └── package.json
│   ├── shared
│   │   ├── package.json
│   └── site
│       ├── package.json
└── yarn.lock

First, I'm using these 3 buildpacks:

https://github.com/weibeld/heroku-buildpack-run
https://github.com/timanovsky/subdir-heroku-buildpack
https://github.com/danielmahon/create-react-app-buildpack

I use heroku-buildpack-run to run this script to copy the "sibling" shared package into a lib folder within the main package to be deployed, as well as the root level yarn.lock which while it contains ALL the package dependencies, yarn install should still only match the dependencies from package.json.

#!/bin/bash

echo "       Copying shared modules"

mkdir -p packages/app/lib/@myscope
cp -R packages/shared packages/app/lib/@myscope
cp yarn.lock packages/app
# repeat for other simultaneous deployments

I use subdir-heroku-buildpack which just pulls out and replaces your root build folder with one specified at PROJECT_PATH. This is why I needed to copy the shared package from the first step into this one.

I use a forked create-react-app-buildpack because I needed to update it's own dependancy of heroku-buildpack-nodejs with a forked version of my own https://github.com/danielmahon/heroku-buildpack-nodejs that simply adds the --ignore-optional flag to the default yarn install command.

Maybe we can set --ignore-optional with an env variable so the buildpacks don't need forked?

In my package.json for the deployed module, I have the following npm scripts to yalc add the shared package that's now in the lib folder.

  "scripts": {
    "heroku-postbuild": "npm-run-all -s yalc:publish yalc:add ",
    "yalc:publish": "cd lib/@myscope/shared && yalc publish",
    "yalc:add": "yalc add @myscope/shared"
    ...
  },

I also needed to set the shared package as an optional dependency since it is unpublished.

  "optionalDependancies": {
    "@myscope/shared": "*"
  },

As of now, running lerna version --exact --force-publish in the root of the monorepo, properly updates all the package versions, and performs a git push which triggers a new build on Heroku for two apps which share the same local sibling package.

This works but obviously requires much more setup than I would like, let me know if anyone sees anything that can be simplified (I suppose a dedicated buildpack would help), until there is a better option.

[jmorrell]

@danielmahon I'm unclear on why you need the subdir buildpack. With a package.json listing the workspaces at the root you could push the whole repo up and get all of your dependencies installed.

There are a couple of things that would be missing:

• support for build commands like heroku-postbuild, but you could add the scripts at the root
• no automatically picking up the npm start command in the subdirectory, so this command needs to live at the package root

If you open a support ticket, I can help you simplify this setup

[jmorrell]

re: using yarn workspaces more generally

I got a support ticket recently from a user who had multiple services within the same repo, using yarn workspaces. They used an env var in each app ($APP_DIR) to direct that dyno to start that service.

One drawback to this is that it installs all of your dependencies for all of your applications, which was ballooning their app size over Heroku's limits. Here was the workaround we found.

tl;dr - delete all of the directories not needed by the app in $APP_DIR

---

Our temporary workaround was to delete folders we didn’t need in a heroku build script in our package.json. This works OK, but it means we have to maintain a list of folders to delete in each service which is kind of a pain.

To resolve this, you can get a map of how each workspace depends on the others using yarn workspaces info. In this example workspace-b depends on workspace-a, but workspace-a and workspace-c are independent:

❯ yarn workspaces info       
yarn workspaces v1.9.4
{
  "workspace-a": {
    "location": "workspace-a",
    "workspaceDependencies": [],
    "mismatchedWorkspaceDependencies": []
  },
  "workspace-b": {
    "location": "workspace-b",
    "workspaceDependencies": [
      "workspace-a"
    ],
    "mismatchedWorkspaceDependencies": []
  },
  "workspace-c": {
    "location": "workspace-c",
    "workspaceDependencies": [],
    "mismatchedWorkspaceDependencies": []
  }
}
✨  Done in 0.04s.

You could create a heroku-prebuild Node script that uses this output to delete any directories not required by $APP_DIR. Here is a quick sketch to what that might look like:

"scripts": {
  ...
  "heroku-prebuild": "node remove-workspaces.js"
}

remove-workspaces.js

const { exec } = require('child_process');
const app = process.env['APP_DIR']

exec('yarn workspaces info --json', (err, stdout, stderr) => {
  const output = JSON.parse(stdout);
  const info = JSON.parse(output.data);

  const dependencies = gatherDependencies(info, app);
  const unneeded = Object.keys(info).filter(i => !dependencies.includes(i));

  unneeded.forEach(i => exec(`rm -rf ${i}`));
});

// Gather all of the workspaces that `workspace` depends on
function gatherDependencies(info, workspace) {
  let deps = [workspace];
  let ws = [workspace];
  while (ws.length) {
    info[ws[0]].workspaceDependencies.forEach(w => {
      ws.push(w);
      deps.push(w);
    });
    ws.shift()
  }
  return deps;
}

This should leave only the workspaces needed by $APP_DIR, and yarn will only install the dependencies needed by that directory.