Deploy functions with node_modules included

[dominicbartl]

Tools: 5.1.1
Platform: Ubuntu

Steps to reproduce

I was upgrading the firebase-tools from 3.17.7 to the latest version 5.1.1 since our CI was unable to deploy the functions due to the following error:

HTTP Error: 410, This version of the Firebase CLI is no longer able to deploy to Firebase. Please upgrade to a newer version (>= 4.1.0). If you have further questions, please reach out to Firebase support.

After upgrading to the latest firebase-tools and from node 6 to node 8, functions still fail to deploy. It appears to me that now npm install is executed before finally deploying the new functions. This is problematic since one of your dependency is a private git repository and we use yarn instead of npm

We would like to pre-package the node_modules directory and skip the npm install. I checked the package after running firebase deploy and it includes the node_modules directory. I guess since we have some ignore rules in our firebase.json

Is it possible to upload the fully built package to firebase without running npm install? I checked the documentation about handling dependencies and it seems that it's possible using glcoud, though without further description on how to achieve this.

[mbleigh]

The node_modules directory is always excluded from Cloud Functions deploys (this was always intended to be the case).

One way to work around this is to either put a tarball of the dependency in the functions folder or clone the dependency into a subfolder and use a file: dependency (see NPM docs).

[dominicbartl]

@mbleigh thanks for the reply.

I don't think this was always the case. The handling dependencies documentation from May 17th says:

You can either prepackage fully materialized dependencies within your function package or declare them in the package.json file, and Cloud Functions will download them for you when you deploy.

and I assume with fully materialized dependencies they meant the node_modules directory. Furthermore running npm install would always have failed because of our private git repository.

Is it possible to switch and deploy with the gcloud CLI and pre-package the node_modules directory?

[p2lvoizin]

I had the same issue on debian. Because like you I upgraded my firebase-tools to 6.0.0. I fixed it by unistaling firebase-tools and re-install the [email protected].

[dominicbartl]

@p2lvoizin You mean npm install isn't executed server-side when you deploy your functions?

[dominicbartl]

@mbleigh any updates on this? Or can someone from the functions team help out?

I'm currently working on a workaround. The first thing I tried is copying all files into a new deployment directory, installing the dependencies and overriding the dependencies in the package.json file.

cp -r ${base}/dist/app ${dirDeployment}/dist
cp -r ${base}/dist/functions ${dirDeployment}/dist
cp -r ${base}/config ${dirDeployment}
cp ${base}/index.js ${dirDeployment}
cp ${base}/package.json ${dirDeployment}
cp ${base}/yarn.lock ${dirDeployment}
cp ${base}/firebase.json ${dirDeployment}
cp ${base}/firestore.indexes.json ${dirDeployment}

cd ${dirDeployment}
yarn install --prod
cat ../package.json | jq 'del(.dependencies,.devDependencies)' > ${dirDeployment}/package.json
npm i -g firebase-tools@${toolsVersion}
firebase deploy --project ${project} --token="${FIREBASE_TOKEN}";

The zip package including the node_modules gets generated and uploaded successfully. Though after uploading the deployment stops with the following error:

functions: updating Node.js 8 function api(us-central1)...
functions[api(us-central1)]: Deployment error.
Function load error: Code in file index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'firebase-functions'

Running the index.js locally and requiring local node_modules works fine in the generated deployment directory.

Second workaround I tried is doing the same as above but instead of removing the dependencies from the package.json file I overrode them with local paths.

so this:
...
"express": "^4.15.3",
"express-json-views": "^0.1.0",
"express-session": "^1.15.3",
"express-winston": "^2.4.0",
"ffc-node": "^12.7.0",
...

becomes this

...
"express": "file:./node_modules/express",
"express-json-views": "file:./node_modules/express-json-views",
"express-session": "file:./node_modules/express-session",
"express-winston": "file:./node_modules/express-winston",
"ffc-node": "file:./node_modules/ffc-node",
...

This doesn't work as well and weirdly multiple functions randomly throw different errors.

Either this:

Build failed: USER ERROR:
`npm_install` had stderr output:
npm ERR! code ENOLOCAL
npm ERR! Could not install from "node_modules/@google-cloud/logging" as it does not contain a package.json file.

Or this:

Build failed: {"message": "`yarn_install` had stderr output:\nerror Package \"\" refers to a non-existing file '\"/workspace/node_modules/@google-cloud/logging\"'.\n\nerror: `yarn_install` returned code: 1", "code": 1, "type": "USER_ERROR"}

Any guidance on how to resolve this issue would be much appreciated. We're unable to update our production app for close to a week now.

[mbleigh]

A few things:

1. You should really only need to ship your private code separately. All public repos should be in package.json as normal.
2. You don't want to put the downloaded private package in a folder called node_modules. Instead you would do something like:

functions/
  index.js
  package.json
  my-private-module/
    package.json
    ...

Then, in package.json you would do:

{
  "dependencies": {
    "express": "^4.15.3",
    "express-json-views": "^0.1.0",
    "express-session": "^1.15.3",
    "express-winston": "^2.4.0",
    "ffc-node": "^12.7.0",
    "my-private-module": "file:./my-private-module"
  }
}

You could automate grabbing the latest version of your module with a script that just blows away the directory and re-clones from git.

For what it's worth, we're discussing how we might support this better, but that's going to take some time.

[bkendall]

Soooooo here's a fun story:

I went diving through the code a bit to see why it would be ignoring node_modules and found a neat line of code. Basically, if in your firebase.json you set functions.ignore to an empty array, the CLI will not ignore node_modules when it packages the function(s). This does work - I've confirmed it myself - but it does effect all the functions that are deployed and cannot target one specific function.

[mbleigh]

To do this, you would add to firebase.json:

{
  "functions": {"ignore": []}
}

[dominicbartl]

@bkendall I found that too when I was debugging the ignore rules for #751 That's why we had "ignore": [ "rule-to-override-defaults-and-package-node-modules" ] in our firebase.json and I assume that this is the reason the deployment used to work. The only difference is that before upgrading the deployment didn't throw an error when trying to upgrade the functions. This could either be that npm install wasn't executed if node_modules were already included or the error due to the private package was ignored.

@mbleigh We would have preferred to use yarn and package the dependencies on our CI system after all tests have passed.

I managed to do it by installing the dependencies, moving them into a different sub-directory lib and updating the package.json dependencies with "some-dep": "file:./lib/some-dep". I must say that this feels like a hacky solution but is working for now. I might update it our deployment script as you described later.

Do you see any advantages by installing them via the functions instead of pre-packaging them? We were doing it this way to deploy a full package with which all unit and integration tests passed. Are there any reasons why this isn't/wasn't supported? Would really like to know what you guys think. Thanks

[p2lvoizin]

@p2lvoizin You mean npm install isn't executed server-side when you deploy your functions?

yes

[laurenzlong]

@Bartinger We do that primarily to speed up deployments.

As well, Bryan's suggestion in https://github.com/firebase/firebase-tools/issues/968#issuecomment-434766838 should work for you as well.

[shukob]

I have git dependency on my package.json and it seems not being deployed to firebase when doing firebase deploy.

[bkendall]

@shukob if it ends up in your node_modules folder, you can use the information in the comment above (and the one below it) to include your node_modules folder.

[thechenky]

Hi everyone - catching up on this thread. Unfortunately the functions team won't be able to get around to this in the near term so please refer to the workaround provided by @bkendall and @mbleigh. Also - PRs are very welcome :)

[dominicbartl]

Thanks for your reply @thechenky. I can see that this isn't a priority now. One reason I opened the issue is we exclusively use yarn for all projects at our company and we would like to deploy the dependencies in our yarn.lock files.

Since functions uses npm, it potentially installs different versions which can lead to unexpected results. Are there any plans on detecting which package manager is used and using this to install dependencies?

And thanks for all your support here 😊

[thechenky]

@Bartinger great question, I actually just double checked with Google Cloud Functions and saw there's already a bug being tracked for this work, specifically to supply a yarn.lock file and have yarn be used to install the dependencies during function deployment (Internal bug reference: 35994441). There seem to be some blockers around this since it would impact the caching mechanism we currently use to make dependency fetching fast. However, I will add this thread to the bug to make sure we're capturing all similar requests from our developers :) and check in on progress again.

[dominicbartl]

@thechenky That's great to hear, thanks for your effort. Looking forward when yarn is supported.

[adevine]

I've had a similar issue, and have seen lots of other folks with similar issue, where folks are including relative dependencies (e.g. "myPeerPackage": "file:../myPeerDirectory") in their package.json, but then that code isn't included with firebase deploy. Rather than move myPeerDirectory as a subdirectory of functions (which can cause a host of other problems and feels pretty "unnatural"), I took a hint from @mbleigh 's suggestion of just including the tarball.

In my functions package.json I include the following preinstall script:

"preinstall": "if [ -d ../myPeerPackage ]; then npm pack ../myPeerPackage; fi"

This insures the tarball is created when I install locally (and should be there before running firebase deploy), but the if check means it does NOT run in the cloud.

The in my dependencies:

"myPeerPackage": "file:./myPeerPackage-1.0.0.tgz"

[danieldanielecki]

1. Let say you made a change in a dependency called protobufjs in your node_modules.
2. Copy this package protobufjs from your node_modules to functions folder.
3. Add to your dependencies the following: "protobufjs": "file:./protobufjs" (assuming you placed the protobufjs folder in the root of functions folder).
4. Run: firebase deploy.

[igl]

2020 now. Firebase functions are still snowflakes

[manishPh]

@dominicbartl Were you able to find solution to your problem? @mbleigh, The issue we are facing is similar and we want to create a fat .zip that includes everything and no npm install during function deployment, for some security reasons. Does that mean I have to create myPackage.tgz for every dependency in my package.json and then replace the dependency as "myPackage": "file:./myPackage-1.0.0.tgz"? What happens to the dependencies of myPackage? Will they still be pulled from npm?

[dominicbartl]

Hi @manishPh, I'm no longer packaging my node_modules but I think the way I solved it was I copied the package.json, code and dependencies into a deployment directory. Then i overrode the dependencies in the package.json file with the paths to the local directories. I believe you can just reference the directory, you don't have to package each dependency on its own

[Imprasna]

[email protected] preinstall /usr/local/lib/node_modules/firebase-tools/node_modules/publish-please node lib/pre-install.js

Please tell me what do I have to do about this? Do I have to install pre-install.js what's that about? I got this when I installed firebase globally

[mike-hogan]

Folks, I did what some others mentioned in this thread and the discussion was informative, so hoping to add a bit back.

Basically I re-wrote the package.json files to have to have file: dependencies.

The oddity in my case is that its a rushjs monorepo.

But in case folks can learn from what I did, here is the gist: https://gist.github.com/mike-hogan/336de1a55ad1a776398ea85c2691dfd2

[loginov-rocks]

Spent few hours resolving this issue which is very frustrating and includes the following concerns: npm local and CI/CD machine cache, binding to a particular v1.2.3 tag version, remote CI/CD workflows, package-lock.json binding to a particular integrity check - so many issues, Firebase team, please help!

Finally, got to the solution: shared package installed as it should be - via symlink meaning "file:../../shared" in the functions/package.json giving the proper local development experience.

However, for the CI on GitHub, I added a few commands (npm ci and npm run build) to build the shared package so it can be picked by the functions CI/CD process as a correct local dependency: https://github.com/loginov-rocks/Portfolio/blob/main/.github/workflows/firebase-ci.yml#L37

For the CD (deploy) process, independently whether it's happening on a local or CI/CD machine I added the following configuration to "predeploy" script https://github.com/loginov-rocks/Portfolio/blob/main/firebase/firebase.json#L8

{
  "functions": {
    "predeploy": [
      "npm --prefix \"$RESOURCE_DIR/../../shared\" ci",
      "npm --prefix \"$RESOURCE_DIR/../../shared\" run build",
      "npm pack \"$RESOURCE_DIR/../../shared\"",
      "mv loginov-rocks-portfolio-shared-*.tgz \"$RESOURCE_DIR/loginov-rocks-portfolio-shared.tgz\"",
      "npm --prefix \"$RESOURCE_DIR\" install \"$RESOURCE_DIR/loginov-rocks-portfolio-shared.tgz\"",
      "npm --prefix \"$RESOURCE_DIR\" run lint",
      "npm --prefix \"$RESOURCE_DIR\" run typecheck",
      "npm --prefix \"$RESOURCE_DIR\" test",
      "npm --prefix \"$RESOURCE_DIR\" run build"
    ],
    "source": "functions"
  }
}

So what it does:

1. Install the shared package dependencies
2. Build the shared package
3. Pack the shared package into the tgz with the v1.2.3 in the file name
4. Move the shared package to the functions directory and removes the version tag
5. Install tgz dependency
6. Do all the checks, but it's relevant to my project only

This appears as the correct workaround since it's using the actual shared package version and doesn't require additional actions to deploy. The only downside on the local development machine is that it updates functions/package.json and functions/package-lock.json - but this should be controlled by the developer, tho it's not a big problem on CI/CD machine.

GitHub Actions for those who interested: https://github.com/loginov-rocks/Portfolio/tree/main/.github/workflows

[idudinov]

Hi there! just sharing another version of a custom solution of the issue.

Script for adding/restoring packages: https://gist.github.com/idudinov/ca12ba99f615a23776bed03aa662c29d

The idea here is still to use firebase.json hooks but with the script from the gist above which doesn't pack the dependency but just temporary installs the same version the parent has; then, on postdeploy, just restore to the initial version (or remove). Should work correct and with no artefacts both on local machine and CD.

Usage in firebase.json:

{
  "functions": {
    "predeploy": [
      "cd \"$RESOURCE_DIR\" && yarn ts-node utils/install-ext-deps.ts",
      "cd \"$RESOURCE_DIR\" && yarn build"
    ],
    "postdeploy": [
      "cd \"$RESOURCE_DIR\" && yarn ts-node utils/install-ext-deps.ts --restore"
    ]
}

[LuisAlejandro]

To do this, you would add to firebase.json:

{
  "functions": {"ignore": []}
}

For anyone trying this in 2022: uploading your node_modules almost always won't work because the maximum allowed upload size is 10MB. When your upload is larger than that, this error pops out:

⚠  functions: Upload Error: HTTP Error: 400, <?xml version='1.0' encoding='UTF-8'?><Error><Code>EntityTooLarge</Code><Message>Your proposed upload is larger than the maximum object size specified in your Policy Document.</Message><Details>Content-length exceeds upper bound on range</Details></Error>

[knownasilya]

I'm trying to do something similar for Ember Fastboot (SSR) which requires the node_modules dir for SSR stuff. I'm running into this error:

Error: Could not read source directory. Remove links and shortcuts and try again.

I tried yarn install --no-bin-links but still get that error.

---

This seems to have helped: "cd $RESOURCE_DIR/dist/node_modules && find . -type d -name .bin -exec rm -rf {} +"