turbo
turbo copied to clipboard
Examples: Add Firebase functions
Discussed in https://github.com/vercel/turborepo/discussions/640
Originally posted by snorreks January 29, 2022 Is anyone using firebase functions with turborepo? If so, how are you importing packages?
I was previously hacking lerna to make firebase functions work within a monorepo setup. With lerna now not being maintained, I am going to investigate other tools. Nx seems too much for me compared to turborepo. Would be looking to get something up and running but need to dip my feet into turborepo first.
Here is my first pass on this and seems to work ok but I'm rather green to the turbo scene so I could be doing something completely incorrect.
From my initial testing after the first npm run deploy
everything will get cached and subsequent deploys will only deploy the entity that changed.
https://github.com/Hacksore/turborepo-firebase-example
Here is my first pass on this and seems to work ok but I'm rather green to the turbo scene so I could be doing something completely incorrect.
From my initial testing after the first
npm run deploy
everything will get cached and subsequent deploys will only deploy the entity that changed.https://github.com/Hacksore/turborepo-firebase-example
Thanks so much for sharing this. I'm struggling to get rollup to compile correctly for deployment :(! Have you gotten it to compile / deploy correctly?
@ecam900 I just loaded up my demo and seems to compile for me fine. Are you having issues with turbo or just rollup?
@Hacksore ➜ /workspaces/turborepo-firebase-example (main ✗) $ npm run build
> [email protected] build
> turbo run build
• Packages in scope: api, config, shared, tsconfig, web
• Running build in 5 packages
web:build: cache hit, replaying output 24e63bf6f06b6eeb
web:build:
web:build: > [email protected] build
web:build: > tsc && vite build
web:build:
web:build: vite v2.9.7 building for production...
web:build: transforming...
web:build: ✓ 34 modules transformed.
web:build: rendering chunks...
web:build: dist/assets/favicon.17e50649.svg 1.49 KiB
web:build: dist/index.html 0.46 KiB
web:build: dist/assets/index.62f502b0.css 0.75 KiB / gzip: 0.48 KiB
web:build: dist/assets/index.63fb99db.js 139.96 KiB / gzip: 44.89 KiB
api:build: cache hit, replaying output f984330b6a8446d0
api:build:
api:build: > build
api:build: > rimraf ./dist/**/* && rollup -c rollup.config.js
api:build:
api:build:
api:build: src/index.ts → dist/index.js...
api:build: created dist/index.js in 3.1s
Tasks: 2 successful, 2 total
Cached: 2 cached, 2 total
Time: 28ms >>> FULL TURBO
@Hacksore - I got mine to compile as well. However, due to how my code is structured, the bundled result is not suitable for firebase deployment.
My src
directory is as follows: (deleted some folders for brevity)
├── config
│ └── methodConfig.ts
├── payments
│ ├── config.ts
│ ├── getPaymentToken.ts
│ ├── types.ts
│ └── utils.ts
├── functions
│ ├── algolia
│ │ ├── base.ts
│ │ ├── items.ts
│ │ └── warranties.ts
│ ├── backups
│ │ └── backups.ts
│ ├── index.ts
│ └── warranties
│ ├── base.ts
│ └── processApproval.ts
I then export from functions/index.ts
:
import admin from "firebase-admin";
admin.initializeApp();
export * as algolia from "./algolia/base";
export { scheduledBackup } from "./backups/backups";
export * as warranties from "./warranties/base";
Then on the bundled result, admin.initializeApp()
- which must be called at top level, gets called at the end of the bundle result (literally line 2937 out of 2940).
I changed rollup to output dir
so that I could set preseveModules:true
// rollup.config.js
output: [
{
dir: `dist`,
format: 'es',
sourcemap: true,
preserveModules: true,
preserveModulesRoot: "./"
},
],
I also added the generatePackageJson
plugin, to generate a package.json
in dist
- firebase asks for a package json to be present with the code.
This ended up working well. Your repo helped tremendously, thanks again.
@Hacksore – Thanks for sharing a demo!
From my understanding, when deploying to Cloud Functions, Cloud Build installs your dependencies with npm ci
, but your apps/api
directory doesn't have a package-lock.json
file, so your dependencies are installed in a non-deterministic way, which isn't desirable. If any dependency (e.g. firebase-functions) were introduced a bug in a minor release, this bug would suddenly appear on production.
I've been trying to set up a deployment task/flow to get around this, but couldn't figure it out. Any thoughts on this?
@alextouzel Oh I didn't event think about that and you're most likely right.
The main issue with turbo+firebase is as follows.
- Bundling(using rollup to get around this)
- Idempotent function builds (mentioned by @alextouzel)
If anyone finds a solution to the package-lock.json
problem feel free to send a PR to turborepo-firebase-example.
At some point when we have a good base I can update #1966 with the desired implementation.
@Hacksore love what you've done so far.
Context: We use Firebase Hosting, Functions and Cloud Run for our web app, background functions and a couple APIs respectively.
In the last week, I've migrated our API deployments to follow the Docker example which is working nicely but still having issues getting Firebase Functions to play well with local deps.
I'm wondering whether the turbo prune --docker
command might be able to solve the lock file issue and allow us to run the pruned workspaces together through the Cloud Build stage.
@jaredpalmer and @alextouzel, any thoughts or obvious issues with this approach would be appreciated. Would be awesome if someone with a better understanding of lock file pruning than me could weigh in
Thanks
Hey @MitchSRobinson,
We're also using turbo prune
to deploy an api on App Engine (which uses Cloud Build). Works wonderfully. I tried using a similar approach to deploy Cloud Functions but no luck. I tried the following:
- Run
turbo prune --scope=cloud-functions
- Move the generated
out/packages
directory andout/package-lock.json
file inside the cloud-functions workspace - Deploy from the cloud-functions workspace
Result
I keep getting the following error on deployment: Build failed: npm ERR! Cannot read property '[name of our first dependency in package.json]' of undefined
I've tried some variations of this approach but none were successful. When I deploy from our cloud-functions workspace directly (without a package-lock.json
), it works, but as mentioned, it's not a good idea to deploy without a package-lock.json
.
Did you manage to get it working on your end using turbo prune
(with or without the --docker
flag)?
Hi @alextouzel!
That's really interesting, do you have any understanding of why App Engine and Cloud Functions are having different outcomes? I've used App Engine in the past but now default to Cloud Run so I don't have much active knowledge of where the build steps might be differing.
I haven't managed to test the prune
approach but tried out Firelink as an option but because our functions have dev dependencies (tsconfig and eslint local packages), Firelink wasn't sufficient.
I'm hoping to test either this evening or tomorrow and will chuck any progress here. What environment are you using to handle your deploys? We using Github Actions so can share my workflow files too if helpful.
Assuming we can get to the bottom of this, I'm happy to send @Hacksore some resources/examples for deploying to Cloud Run too so there's a couple more comprehensive/end-to-end GCP examples.
Thank you for helping out @MitchSRobinson! Turns out our cloud functions were running on Node 12, and therefore npm v6, which doesn't support workspaces 🤦♂️ I moved to Node 16 and it worked, but I haven't figured out the best way to automate deployments yet. I managed to deploy successfully (from local) with the following steps:
- Run
turbo prune --scope=cloud-functions
- Change our
firebase.json
file functions config to point to theout
directory generated byturbo prune
:
"functions": {
"source": "out"
}
- Add
"main": "apps/cloud-functions/dist/index.js"
toout/package.json
- Run
firebase deploy --only functions
from the console - Delete
out
Problems with this approach
- When running Firebase emulators on local, I need the
firebase.json
functions config to point toapps/cloud-functions
and notout
. RIght now I just changed it manually to test deployments. - Adding
"main": "apps/cloud-functions/dist/index.js"
toout/package.json
was also done manually. Thispackage.json
file is actually from the root of the monorepo, so I don't want to add this prop permanently.
There's probably a way around these 2 problems. I just didn't get to it. Let me know if you figure out a graceful (or not) way to do it!
EDIT
Just figured out how to fix problem #1. From the Firebase CLI reference:
While firebase.json is used by default, you can pass the --config PATH flag to specify an alternate configuration file.
So you can create a second firebase.json
file, name it how you like and use it for deployments.
EDIT 2
Just discovered there is a native command to edit a package.json
file, which solves problem #2:
npm pkg set 'main'='apps/cloud-functions/dist/index.js'
Given that our cloud functions live in apps/cloud-functions
in our monorepo, I now have the following script in apps/cloud-functions/package.json
. I have very few experience with monorepos and automation of deployments so please correct me or point out anything that looks wrong or could be improved.
"deploy:staging": "cd ../.. && turbo prune --scope cloud-functions && cd out && npm pkg set 'main'='apps/cloud-functions/dist/index.js' && cd .. && firebase deploy --only functions --config firebase.deploy.json && rm -rf out"
After hours of researching and trying the workarounds listed in other issues (e.g. firebase-tools#653 and firebase-functions#172) I came up with the following solution, which I now realize is discussed above as well. It relies on pruning and deploying the monorepo from the root.
The structure of my app:
apps/cf # Cloud Functions app
packages/db # 'db' package used by the cf app
The goal is to use the turbo prune
command to "bundle" the cf
app. The steps below which you can put in a Shell script achieve this in a simple way.
npx turbo build --filter=cf # Compile the TS source code
rm -rf out # Clean the 'out' directory
npx turbo prune --scope=cf # Create the pruned monorepo
rm -rf out/apps/cf/src # Optional. Delete unused TS source code
rm -rf out/packages/db/src # Optional. Delete unused TS source code
# Modify root-level package.json
cd out && npm pkg set name="my-cf-app" main=apps/cf/dist/index.js engines.node=16
Since prune
does not seem to take an --outDir
argument we need to update the functions source in firebase.json (unless you don't mind copying the 'out' dir to some other location after bundling:
...
"functions": {
// ...
"source": "out"
}
...
The out
directory is now ready to be deployed to Firebase. The pruning solution turned out to be much simpler and more reliable than bundling the app with other tools like rollup or tsup. It works thanks to the "workspaces"
field in the root-level package.json of the final bundle.
@kafkas Good stuff!
It does look like newer versions of turbo support an --out-dir
flag for prune so maybe you can tweak your workflow to account for that.
--out-dir string Set the root directory for files output by this command (default "out")
@Hacksore Thanks! Yes, you're right. Didn't realize that.
So I tried using the prune method that @kafkas has outlined and it seems to work pretty well.
Hacksore/turborepo-firebase-example has the latest working code.
Here is my solution which i believe its much better than @Hacksore solution:
- Bundle functions index.js with vite lib mode
- Mark all dependencies as externals except monorepo packages
- Finally don't add custom monorepo packages to function package.json since firebase will throw an error if they can't install a package from npm registry.
@Hacksore in your solution you are naming monorepo packages shared
, if you use a custom name for the package like @yourname/shared firebase will throw an error in deployment.
@benrandja-akram Would you mind setting up a repo to show this in action?
@Hacksore yeah its available her https://github.com/ba-oss/turborepo-firebase
@benrandja-akram
Great example. One additional note, it can be even a little better because now this example deploying all content of api folder to the cloud. Check below commit with also example of Gen 2. Cloud function.
https://github.com/ahanusek/turborepo-firebase/commit/1df1d4d98e621dbdd19b83769524f2eea3af8c02
Before:

After:

EDIT: BTW with my approach don't need to build packages before functions deploy. We can remove the build script and vite config for @shared/core and then:
"name": "@shared/core",
"version": "0.0.1",
"private": true,
...
"types": "./index.ts",
"main": "./index.ts",
apps/api/vite.config.ts
...
rollupOptions: {
external: Object.keys(dependencies).filter((packageName) => !packageName.startsWith("@shared")),
plugins: [generatePackageJson({ baseContents: basePackage, additionalDependencies: { "firebase-functions": "^4.2.1" } })],
}
...
and still keep local package in apps/api/package.json:
"dependencies": {
"@shared/core": "*",
"cors": "^2.8.5",
"express": "^4.17.3",
"firebase-admin": "^11.5.0",
"firebase-functions": "^4.2.1",
"vite": "^4.0.3"
},
"devDependencies": {
"rollup-plugin-generate-package-json": "^3.2.0"
},
@ahanusek @benrandja-akram
Thanks ❤️ Very neat solution, love it
Added a few extra lines to avoid coupling vite config with @shared
:
// vite.config.js
const externalDepsList = [];
const externalDepsObj = {};
Object.keys(pckJson.dependencies).forEach((packageName) => {
if (pckJson.dependencies[packageName] !== "*") {
externalDepsList.push(packageName);
externalDepsObj[packageName] = pckJson.dependencies[packageName];
}
});
export default defineConfig({
rollupOptions: {
external: externalDepsList,
plugins: [
additionalDependencies: externalDepsObj
}),
],
},
},
});
// firebase.json
"functions": [
{
"source": "apps/functions/build",
"predeploy": [
"npm --prefix \"$RESOURCE_DIR\"/.. run build"
]
}
],
FYI I pushed the latest changes discussed in this thread to Hacksore/turborepo-firebase-example and it seems to be working.
However, I have yet to add @shelooks16's nice addition but may do that next.
This whole thread has been a life saver for me! Thanks guys!
I implemented all of the suggestions including what @shelooks16 said, and it works great in my app.
The only problem was that subpackages of the external dependencies (e.g. 'firebase/auth', firebase-functions/v1', etc.) were still being bundled in (leaving me with an output bundle of more than 100,000 LoC!). I fixed it by using this regex in the external dependencies list:
// vite.config.js
const externalDepsList = [];
const externalDepsObj = {};
Object.keys(pckJson.dependencies).forEach((packageName) => {
if (pckJson.dependencies[packageName] !== "*") {
const re = new RegExp(`^${packageName}(\/|$)`)
externalDepsList.push(re);
externalDepsObj[packageName] = pckJson.dependencies[packageName];
}
});
This matches all dependencies and their subpackages, e.g. 'firebase', 'firebase/auth', 'firebase-functions/v1' and includes them as external dependencies. My resulting bundle was only 407 LoC :)
Heh, I've switched to webpack eventually. With Rollup/Vite had problem with the bundle size because still had bundling external packages into the output. Works fine with functions v1 and also v2. Bundle size for one function with express.js and a few other dependencies has something about 10-20kB. Also generated package.json contains all the required deps because with rollup I had issue with missing deps from imported @shared packages.
webpack.config.js
const { engines, name, version } = require("./package.json");
const path = require("path");
const nodeExternals = require("webpack-node-externals");
const GeneratePackageJsonPlugin = require("generate-package-json-webpack-plugin");
const resolveTsconfigPathsToAlias = require("./resolve-ts-config-to-aliases");
/**
* A Webpack 5 plugin that can be passed a list of packages that are of type
* ESM. The typescript compiler will then be instructed to use the `import`
* external type.
*/
class ESMLoader {
static defaultOptions = {
esmPackages: "all",
};
constructor(options = {}) {
this.options = { ...ESMLoader.defaultOptions, ...options };
}
apply(compiler) {
compiler.hooks.compilation.tap("ECMAScript Module (ESM) Loader. Turns require() into import()", (compilation) => {
compilation.hooks.buildModule.tap("Hello World Plugin", (module) => {
if (
module.type === "javascript/dynamic" &&
(this.options.esmPackages === "all" || this.options.esmPackages.includes(module.request))
) {
// All types documented at
// https://webpack.js.org/configuration/externals/#externalstype
// module.externalType = 'import';
module.request = "@babel/runtime/helpers/objectSpread2";
}
});
});
}
}
const basePackage = {
name,
version,
main: "./index.js",
scripts: {
start: "yarn run shell",
},
engines,
};
module.exports = {
target: "node",
mode: "production",
entry: "./src/index.ts",
module: {
rules: [
{
test: /\.tsx?$/,
loader: "ts-loader",
exclude: /node_modules/,
options: {
configFile: "tsconfig.json",
transpileOnly: true,
},
},
],
},
resolve: {
extensions: [".tsx", ".ts", ".js", ".json"],
alias: {
...resolveTsconfigPathsToAlias(path.resolve(__dirname, "tsconfig.json")),
},
},
output: {
filename: "index.js",
path: path.resolve(__dirname, "lib"),
libraryTarget: "commonjs",
},
externals: [
/^firebase.+$/,
/^@google.+$/,
"pg",
nodeExternals({
allowlist: [/^@shared/],
}),
nodeExternals({
modulesDir: path.resolve(__dirname, "../../../node_modules"),
allowlist: [/^@shared/],
}),
],
plugins: [
new GeneratePackageJsonPlugin(basePackage),
new ESMLoader({ esmPackages: "@babel/runtime/helpers/esm/objectSpread2" }),
],
};
For reference, I have created an example repo here including all of the suggestions up to this point (still using vite lib mode):
https://github.com/parkernilson/sharing-code-with-functions
I created my own repo which uses Svelte Kit because I am more familiar with it! For me everything is working great!
@parkernilson Do you mind breaking down step by step what we need to add or change to get this working? I'm having trouble using Firebase Functions with Turborepo and I'm confused how to implement what is discussed here.
@BenJackGill Hi Ben! Sure! It took me like 2 months of tinkering to understand it so I am happy to break it down for you 😅
Essentially, firebase functions don't work with symlinked packages (which is how any monorepo works, it adds symlinks for your local dependencies in the node_modules
folder so you can use them like normal packages). This is because when you deploy a firebase function, the functions
folder with your function's source code is uploaded to firebase, but the symlinks get left behind.
For a while, most of us were essentially just adding pre-deploy
and post-deploy
scripts to manually copy all of our symlinked packages into the functions folder before deploying, and then taking them back out again so that they could get uploaded with the functions, but that was slow and came with its own host of problems.
Then, benrandja-akram came up with a much more elegant solution higher up in this thread:
Here is my solution which i believe its much better than @Hacksore solution:
- Bundle functions index.js with vite lib mode
- Mark all dependencies as externals except monorepo packages
The idea here is that instead of trying to copy / paste our symlinked packages, we can instead use a javascript bundler (like Vite or Webpack) to bundle everything into one javascript package (all of your dependencies end up being included inside index.js) and then deploying that to firebase functions. Sounds great right? Well, there are a couple things to keep in mind.
For one, if you include big dependencies like Firebase SDK
or some other large npm package, that will also get included in your bundled js package and it ends up being really large, then it has to get uploaded to firebase which takes a really long time. This might work (I honestly don't remember if it does or not), but a better solution is to leave those external
dependencies (dependencies that you got from npm / yarn / etc.) out of the bundle and let firebase install them after you upload your source code (which is how firebase functions work, they look at the package.json of your uploaded code and use that to install dependencies). We do this by setting external
dependencies in vite.config.js
Another problem with using a bundler is that sometimes they have a hard time resolving symlinks. Luckily, Vite (and apparently Webpack too, but I haven't used it) has a setting that tells it to treat symlinked files like real files and include them in the bundled output.
Keeping this in mind, we can bundle our function's source code into a new package which includes the local symlinked dependencies inline, and then upload that as the source code for our firebase function instead. That is the solution. Every time we deploy to firebase, vite will build our package into a new bundled package and that gets uploaded to firebase.
That's an overall explanation, now I will briefly explain all the essential pieces of code from my example repo that have to do with getting this to work.
Essentially all the magic is happening in /apps/functions/vite.config.ts
. In this file we are doing the following:
- Telling vite to bundle our whole
/app/functions
package into one output file (/dist/index.js
) - Generating a package.json for that new bundled version of our functions code
- Copying over other important files from our
/functions
directory that should be uploaded to firebase with our new bundled package (for example, I use .env files to set environment variables for my cloud functions).
Some other notes:
- We need to set all of our
external
dependencies (the ones we got from npm / yarn / etc.) in therollupOptions
so that it does not include them in the bundle. That way we can just leave them as dependencies in the new/dist/package.json
so that firebase installs them after they are uploaded - You should make sure that you are pointing at
/apps/functions/dist
in your firebase.json instead of/apps/functions
- On a somewhat unrelated note, I also had to set
preserveSymlinks: true
andoptimizeDeps.exclude: [<monorepo packages here>]
for my web app to consume the monorepo packages properly as well. See the source code for more detailed comments
TLDR; This all may have been a way to verbose way to say that the solution is to use a javascript bundler (I used vite lib mode) to bundle your functions code, leaving out external dependencies and creating a new package.json for the bundled code and uploading the bundled code to firebase. This way, symlinked packages are included directly in the source code and uploaded to firebase with everything else.
Please let me know if you have any other questions! I am happy to help 😁
Just confirming that @parkernilson 's solution works well - we have done something similar in our monorepo.
About half a year ago we started using this instead: https://github.com/willviles/firebase-pnpm-workspaces
We run it in CI - when PR are merged to main.
@parkernilson Ok thanks, I get it now!
Previously, I was using a bash script to pack everything before deployment, but I like this method much better.
I almost got it working using your explanation and example repository - almost.
But I have some follow-up questions and issues, if you don't mind:
- For each file I want include in
dist
do I just add another{ src: '.env*', dest: 'dist' }
object withdest: 'dist'
? For example{ src: '.env*', dest: 'dist' }, { src: '.some-file.json', dest: 'dist' }, { src: '.*some-other-glob-pattern*.*', dest: 'dist' }
? - Does this setup have the added advantage of tree-shaking? I assume we don't get tree-shaking unless we use a bundler like Vite. I was hoping that a smaller, tree-shaked size would help with reducing cold starts.
- Looking at the external package regex in
vite.config.ts
, I think that everything marked with version*
is an internal monorepo package that gets bundled, while everything else is marked as external and isn't bundled. Does this mean we cannot use a version of*
for external, non-monorepo packages, or they will become marked as internal? - What is the purpose of the
"dev": "vite build --watch",
script? When would I use that? Am I supposed to use it with the firebase emulator somehow? - Unless already answered above, what is the correct npm script for the Firebase emulator? I need something that will watch for file changes and reload automatically.
- I am using Firebase Functions Generation 2, and with that, I am using the
setGlobalOptions
method inindex.ts
. But it seems thatsetGlobalOptions
is ignored during deployment. I can see in the bundledindex.js
thatsetGlobalOptions
is used, but it is placed at the very end of the file. I assume it needs to go near the top to have an effect during deployment. I think @ecam900 had a similar problem withadmin.initializeApp()
being put at the bottom, but I can't figure out how to solve the problem for your Vite solution.
For example, here is what my index.ts
looks like. It's supposed to deploy to the region australia-southeast1
, but all my functions are still deployed to the default us-central1
region:
apps/functions/src/index.ts
import { setGlobalOptions } from "firebase-functions/v2";
setGlobalOptions({
region: "australia-southeast1"
});
export { helloWorld as helloworld } from "./api/http/on-request/test";
Here is the full repo if it helps. It's a slightly modified version of your one.
** UPDATE: **
I figured out point 6.
You need to put it in it's own module and import it at the top. That will keep everything in order when it's bundled by Vite. Makes for cleaner code also.
For example:
apps/functions/src/config/firebase.ts
import { setGlobalOptions } from "firebase-functions/v2";
export const globalOptions = setGlobalOptions({
region: "australia-southeast1"
});
apps/functions/src/index.ts
// Must import this at the top so that it stays here when bundled
export { globalOptions } from "./config/firebase";
// Import functions here as normal
export { helloWorld as helloworld } from "./api/http/on-request/test";
I also figured out point 5.
To use the emulator withy Vite we can use this script npm run build:watch & firebase emulators:start --only functions
. Take note that command uses a single &
not a double &&
. This will first run the build:watch
script vite build --watch
in the background and then the emulator. Changes to files should reload, but you need a few seconds for the emulator to reload with the changes each time.
For example:
"scripts": {
"build": "rm -rf ./dist && vite build",
"build:watch": "vite build --watch",
"serve": "npm run build:watch & firebase emulators:start --only functions",
"shell": "npm run build:watch & firebase functions:shell",
"start": "npm run shell",
"deploy": "firebase deploy --only functions",
"logs": "firebase functions:log"
},
Just confirming that @parkernilson 's solution works well - we have done something similar in our monorepo.
About half a year ago we started using this instead: https://github.com/willviles/firebase-pnpm-workspaces
We run it in CI - when PR are merged to main.
@ecam900 thanks for the tip, is there an npm version because I'm using npm not pnpm. I don't need a CI pipeline but it might be helpful to know for the future.