esbuild Process Using 13+ GB of Memory

[cdalton713]

Describe the bug The esbuild process is using a ridiculous amount of memory. Our backend stack is relatively small, less than two dozen endpoints.

To Reproduce

Serverless Config:

custom: {
...
    esbuild: {
      bundle: true,
      minify: false,
      sourcemap: true,
      incremental: true,
      exclude: ["aws-sdk", "pg-hstore"],
      target: "node16",
      define: { "require.resolve": undefined },
      platform: "node",
      concurrency: 5,
      // plugins: "./esbuild-plugin.js",
      packager: "pnpm",
      packagePath: "./package.json",
    },
    ...
}

Expected behavior Not 15 GB worth of memory for a build.

Screenshots or Logs

Versions (please complete the following information):

• OS: M1 Macbook 12.2.1
• Serverless Framework Version: 3.24.0
• Plugin Version: 0.15.10

[Hideman85]

I'm having the same issue, I try to run esbuild in CICD with limited ram capacity and the process is just growing exponentially to death of course. Can we force esbuild to run under a certain amount of ram for production build?

[cdalton713]

Setting concurrency = 1 and disableIncremental = true has worked as an interim fix; this reduced esbuid's memory usage to around 600 mb for us. However, these settings of course affect build times - local development is now becoming increasingly difficult because each rebuild takes so long to complete...

[Hideman85]

Oh thank you so much men, it saved me a lot of pain :+1: and actually in my side the speed is not that much impacted.

[mhamadyhya1]

Thanks very much for the solution

[zack37]

I'm having this same problem, even after updating to latest. I've set concurrency to 1 but still have really high cpu and memory usage

esbuild.config.js

module.exports = () => ({
  // concurrency: os.availableParallelism(),
  concurrency: 2,
  target: ["node18"],
  sourcemap: true,
  sourcesContent: false,
  platform: "node",
  format: "cjs",
  loader: loaderMap,
  exclude: ["*"],
  packager: "yarn",
  keepNames: true,
  packagerOptions: {
    ignoreLockfile: true
  },
  watch: {
    pattern: ["./**/*.(js|ts|graphql|html)"],
    ignore: [".esbuild", "dist", "node_modules", ".build", "**/generated", "scripts"]
  }
});

The CPU will spike ot 500-800% while building and then calm down to 0 after it's done, but the memory stays high and climbs on each rebuild.

Versions (please complete the following information):

OS: M1 Macbook 13.5.2 Serverless Framework Version: 3.34.0 Plugin Version: 1.48.2

---

Edit After playing around with the source files and whatnot, I've determined that the memory grows with each function. In our case, we have 13 functions registered, which would give us ~13GB memory consumed by the esbuild process. When commenting out all but 1 function, it was only using 1GB. My guess is that it's bundling up the entire project for each function, and then just holding onto the memory through things like this.buildCache, this.buildResults, etc and not letting go of the memory for the entire lifespan of the EsbuildServerlessPlugin class

---

Edit the second: Realized that all my memory consumption was because esbuild was bundling node_modules locally. Adding

...(process.env.IS_OFFLINE === "true" && { packages: "external" })

fixed the memory hogging for me, but I still have issues with every time I save, the memory footprint increases

[Joshuabaker2]

I believe I found the culprit. In the bundle.ts file, in the bundleMapper function, the "context" is getting created and returned, which I assume means that the context is not getting properly garbage collected.

I changed the code in the following ways (I just patched it);

• I pulled the esbuild import out to the top-level rather than re-importing it every time.
• I swapped out the context creation for a simple build command, and removed the context return.

-        const context = await pkg.context?.(options);
-        let result = await context?.rebuild();
-        if (!result) {
-            result = await pkg.build(options);
-        }
+        const result = await pkg.build?.(options);
...
-        return { bundlePath, entry, result, context };
+        return { bundlePath, entry, result };

This took my esbuild memory usage from ~16gb to ~200mb.

[juanito-caylent]

I was having the same problem, but even more dramatic (~80 Lambdas).

1. Setting concurrency to 1 did not fix it.
2. As a temporary workaround, deploying only the function code individually did work.
3. Manually patching the way @Joshuabaker2 suggested does fix the issue for me and keeps the memory in much better shape.

How can we get this into the real implementation? I'd upvote.

[Joshuabaker2]

I'd hazard a guess that there are competing concerns here in that the context is being returned so serverless-offline works faster with incremental rebuilds (I didn't test this since serverless-offline was so slow for me that I wrote my own implementation to do local testing).

@juanito-caylent do you use serverless-offline with serverless-esbuild? If so, did the patch make a difference for that?

[taijuten]

Seeing the same behaviour. My solution in the meantime was to do packaging without this plugin, and use a custom esbuild script, inspired by https://adieuadieu.medium.com/using-esbuild-in-your-serverless-framework-project-13723db5e32a.

It's not ideal, and I'll be glad to remove this if a solution is found

[walterholohan]

@Joshuabaker2 which bundle.ts file are you referencing? is it https://github.com/floydspace/serverless-esbuild/blob/master/src/bundle.ts

Im still seeing some very high memory usage

[Joshuabaker2]

@Joshuabaker2 which bundle.ts file are you referencing? is it https://github.com/floydspace/serverless-esbuild/blob/master/src/bundle.ts

Im still seeing some very high memory usage

Yes that file @walterholohan , the return is on line 122.

@taijuten did you try removing the import and moving it top-level instead? Perhaps that is also needed.

[walterholohan]

Thanks @Joshuabaker2 , do you think we should raise a PR for this? As for large projects this can be a big issue when trying to run builds on CI especially if you are using cheap linux machines (such as Github actions ubuntu machines)

[Joshuabaker2]

I think it should be merged in, the confounding variable that I mentioned above would need some consideration though and since I don't actually use serverless-offline I'm not really in the position to evaluate it:

I'd hazard a guess that there are competing concerns here in that the context is being returned so serverless-offline works faster with incremental rebuilds (I didn't test this since serverless-offline was so slow for me that I wrote my own implementation to do local testing).

[walterholohan]

@floydspace what do you think?

[MMMikeM]

I'm seeing a similar issue here. I'm curious about what would prevent these changes from coming in?

[AlesioSinopoli]

I'm on the same boat, experienced this problem a couple of times now, specially when making deploys with GitHub Actions and a moderate number of lambdas.

The GitHub Action runner just gets killed out of the blue, via external signal. No error happens in the action for it to be stopped, it just gets killed. I'm assuming the problem here is that the bundling process is taking more resources that the action runner has.

[randomhash]

consistent memory leak while bundling @aws-sdk v3

[jeromeheissler]

I had the same problem, by removing all the import * as ...from my code I no longer had that problem