Difficulty optimizing dependencies for AWS Lambda

[spaceemotion]

Prerequisites

• [X] I have written a descriptive issue title
• [X] I have searched existing issues to ensure the issue has not already been raised

Issue

Hi there,

I am currently trying to make my lambda as lean as I can to reduce our cold start times. I noticed that this package pulls in quite a bit, and also uses a couple outdated dependencies.

All my findings are based on the metafile esbuild produces, to check which code actually gets pulled at the end.

1. mime-db seems to be about 130kb in size, just to check which mime-types are compressible. My API only ever deals with JSON data, and also sits in front of an API Gateway, so I was wondering if there is a way to make this optional? (or at least lazy-loaded)
2. The build currently has multiple different versions of string_decoder and readable-stream. I tried to see if there is an easy way to upgrade them all, but even newer versions of through2 only depend on readable-stream@3, whereas version 4 is already out. I am using the pino logger with fastify, which pulls in the up-to-date versions of these packages.
3. Similar case with duplexify:

   dependencies:
   @fastify/compress 7.0.3
   ├─┬ peek-stream 1.1.3
   │ └── duplexify 3.7.1
   └─┬ pumpify 2.0.1
     └── duplexify 4.1.3
   

I only ever need to compress responses, never decompress the input. Are there simpler/leaner/easier ways to do this?

Thanks in advance!

[mcollina]

My rule of thumb is to never compress JSON data. If you target slow networks (anything slower than a 3G phone), refactor your application to transfer less data. The latency and CPU cost of compression is often too high for small payloads. In other terms, I don't think you would have any benefit in using this module.

Having said that, we should be splitting this module into 2:

1. @fastify/compress
2. @fastify/decompress

As well as updating through2, pumpify and all the logic here to modern standards. I think most of those are not needed anymore, and we could implement this more straightforwardly as most of those functionalities are now in Node.js core.

This is quite a bit of work. Would you like to volunteer to do it, or would you like to sponsor it?

[spaceemotion]

Thanks for the fast response!

Reason for me compressing JSON is that there's actually a lot of duplication going on in the data (e.g. ProseMirror JSON formatted data, objects with similar structures). Right now I can send over a 1MB blob of data in about 100kb using brotli.

I tried a bunch of compression factors and landed on something that's still fast for most users. Most requests return within 80ms or so, but being able to control this on a per-route (or rather: size-based) mechanic would be nice (I know we can already do this, so keeping this feature is important to me).

I only need to compress the data in certain endpoints, but I am also using tRPC inside of Fastify so I can't just add this as a middleware to the fastify routes that need it.

Being able to only import the compression middleware (no decompression) would probably help a lot.

I don't have the time to work on this, and also not the experience with fastify itself - would this be a one-time sponsorship or a repeating one (unclear on how the rules on this are)?

[mcollina]

We don't have any good rules for this, and I'm open to suggestions. You can contact me privately if you want.

cc @Eomm @jsumners

[jsumners]

I really don't have any input to provide here. I wouldn't use AWS Lambda for anything that would require any Fastify branded module. Lambda is a tool for background batch processing, in my opinion.