TypeScript typings cause very high memory usage in tsc

[yjwong]

When importing googleapis within a TypeScript project, tsc takes up a large amount of memory (500 MB+). This appears to be the case because when googleapis is imported, it imports the type definition files for every single API that Google has, even if only a single API (e.g. drive_v3) is required.

Environment details

• OS: Ubuntu 20.04 LTS
• Node.js version: 10.20.1 (LTS)
• npm version: N/A (We don't use npm instally, but we use yarn v1.22.4)
• googleapis version: 51.0.0
• typescript version: 3.8.3

Steps to reproduce

Please see reproduction repository here: https://github.com/yjwong/tsc-googleapis-memory-usage

Essentially, even just a simple:

import { google } from 'googleapis';

Will result in high memory usage from tsc.

There is also a related StackOverflow question: https://stackoverflow.com/questions/56599477/typescript-googleapis-surge-in-memory-used

Workaround

Import the APIs individually, like so (example is Google Drive SDK v3):

import { drive_v3 } from 'googleapis/build/src/apis/drive/v3';
import { AuthPlus } from 'googleapis-common/build/src/authplus';

[JustinBeckwith]

Short of splitting the module up into their own sub-modules, I doubt there's anything that we could really do about this :/

[yjwong]

@JustinBeckwith I'm guessing that would be related to this issue? #806

If splitting the module up to individual packages is going to be a massive undertaking, I think it would be also helpful if there will be an officially supported way of including individual modules, like the workaround above. I'm concerned that for example, AuthPlus would be an internal implementation detail.

Requiring an additional 500 MB of memory per build job using tsc just to use one of Google APIs is quite a huge ask. (The alternative is to use the REST APIs directly, but we lose type checking)

[JustinBeckwith]

Yeah, there are a lot of tradeoffs here to be sure. We used to dynamically discover available APIs in the module at runtime, but this badly broke things like webpack and rollup. We can't really reduce the raw volume of TypeScript types on a per-API level, because that gets us in trouble with correctness.

The good news here is that we're actually pretty close to having split modules (20% left is 80% of the work), and it's possible you can take advantage of what we have so far. You can cd into any API dir, and run npm pack to get a single standalone module with just that API. For example:

cd src/apis/drive
npm install
npm pack

That actually should give you a self contained module that only contains what you need. I know this ain't ideal, but it does provide a way to get the isolated module.

[rommguy]

I had trouble packing the relevant packages. Eventually I mapped googleapis package types to an empty types file and the memory issue was resolved. Of course, no types support this way.

[bcoe]

@rommguy are you able to import the APIs that you're using individually, as @yjwong suggests. I believe this is partially a documentation problem, as we should be documenting this as the recommended approach.

[rommguy]

@bcoe I wasn't able to do as @yjwong recommended.
I'm using version 59.0.0 Example -

const { google } = require("googleapis");
const iam = new google.iam("v1");

Now I tried importing like so -

import { iam } from "googleapis/build/src/apis/iam";
const iam = new iam("v1");

Getting an exception: TypeError: google.iam is not a constructor I'm guessing I imported the wrong property or from the wrong path, but I have no idea what should be the correct way.

[bcoe]

@rommguy I believe you can use an API directly like this:

import {iam_v1 as v1} from "googleapis/build/src/apis/iam/v1";
const client = new v1.Iam({});

Let me know if this works for you, and if it's an approach that's viable, we can work on making this easier to do.

[rommguy]

@bcoe Thank you, I really appreciate your help! I'm using several of the apis, and I think it would be best if there was some documentation of how to import each service separately. I'll try what you suggested about the iam API.

[sgb-io]

Thanks @bcoe for the workaround.

This OOM problem is a serious issue for adoption of this library! I wasn't able to tsc a small project (using 512mb memory, which should be enough) that uses 1 api (sheets) due to this.

Sheet-specific workaround for others that view this:

import { sheets_v4 as v4 } from "googleapis/build/src/apis/sheets/v4";

const sheets = new v4.Sheets({});

[bcoe]

@sgb-io @rommguy we have now also started publishing individual modules for any libraries in googleapis that don't have an alternative client:

• https://www.npmjs.com/package/@googleapis/sheets
• https://www.npmjs.com/package/@googleapis/iam

Give these a shot, rather than the hack I proposed?

[geoguide]

I'm was shocked to find this issue in my app after a ton of digging. I feel like there should be a major callout to use the individual modules since it's written in typescript.

If you need to reduce startup times

doesn't describe why to use the smaller libraries. It caused my app to crash due to memory no matter what i tried if i imported anything.

[matsko]

This is still a huge issue. It's really hard to pin down since tools like PNPM and Turborepo do not tell you that the process quit because of a memory surge. Most NodeJS devs do not know that 512m is the memory limit for Node processes.

Can you guys please add some exports to your package.json? Things like googleapis/youtube and googleapis/oauth? Having devs find the build artifacts in the path is a problem since the path values could change at any time.

For now you can use NODE_OPTIONS to force a higher limit. I know my project goes close to 2GB.

NODE_OPTIONS="--max-old-space-size=4096" pnpm tsc --project ./tsconfig.json