How to extend a mongoose schema without duplicating its memory footprint?

[mcat95]

Prerequisites

• [X] I have written a descriptive issue title

Mongoose version

6.5.4

Node.js version

16.17.0

MongoDB version

6.0.5

Operating system

Linux

Operating system version (i.e. 20.04, 11.3, 10)

Arch

Issue

Hello!

We use mongoose in a big production application, and recently I am starting to become concerned about the memory usage of our schemas. We have a multi tenant application, in which every tenant can extend some schemas adding what we call "custom" fields. We have around ~50 base schemas, and for every tenant, we search the defined customs fields, and clone the schema. After this process, we have around 900 schemas in memory.

The issue is that these schemas are starting to take up a significant amount on memory, and I am wondering if there is any way to reduce this memory footprint.

The main approach I was thinking about was if there was any way to create a new schema that extends another schema, adding some fields, without the need to clone the entire base schema, and 'reusing' the definitions from the base schema. I do not know enough about the mongoose internals to know if this approach would be possible, but theoretically it should be doable.

To summarize, my questions would be the following:

• Is it possible to extend a mongoose schema without the need to clone the schema?
• Do you recommend any other approach in order to solve this problem?

[vkarpov15]

Is it possible to extend a mongoose schema without the need to clone the schema? Unfortunately not currently

Do you recommend any other approach in order to solve this problem? I don't have any good suggestions, other than splitting up tenants across multiple application servers.

Can you provide a little more info on what your schemas look like and how you're measuring that the schemas are taking up a significant amount of memory? It's hard to imagine 900 schemas causing issues. Quick experiment shows that a schema with 100 properties takes up about 22kb:

$ node
Welcome to Node.js v16.17.0.
Type ".help" for more information.
> const mongoose = require('.')
undefined
> const schema4 = new mongoose.Schema(Object.fromEntries(Array(100).fill(0).map((_, i) => [`key${i}`, 'String']))) 
undefined
> require('object-sizeof')(schema4)
22660
> 

So 900 schemas would take up maybe 20 MB, which is not insignificant, but also unlikely to be too concerning given that most modern app servers have at least a few GB. Are you seeing larger memory usage?

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days

[mcat95]

Hello! Sorry for the delay in response. It's difficult to get a proper size of our individual schemas, and object-sizeof does not quite match the data I can see in the memory devtools for example. We store schemas which can easily have more than 100 fields, including deeply nested subschemas.

As you can see, we are using 165MB of memory to store all schemas. That is okay for now, but we are worried about not being able to control that number as it grows. We'll have to look into splitting our application or dynamically generating the new schemas/models as requests come.