payload icon indicating copy to clipboard operation
payload copied to clipboard

MongoDB adapter: no way to use Vercel attachDatabasePool to avoid MongoDB Atlas connection limit

Open jhb-dev opened this issue 2 months ago • 18 comments

Describe the Bug

On Vercel (Fluid Compute enabled) with Payload v3 and the MongoDB adapter, I frequently hit MongoDB Atlas Flex’s 500 concurrent connections limit. Other developers have reported similar connection limit issues on Discord.

Vercel recommends attaching database clients to its function lifecycle using @vercel/functions attachDatabasePool so connections are cleaned up when functions are suspended. This pattern is not currently possible with Payload’s MongoDB adapter, since there is no supported way to inject or wrap the MongoClient instance.

Reference (Vercel announcement and sample code):

import { MongoClient, MongoClientOptions } from 'mongodb';
import { attachDatabasePool } from '@vercel/functions';

const options: MongoClientOptions = {
  appName: "devrel.vercel.integration",
  maxIdleTimeMS: 5000
};
const client = new MongoClient(process.env.MONGODB_URI, options);
   
// Attach the client to ensure proper cleanup on function suspension
attachDatabasePool(client);

// Export a module-scoped MongoClient to ensure the client can be shared across functions.
export default client; 

In general, I could not find any official resources from Payload explaining how to prevent excessive database connections when hosting on a serverless platform like Vercel.

Link to the code that reproduces this issue

https://github.com/payloadcms/payload/tree/main/packages/db-mongodb

Reproduction Steps

Which area(s) are affected? (Select all that apply)

db: mongodb

Environment Info

Binaries:
  Node: 24.3.0
  npm: 11.4.2
  Yarn: N/A
  pnpm: 10.12.4
Relevant Packages:
  payload: 3.62.0
Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041
  Available memory (MB): 24576
  Available CPU cores: 14

jhb-dev avatar Nov 10 '25 09:11 jhb-dev

Same issue here, last night my application on Vercel fully bricked as i hit the 500 connection limit with MongoDB Atlas. Website became fully unresponsive as no more connection could be made. I assume this would solve it. Maybe also related to #7312 ?

joshuademarco avatar Nov 13 '25 09:11 joshuademarco

Yeah I'm having the same issues on a few projects. All of them are using the nested docs plugin

ChristopherNowlan avatar Nov 18 '25 07:11 ChristopherNowlan

Please test out https://github.com/payloadcms/payload/pull/14649 and let me know if it works (see PR description for example).

Pre-release: 3.65.0-internal.ef3958a

@ChristopherNowlan @joshuademarco @jhb-dev

AlessioGr avatar Nov 18 '25 07:11 AlessioGr

Please test out #14649 and let me know if it works (see PR description for example).

Pre-release: 3.65.0-internal.ef3958a

@ChristopherNowlan @joshuademarco @jhb-dev

Didn't test it in length but it looks solved to me. Connections stabilized around 100 compared to the volatile 200-500 connections before: Image

Thanks for jumping on this so quickly. Really appreciate it!

joshuademarco avatar Nov 18 '25 14:11 joshuademarco

I think its better. Will let you know how it goes during heavy admin use Image

ChristopherNowlan avatar Nov 19 '25 00:11 ChristopherNowlan

On another project as soon as a user started using the CMS it spiked. Happy to share access to repo. See if its something I have done. Image

ChristopherNowlan avatar Nov 19 '25 06:11 ChristopherNowlan

@AlessioGr I don't think the above fix actually fixed my issue. When a user uses the admin the connections spike and the cms pretty much become unusable.

ChristopherNowlan avatar Nov 19 '25 23:11 ChristopherNowlan

I can confirm that the 500-connection limit still gets hit and breaks the application. However, i have a hunch that we have two separate issues here.

Referring to my post above, connections (at least in my example) have become generally les and more granular after implementing attachDatabasePool. Meaning that even if connection spikes, it does not hold for long and quickly reverts to a low number. I would say this confirms that the implementation was correct and the function works as intended:

"This function ensures that idle pool clients are properly released before functions suspend." - Vercel - Docs

Something i realized is that connection count definitively spikes whenever i open media collection in the admin panel. Can somebody confirm this behaviour? Even after setting disablePayloadAccessControl: true

Also, if somebody is interested about the problem: The real serverless compute to database connection problem, solved

joshuademarco avatar Nov 20 '25 21:11 joshuademarco

@joshuademarco Same when the media collection is open it spikes or uploading a image. Then seems to stay high for awhile. Im using @payloadcms/storage-s3 . I haven't disable access control yet. Is it worth adding?

ChristopherNowlan avatar Nov 21 '25 00:11 ChristopherNowlan

@joshuademarco @ChristopherNowlan I've had similar issues with the media collection (also using storage-s3). Implementing disablePayloadAccessControl: true helped, but what (almost) solved it for me was removing the image preview from the Media list view by setting defaultColumns to exclude the image/media preview.

I never confirmed this, but it looked to me that a new DB connection was being opened for each image that was displayed on the list view - If I set to the page to display 100 entries at a time, my DB connections would blow up.

mquayle-viewa avatar Nov 21 '25 00:11 mquayle-viewa

@mquayle-viewa removing the filename column from the default columns show seems to have reduced the connections. @AlessioGr maybe something worth investigating there.

ChristopherNowlan avatar Nov 21 '25 01:11 ChristopherNowlan

Thanks for the quick fix and the pre-release, @AlessioGr!

I can also confirm that connections now close significantly faster, resulting in a more granular and overall lower number of open connections.

@ChristopherNowlan @mquayle-viewa @joshuademarco I previously encountered the same issue where opening the media list view, particularly with a large pagination limit, caused a big surge in new connections. The underlying reason was that the /api/media endpoint was hit with many parallel requests to load the thumbnail images.

I resolved this by setting disablePayloadAccessControl: true and ensuring that thumbnailURL points directly to the storage provider (in my case S3). With this approach, there is no need to disable the filename (and therefore the thumbnail) column in the list view.

I recommend opening the media list view with your browser’s dev tools to inspect the outgoing requests. If you observe numerous calls to /api/media, this is likely the cause of the elevated number of database connections.

jhb-dev avatar Nov 21 '25 06:11 jhb-dev

@jhb-dev I tried that but than all my images 404. Do I need to update my image component?

ChristopherNowlan avatar Nov 21 '25 07:11 ChristopherNowlan

@ChristopherNowlan so you mean the thumbnail images do not load in the list view? I think then you should check if the thumbnailURL field is correctly generated for your images. It might also be worth checking if these are incorrectly stored in the DB (may be related to https://github.com/payloadcms/payload/issues/14562)

jhb-dev avatar Nov 21 '25 07:11 jhb-dev

@jhb-dev the image don't load on the front end either - https://drkaylenehenderson.vercel.app/

ChristopherNowlan avatar Nov 21 '25 08:11 ChristopherNowlan

@ChristopherNowlan It looks like there is an issue with the url and thumbnailURL fields in your media collection. When inspecting the media entries via the API (https://drkaylenehenderson.vercel.app/api/media), you can see URLs such as undefined/drkaylenehenderson.com/Resource Pack 1-300x188.webp

jhb-dev avatar Nov 21 '25 08:11 jhb-dev

@ChristopherNowlan @jhb-dev @mquayle-viewa Definitively the issue, setting disablePayloadAccessControl: true does indeed still cause thumbnails to be queried through db and ultimately result in multiple not found.

A shame that Payload does not point the thumbnail automatically to the blob like it does with file url. Setting the following in the upload field, is working for me:

adminThumbnail: ({ doc }) =>
      `https://<STORAGE URL>/${(doc.sizes as any)?.thumbnail?.filename ?? doc.filename}`,

@AlessioGr I don't know how easy the above is to fix but maybe we could get a little documentation section under Deployment > Performance? To prevent others from going down that rabbit hole too

joshuademarco avatar Nov 21 '25 09:11 joshuademarco

🚀 This is included in version v3.65.0

github-actions[bot] avatar Nov 25 '25 18:11 github-actions[bot]