payload
payload copied to clipboard
payloadcms
Cloud storage plugin socket usage at capacity
Link to reproduction
No response
Describe the Bug
I am using the cloud storage plugin with an S3 adapter pointing to DigitalOcean Spaces. I have users making a lot of edits relating to images lately. After a significant amount of uploads and image swapping, I get reports of performance issues on the admin dashboard and uploads failing. I looked at the logs and I found these error messages:
@smithy/node-http-handler:WARN socket usage at capacity=50 and 102 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 114 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 120 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 121 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 125 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 135 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 145 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 156 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 158 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 161 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 163 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 166 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 169 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 170 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
These socket capacity warnings are sometimes preceded by error messages like this:
ERROR (payload): UnknownError
err: {
"type": "NoSuchKey",
"message": "UnknownError",
"stack":
NoSuchKey: UnknownError
at de_NoSuchKeyRes (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4809:21)
at de_CommandError (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4747:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /home/node/app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
at async /home/node/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:225:18
at async /home/node/app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
at async /home/node/app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:173:18
at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:97:20
at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:120:14
at async /home/node/app/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
"name": "NoSuchKey",
"$fault": "client",
"$metadata": {
"httpStatusCode": 404,
"requestId": "[request id]",
"attempts": 1,
"totalRetryDelay": 0
},
"Code": "NoSuchKey",
"BucketName": "bucket-name",
"RequestId": "[request id]",
"HostId": "[host id]"
}
I have my Payload app hosted on a basic DigitalOcean Droplet (1 vCPU, 1GB RAM, 25GB storage, added 1GB swap space) if that helps at all.
To Reproduce
It seems like it happens when a lot of uploading is done. I haven't seen a clear-cut way to reproduce, but the best I can do is:
- Set up a payload app with the cloud storage plugin
- Point to a DigitalOcean Spaces bucket
- Create an upload collection
- Upload images in quick succession. Upload new images into existing records. Whatever can be done to trigger the errors.
Payload Version
2.14.1
Adapters and Plugins
"@payloadcms/bundler-webpack": "^1.0.6", "@payloadcms/db-mongodb": "^1.5.1", "@payloadcms/plugin-cloud-storage": "^1.1.2", "@payloadcms/plugin-nested-docs": "^1.0.12", "@payloadcms/plugin-seo": "^2.3.1", "@payloadcms/richtext-lexical": "^0.9.1", "@aws-sdk/client-s3": "^3.556.0", "@aws-sdk/lib-storage": "^3.556.0", "aws-crt": "^1.21.2"
Also having this problem, after a while(not more than hour) we stop getting images and the same socket warning start showing up... Increasing the socket number didn't help. We are hosting on render, images in S3 and DB is Mongo
@TimLanzi @Stroi Can either of you give me an idea of your throughput for uploads? Seems like this might be tricky to reproduce.
It's been a while so I can't really give exact numbers. If I had to guess, I'd say 15 upload events in 10 minutes. Image sizes were around 1MB each. I also have imageSizes set up as follows:
imageSizes: [
{
name: 'thumbnail',
width: 400,
height: 300,
position: 'centre',
},
{
name: 'card',
width: 768,
height: 1024,
position: 'centre',
},
{
name: 'tablet',
width: 1024,
height: null,
position: 'centre',
},
],
I have the same issue with almost the same payloadcms settings as @TimLanzi, same payloadcms and s3 errors when uploading images.
Edit: looks like the plugin is creating s3 socket connections that its not closing @denolfe
I'm having exactly the same issue.
"name":"payload","err":{"type":"NoSuchKey","message":"The specified key does not exist.","stack":"NoSuchKey: The specified key does not exist.\n at de_NoSuchKeyRes (/home/node/backend/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4791:21)\n at de_CommandError (/home/node/backend/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4729:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/node/backend/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /home/node/backend/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:226:18\n at async /home/node/backend/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n at async /home/node/backend/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:174:18\n at async /home/node/backend/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:98:20\n at async /home/node/backend/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:121:14\n at async /home/node/backend/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22","name":"NoSuchKey","$fault":"client","$metadata":{"httpStatusCode":404,"requestId":"17E092AADF93FA23","extendedRequestId":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","attempts":1,"totalRetryDelay":0},"Code":"NoSuchKey","Key":"BPRO-Spender-EN.pdf","BucketName":"production","Resource":"/production/BPRO-Spender-EN.pdf",,"},"msg":"The specified key does not exist."}
2024-07-09T15:19:53.037446982Z stdout F {"level":50,"time":1720538393036,"pid":7, "name":"payload","err":{"type":"Error","message":"Socket timed out without establishing a connection within 10000 ms","stack":"TimeoutError: Socket timed out without establishing a connection within 10000 ms\n at Timeout._onTimeout (/home/node/backend/node_modules/@smithy/node-http-handler/dist-cjs/index.js:67:21)\n at listOnTimeout (node:internal/timers:573:17)\n at process.processTimers (node:internal/timers:514:7)","name":"TimeoutError","$metadata":{"attempts":3,"totalRetryDelay":146}},"msg":"Socket timed out without establishing a connection within 10000 ms"}
I found a thread that deals with this specific issue: https://github.com/aws/aws-sdk-js-v3/issues/3560
I would give a few of these solutions a try.
My cloud hosting provider pointed also other issues related to this:
https://github.com/aws/aws-sdk-js-v3/issues/6263
https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/CLIENTS.md#request-handler-requesthandler
I believe I have finally managed to recreate the issue with my test setup. It required me to upload a video to payload, I then ran 120 curl instances at once with a timeout shorter than the video length and that seems to trigger the problem. There is an interesting comment on the AWS SDK regarding socket exhaustion which helped me identify the problem.
I found a thread that deals with this specific issue: aws/aws-sdk-js-v3#3560
I would give a few of these solutions a try.
Hello, Hi can I implement this solution using the adapter?
httpsAgent: new Agent({
maxSockets: 500,
// keepAlive is a default from AWS SDK. We want to preserve this for
// performance reasons.
keepAlive: true,
keepAliveMsecs: 1000,
}),
socketTimeout: 5000,
}),
When I tried to import { NodeHttpHandler } from '@smithy/node-http-handler';
I had this error
Module not found: Error: Can't resolve 'http2' in '/home/node/app/node_modules/@smithy/node-http-handler/dist-es'
I found a thread that deals with this specific issue: aws/aws-sdk-js-v3#3560 I would give a few of these solutions a try.
Hello, Hi can I implement this solution using the adapter?
httpsAgent: new Agent({ maxSockets: 500, // keepAlive is a default from AWS SDK. We want to preserve this for // performance reasons. keepAlive: true, keepAliveMsecs: 1000, }), socketTimeout: 5000, }),When I tried to import { NodeHttpHandler } from '@smithy/node-http-handler'; I had this error
Module not found: Error: Can't resolve 'http2' in '/home/node/app/node_modules/@smithy/node-http-handler/dist-es'
My hosting provider gave me a similar solution to try
import https from "https";
requestHandler: {
connectionTimeout: 10 * 1000,
httpsAgent: new Agent({
maxSockets: Infinity,
keepAlive: false,
}),
keepAlive: false,
maxSockets: Infinity,
}
However, I got stuck on the same issue you are facing. I couldn't resolve the package within Payload.
I found a thread that deals with this specific issue: aws/aws-sdk-js-v3#3560 I would give a few of these solutions a try.
Hello, Hi can I implement this solution using the adapter?
httpsAgent: new Agent({ maxSockets: 500, // keepAlive is a default from AWS SDK. We want to preserve this for // performance reasons. keepAlive: true, keepAliveMsecs: 1000, }), socketTimeout: 5000, }),When I tried to import { NodeHttpHandler } from '@smithy/node-http-handler'; I had this error
Module not found: Error: Can't resolve 'http2' in '/home/node/app/node_modules/@smithy/node-http-handler/dist-es'My hosting provider gave me a similar solution to try
import https from "https"; requestHandler: { connectionTimeout: 10 * 1000, httpsAgent: new Agent({ maxSockets: Infinity, keepAlive: false, }), keepAlive: false, maxSockets: Infinity, }However, I got stuck on the same issue you are facing. I couldn't resolve the package within Payload.
@denolfe could you help with this, please?
It's possible http2 was added in a later node version. Try node 20 or newer if you are not already. Pure speculation on my part as there are quite a few possible solutions in that thread.
That's definitely a thing. I have "node": "20.10.0". NodeHttpHandler won't work on browser that's why webpack compilation will fail with Error: Can't resolve 'http2' in '/home/node/app/node_modules/@smithy/node-http-handler/dist-es'. There is no polyfil for that and this workaround http2: m, // m is emptyModule won't work too. So is there no solution to this problem? Cause increasing maxSockets through NodeHttpHandler constructor is a way to go solution according to aws-sdk docs and answers from thread
Okay. So there is a dirty solution. One can implement custom NodeHttpHandler that don't rely in any way on http2 (apparently @smithy/node-http-handler use it only for some constant from there anyway). Then polyfill https and http module in webpack (using 'stream-http' and 'https-browserify'). Maybe ditching payload cloud storage and just using aws-s3-client directly in media collection hooks would be a better solution
i'm facing this issue too:
@smithy/node-http-handler:WARN socket usage at capacity=50 and 102 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handle…
rebuilding the functionality via collection hooks doesn't seem to be the best use of effort, wonder if there is a way to fix this in the cloud storage plugin
We have the same issue in several projects. Changing the request handler causes several webpack errors. Tried to mock some modules, but didn't work. The only workaround is to restart the payload service. So no solution. At this point the plugin is unusable :(
To use this config
requestHandler: {
connectionTimeout: 10 * 1000,
httpsAgent: new https.Agent({
maxSockets: Infinity,
keepAlive: false,
}),
keepAlive: false,
maxSockets: Infinity,
},
We use:
"https-browserify": "^1.0.0",
and add it to the webpack config as fallback
fallback: {
crypto: false,
fs: false,
http: require.resolve("stream-http"),
https: require.resolve("https-browserify"),
url: require.resolve("url"),
},
},
Can anyone else confirm @creative-andrew 's solution works for them?
Alternatively, here is another request handler config formed from a solution given in the linked thread
pnpm add @smithy/node-http-handler https-browserify url
import { NodeHttpHandler } from '@smithy/node-http-handler'
import { Agent as HttpAgent } from 'http'
import { Agent as HttpsAgent } from 'https'
// Pass this in your s3Adapter config
const s3AdapterConfig = {
// .. other options
requestHandler: new NodeHttpHandler({
connectionTimeout: 5000,
socketTimeout: 120000,
httpAgent: new HttpAgent({ maxSockets: 500, keepAlive: true }),
httpsAgent: new HttpsAgent({ maxSockets: 500, keepAlive: true }),
}),
}
Modify webpack as needed. Might need to play around here, but this is what i got to work.
admin: {
webpack: (config => ({
resolve: {
fallback: {
crypto: false,
http: require.resolve('https-browserify'),
http2: false,
https: require.resolve('https-browserify'),
url: require.resolve('url'),
},
}
}))
}
Hey @denolfe - i've had @creative-andrew's solution in production since the 9th of September and the issue hasn't occurred. Usually after 3-4 days the sockets would be getting maxed out
Unfortunately it doesn't work for me.
ERROR in ./node_modules/@smithy/node-http-handler/dist-es/node-http2-connection-manager.js 1:0-26 Module not found: Error: Can't resolve 'http2' in '/home/node/app/node_modules/@smithy/node-http-handler/dist-es'
payload config:
webpack: (config) => {
return {
...config,
resolve: {
...config.resolve,
alias: {
...(config.resolve?.alias ? config.resolve.alias : {}),
jsonwebtoken: mockModulePath,
},
fallback: {
...(config.resolve?.fallback ? config.resolve.fallback : {}),
http: require.resolve("https-browserify"),
http2: false,
https: require.resolve("https-browserify"),
url: require.resolve("url"),
},
},
};
},
cloud storage adapter config:
const cloudStorageAdapter = s3Adapter({
config: {
region: process.env.AWS_REGION,
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
},
requestHandler: new NodeHttpHandler({
connectionTimeout: 5000,
socketTimeout: 120000,
httpAgent: new HttpAgent({ maxSockets: 500, keepAlive: true }),
httpsAgent: new HttpsAgent({ maxSockets: 500, keepAlive: true }),
}),
},
bucket: process.env.AWS_S3_BUCKET_NAME!,
});
@creative-andrew where does the "stream-http" or "url" module come from? which versions of these packages do you use? @denolfe why don't you get issues with the http2 module?
would you share your package.jsons or lock-files as well? maybe the full webpack config? are there problems when using alias and fallback together? (i don't know what all these settings do - never used it before).
@denolfe where in a payload project does the webpack config you shared go?
webpack: (config => ({
resolve: {
fallback: {
crypto: false,
http: require.resolve('https-browserify'),
http2: false,
https: require.resolve('https-browserify'),
url: require.resolve('url'),
},
}
}))
there's no webpack option in payload v3 from the look of it. I see it in the docs for v2
https://github.com/payloadcms/payload/issues/6382#issuecomment-2325468104
I have this problem in v3, as well - and also with Digital ocean spaces. I wonder if this is a coincidence, or could it be an issue with DO service specifically.
I am running into this problem in v3 now and we need to launch the site next week... does anyone have or know of a hack/workaround that will work in version 3. OR another solution for handling media files in Payload.
While i wasn't able to fix the issue with all these webpack aliases etc. we used a crown job, which checks if the images are still available. If not, it will restart the server.
After other problems with webpack I figured out that a plugin was overriding the config. So probably the already mentioned resolutions will work in my projects as well.
I am getting the same error since upgrading to Payload 3.x. However I only receive this error while trying to load images. It seems like there is an unresolved socket for each image request and thats why it adds up quickly.
Same for me:
@smithy/node-http-handler:WARN - socket usage at capacity=50 and 110 additional requests are enqueued.
There is a configuration change you can make to resolve this issue, if u google it you will find it, my apologies I am not in front of my computer now. I will post it when I get home.
I still continued to get memory issues and my app crashing, I was forced to reduce the number of image sizes I was having payload create using the sharp library.
We are currently testing a v3 app in a staging environment using MinIO as a s3 server and are facing similar problems. Images stop loading after a few minutes after browsing through the media collection.
As a quick fix, I've added the generateFileURL and adminThumbnail functions, as all media that is shown in our admin interface is also publicly available. This way, we are skipping payloads proxying that's currently broken.
Same problem here, using payload 2.29.0 and @payloadcms/plugin-cloud-storage 1.2.0
After a few days, depending on the usage, it stops working. A payload restart solves it, but it's an inconvenience. Moreover, there's no way to configure the node handler library from payload.
