nodejs
Encountered SIGSEGV coming frequently from my nodes after coming up to most recent SHA
Environment
- Platform: AWS ECS Fargate, linux variant v1.4.0
- Docker Version: I am unable to answer this
- Image Tag:
This image has the issue: public.ecr.aws/docker/library/node@sha256:16858294071a56ffd4cce9f17b57136cc39e41507b40e245b4f8e906f7a19463
This is the previous, good image in AWS ECR: public.ecr.aws/docker/library/node@sha256:6178e78b972f79c335df281f4b7674a2d85071aae2af020ffa39f0a770265435
Expected Behavior
My containers should not exit with SIGSEGV.
Current Behavior
Segfault style errors coming in at a rate of about once a minute. Sometimes spiking and leading to a full cluster outage. It appears to be uncorrelated to CPU or memory, as increasing limits either did nothing or exacerbated the crashes.
Possible Solution
Stay on this image for now:
public.ecr.aws/docker/library/node@sha256:6178e78b972f79c335df281f4b7674a2d85071aae2af020ffa39f0a770265435
Steps to Reproduce
I wish I had more here. I'm running a decent amount of traffic, but not too much on my ECS boxes. I tried doubling memory and CPU, no effect. We ran a scan over cloudwatch logs and it was on the 26th, when the new version landed, so we moved back versions here using the old build logs, and that got us out of this problem. Production was really catching on fire a lot though.
I do a lot with building in the container environment and then using COPY at the docker boundary more aggressively than I should (copying modules, Prisma with specified binary targets, etc). So, there's a chance I have some native module build issue. That's my main suspect. However, that hasn't been an issue until this increment.
Thank you for maintaining such important code and if I can provide anything else, please let me know. Thanks!
Just in case it might help, here is a list of my native files. Maybe there is specifically something in my ecosystem causing an issue that you know of:
find node_modules -type f -name "*.node" 2>/dev/null | grep -v "obj\.target"
---
node_modules/lightningcss-darwin-arm64/lightningcss.darwin-arm64.node
node_modules/@tailwindcss/oxide-darwin-arm64/tailwindcss-oxide.darwin-arm64.node
node_modules/@rollup/rollup-darwin-arm64/rollup.darwin-arm64.node
node_modules/prisma/libquery_engine-rhel-openssl-3.0.x.so.node
node_modules/prisma/libquery_engine-darwin-arm64.dylib.node
node_modules/.prisma/client/libquery_engine-rhel-openssl-3.0.x.so.node
node_modules/.prisma/client/libquery_engine-darwin-arm64.dylib.node
node_modules/@prisma/engines/libquery_engine-darwin-arm64.dylib.node
node_modules/@parcel/watcher-darwin-arm64/watcher.node
node_modules/@unrs/resolver-binding-darwin-arm64/resolver.darwin-arm64.node
node_modules/fsevents/fsevents.node
@toddtarsi
public.ecr.aws/docker/library/node@sha256:6178e78b972f79c335df281f4b7674a2d85071aae2af020ffa39f0a770265435
uses Node.js 20.19.5 with tag node:20.19.5-alpine3.22
public.ecr.aws/docker/library/node@sha256:16858294071a56ffd4cce9f17b57136cc39e41507b40e245b4f8e906f7a19463
uses Node.js 20.19.6 with tag node:20.19.6-alpine3.22
You can check the changes in Node.js 20.19.6 in the announcement
In your situation I would check the version of prisma you are using and report your issue on their issue list https://github.com/prisma/prisma/issues.
