sentry-python AWS Layer version 37 doesn't work on arm64 lambda

How do you use Sentry?

Sentry Saas (sentry.io)

Version

arn:aws:lambda:ap-southeast-2:943013980633:layer:SentryPythonServerlessSDK:37

Steps to Reproduce

AWS Lambda layer = arn:aws:lambda:ap-southeast-2:943013980633:layer:SentryPythonServerlessSDK:37

I have tried from version 22, the last version that still works is 24, the first version has this problem is 25.

Also confirm version 37 works on x86 architecture. My guess is something new introduced since layver version 25 only works for x86, most likely the /opt/relay/relay.

Expected Result

Lambda function should work. eg

Response
{
... whatever application returned ...
}

Function Logs
START RequestId: f415d848-dfb6-4361-b01d-b1a3b11cd2e2 Version: $LATEST
[INFO]	2022-09-12T04:22:46.550Z	f415d848-dfb6-4361-b01d-b1a3b11cd2e2	Found credentials in environment variables.
[INFO]	2022-09-12T04:22:47.490Z	f415d848-dfb6-4361-b01d-b1a3b11cd2e2	successfully patched module requests
[INFO]	2022-09-12T04:22:47.550Z	f415d848-dfb6-4361-b01d-b1a3b11cd2e2	successfully patched module botocore
[INFO]	2022-09-12T04:22:47.649Z	f415d848-dfb6-4361-b01d-b1a3b11cd2e2	successfully patched module sqlite3
[INFO]	2022-09-12T04:22:47.907Z	f415d848-dfb6-4361-b01d-b1a3b11cd2e2	Found credentials in environment variables.
... application log ignored...
END RequestId: f415d848-dfb6-4361-b01d-b1a3b11cd2e2
REPORT RequestId: f415d848-dfb6-4361-b01d-b1a3b11cd2e2	Duration: 12914.55 ms	Billed Duration: 12915 ms	Memory Size: 128 MB	Max Memory Used: 93 MB	Init Duration: 423.18 ms	
XRAY TraceId: 1-631eb40c-550e81323a18faa810a21797	SegmentId: 6cc523cc159fcfb8	Sampled: true

Actual Result

Response
{
  "errorMessage": "RequestId: 16beabbe-5555-4162-8aba-efba85e5e2fa Error: exit status 1",
  "errorType": "Extension.Crash"
}

Function Logs
START RequestId: 16beabbe-5555-4162-8aba-efba85e5e2fa Version: $LATEST
+ exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api=127.0.0.1:9001
/opt/extensions/sentry-lambda-extension: line 3: /opt/relay/relay: cannot execute binary file
/opt/extensions/sentry-lambda-extension: line 3: /opt/relay/relay: Success
EXTENSION	Name: sentry-lambda-extension	State: Started	Events: []
END RequestId: 16beabbe-5555-4162-8aba-efba85e5e2fa
REPORT RequestId: 16beabbe-5555-4162-8aba-efba85e5e2fa	Duration: 98.92 ms	Billed Duration: 99 ms	Memory Size: 128 MB	Max Memory Used: 3 MB	
XRAY TraceId: 1-631eb380-49977a6b4c3ad20a2c539e0a	SegmentId: 11d7af5965f61201	Sampled: true	
RequestId: 16beabbe-5555-4162-8aba-efba85e5e2fa Error: exit status 1
Extension.Crash

Sep 12 '22 04:09 zhaowb

@zhaowb yes, sorry we released a new AWS extension based layer and ARM is currently not supported. I would recommend staying on an older version till we also support ARM architectures.

Sep 12 '22 11:09 sl0thentr0py

@sl0thentr0py

Seeing the same issues:

+ exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api=127.0.0.1:9001
--
/opt/extensions/sentry-lambda-extension: line 3: /opt/relay/relay: cannot execute binary file
/opt/extensions/sentry-lambda-extension: line 3: /opt/relay/relay: Success
EXTENSION	Name: sentry-lambda-extension	State: Started	Events: []
+ exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api=127.0.0.1:9001
/opt/extensions/sentry-lambda-extension: line 3: /opt/relay/relay: cannot execute binary file
/opt/extensions/sentry-lambda-extension: line 3: /opt/relay/relay: Success
EXTENSION	Name: sentry-lambda-extension	State: Started	Events: []
START RequestId: dfc1ba91-786e-44b1-84cb-bc51c1f1e816 Version: $LATEST
RequestId: dfc1ba91-786e-44b1-84cb-bc51c1f1e816 Error: exit status 1Extension.Crash
END RequestId: dfc1ba91-786e-44b1-84cb-bc51c1f1e816
REPORT RequestId: dfc1ba91-786e-44b1-84cb-bc51c1f1e816	Duration: 95.27 ms	Billed Duration: 96 ms	Memory Size: 128 MB	Max Memory Used: 3 MB

Using ARM64 architecture is preferable as it is faster and cheaper to run.

Nov 10 '22 08:11 tvb

Update, see https://github.com/getsentry/sentry-javascript/issues/6179#issuecomment-1310191338

Nov 10 '22 12:11 tvb

We do not support ARM Lambda layers at the moment. Sorry. Background: we use a rust binary in our Lambda layers and we do not have a ARM version of this rust binary at the moment.

Nov 10 '22 12:11 antonpirker

Hoping this will be supported soon!

Nov 15 '22 20:11 tvb

Mee too!

Nov 16 '22 12:11 antonpirker

Mee too!

Where can I check this specific rust package? And perhaps poke some people.

Nov 18 '22 22:11 tvb

We're hitting this as well. We don't even launch new functions in x86 anymore.

Dec 22 '22 23:12 chris-erickson

For people blocked by this issue, a custom layer can be an easy solution as a drop in replacement. I have a simple sample mysentry.py https://github.com/getsentry/sentry-python/issues/1736 I use this to bypass the performance problem. I have never tried in arm but I guess it should be fine.

Dec 23 '22 00:12 zhaowb

any update on this ?

Jan 18 '23 13:01 Ionut-Milas

I have same issue with node.js arm64 lambda. Can anyone at least give an estimate?

Mar 31 '23 11:03 capaj

This staying open for half a year is concerning. Would it perhaps be possible to give it a higher priority?

Apr 06 '23 19:04 havrylenkok

Hey @zhaowb ! We updated our AWS Lambda integrations recently, could you check if this problem still exists?

Jan 29 '24 13:01 antonpirker

@antonpirker hey that's great news! I don't have access to that infrastructure any more, it was my previous job. Maybe anyone in the thread can confirm ?

Feb 05 '24 19:02 zhaowb

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

Feb 29 '24 08:02 getsantry[bot]