nitrictech
Building on ARM CPUs causes error on AWS
Describe the bug
The example works fine locally, but when deployed to the AWS platform, the endpoint is not accessible. (I also tried using Pulumi or manually creating a lambda in AWS and adding a gateway, etc., and it worked fine.)
When I call the endpoint directly or try to test under AWS' Lambda test service, I see the following error in the log:
START RequestId: b478530d-bdfe-46e8-8006-70473b2f87ff Version: $LATEST
2022/05/19 05:54:58 Registered Gateway Plugin
2022/05/19 05:54:58 Starting Child Process
2022/05/19 05:54:58 Services listening on: 127.0.0.1:50051
2022/05/19 05:54:58 Membrane Error: there was an error starting the child process: fork/exec /usr/local/bin/ts-node: exec format error, exiting
2022/05/19 05:54:58 gateway 'Stop' called, waiting for lambda runtime to finish
END RequestId: b478530d-bdfe-46e8-8006-70473b2f87ff
REPORT RequestId: b478530d-bdfe-46e8-8006-70473b2f87ff Duration: 15017.11 ms Billed Duration: 15000 ms Memory Size: 128 MB Max Memory Used: 6 MB
2022-05-19T05:55:13.417Z b478530d-bdfe-46e8-8006-70473b2f87ff Task timed out after 15.02 seconds
Expected behavior
Consistent results with local runs after deployment.
To Reproduce
Steps to reproduce the behavior:
- Run
nitric newto create project. - Run
nitric runto check function on local. It works well here. - Run
nitric up -s <my-stack>to deploy. No errors were generated and all resources were successfully deployed. In Pulumi's monitoring panel, I can see that every service is working properly. - Trigger
<endpoint>/hello/world, getInternal Server Error.
I've tried removing all resources and redeploying with the command and got the same result.
Output of nitric info -o yaml:
os: darwin
arch: arm64
goversion: go1.17.9
cliversion: was not built with version info
fabricversion: v0.15.1-rc.8
containerruntime: docker
containerruntimeversion: |
platform:
name: Docker Desktop 4.8.1 (78998)
components:
- name: Engine
version: 20.10.14
details:
ApiVersion: "1.41"
Arch: arm64
BuildTime: "2022-03-24T01:45:44.000000000+00:00"
Experimental: "false"
GitCommit: 87a90dc
GoVersion: go1.16.15
KernelVersion: 5.10.104-linuxkit
MinAPIVersion: "1.12"
Os: linux
- name: containerd
version: 1.5.11
details:
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
- name: runc
version: 1.0.3
details:
GitCommit: v1.0.3-0-gf46b6ba
- name: docker-init
version: 0.19.0
details:
GitCommit: de40ad0
version: 20.10.14
apiversion: "1.41"
minapiversion: "1.12"
gitcommit: 87a90dc
goversion: go1.16.15
os: linux
arch: arm64
kernelversion: 5.10.104-linuxkit
experimental: false
buildtime: "2022-03-24T01:45:44.000000000+00:00"
Brief description of your project
Basic sample, create from nitric new.
Additional context
BTW, Nitric is indeed an amazing service, our team spent a lot of time deploying and debugging multiple lambda's, especially on testing AWS with Pulumi, Nitric's out of the box and 0 configuration was a great fit for us.
I guess our team may be early adopters, feel free to contact me for any subsequent updates or user surveys, and it would be even better if there are features or corporate cooperation (or sponsorship), I am always ready to listen to your voice!
Hi @unix
I tried to reproduce this on our develop branch and it seemed to work. Here are my steps.
nitric new
? What is the name of the project? basic-sample
? Choose a template: official/TypeScript - Starter
? Glob for the function handlers? functions/*.ts
angus@fedora:~/.../github.com/nitrictech/> cd basic-sample/
angus@fedora:~/.../nitrictech/basic-sample/> ls
functions nitric.yaml package.json README.md tsconfig.json
angus@fedora:~/.../nitrictech/basic-sample/> cat nitric.yaml
name: basic-sample
handlers:
- functions/*.ts
angus@fedora:~/.../nitrictech/basic-sample/> ls functions/
hello.ts
angus@fedora:~/.../nitrictech/basic-sample/> cat functions/hello.ts
import { api } from "@nitric/sdk";
const helloApi = api('main');
helloApi.get("/hello/:name", async (ctx) => {
const { name } = ctx.req.params;
ctx.res.body = `Hello ${name}`;
return ctx;
});angus@fedora:~/.../nitrictech/basic-sample/> nitric stack new
? What do you want to call your new stack? test
? Which Cloud do you wish to deploy to? aws
? select the region us-east-1
angus@fedora:~/.../nitrictech/basic-sample/> yarn install
yarn install v1.22.10
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
warning Your current version of Yarn is out of date. The latest version is "1.22.19", while you're on "1.22.10".
Done in 10.20s.
I then did a run
nitric run
SUCCESS Configuration gathered (1s)
SUCCESS Created Dev Image! (0s)
SUCCESS Started Local Services! (17s)
SUCCESS Started Functions! (0s)
Local running, use ctrl-C to stop
In a different window
curl http://localhost:9001/apis/main/hello/angus
Hello angus
I then did the aws deploy
angus@fedora:~/.../nitrictech/basic-sample/> nitric up -s test
SUCCESS Configuration gathered (1s)
SUCCESS Images built (46s)
Deployed Repository/basic-sample-hello (3s)
Deployed Group/basic-sample-test (5s)
Deployed Role/helloLambdaRole (5s)
Deployed RolePolicy/helloListAccess (2s)
Deployed RolePolicyAttachment/helloLambdaBasicExecution (2s)
Deployed Image/hello-image (3m18s)
Deployed Image/hello (3m18s)
Deployed Function/hello (30s)
Deployed AWSLambda/hello (3m48s)
Deployed Api/main (9s)
Deployed Stack/basic-sample-basic-sample-test (4m0s)
Deployed Permission/mainhello (2s)
Deployed Stage/mainDefaultStage (5s)
Deployed Stack (4m9s)
┌───────────────────────────────────────────────────────────────┐
| API | Endpoint |
| main | https://2mjon7cqo6.execute-api.us-east-1.amazonaws.com |
└───────────────────────────────────────────────────────────────┘
and the curl to the above url
angus@fedora:~/.../nitrictech/pro-backend (main %)/> curl https://2mjon7cqo6.execute-api.us-east-1.amazonaws.com/hello/unix
Hello unix
It might be that we just need to release what we have in develop. It looks like it's only about 2 weeks old.
I have tested v1.3.0 and that also works for me. Note: I am using fedora (linux) and you are using a mac. From a quick google it might be something like this: https://github.com/lovell/sharp/issues/1377#issuecomment-420623747 Basically the node_modules have the wrong OS binaries (buildtime instead of runtime OS).
I just tried the same steps on my MacBook without issue using published assets only. That said, I'm using an intel MacBook - I'll see if I can get access to an M1 to test, that's not a platform I've tested on before so it could be where the difference is.
I was able to replicate the issue using an M1 MacBook, we'll do some more testing to find a fix.
Yes, it does seem to be an OS issue with the device, I just replaced an intel MacBook and followed the same process to deploy lambda to AWS and everything works fine. And thank you for debugging and replies.
For anyone else who comes across this issue. We've narrowed it down to an issue with the way we build the container images using docker. Unless we use BuildKit we're unable to build for multiple platforms, it just builds for the host platform. This means ARM-based machines don't currently build amd64 images with nitric up, causing compatibility issues with AWS Lambda.
We'll support this by moving to BuildKit for nitric builds, but it's a bit of work to make the change so may take time. In the meantime, local runs should be fine and we recommend running deployments through a CI server running an Intel or AMD CPU.
This has been merged into our develop branch of the CLI and will soon be available for release.
