[REQUEST] arm64 support

[davidjb]

What would you like to have implemented?

Builds of Chromium for both x86_64 and arm64.

I know this has been discussed a lot previously in #93 and #19 but with the general availability release of provided.al2023 for Lambda (https://aws.amazon.com/blogs/compute/introducing-the-amazon-linux-2023-runtime-for-aws-lambda/), this now seems feasible given previous discussions around blockers.

I wanted to open the discussion around adding this support as some changes would be needed to the build & release processes to handle both architectures. Mostly, I wanted to see what help I could be with that process, to document my findings so far, and contribute PRs/fixes for issues in the compilation process where possible.

Why would it be useful?

arm64-based AWS Lambda function invocation is notably cheaper than x86_64, but also having an arm64 version of Chromium allows for situations such as where emulation of x86_64 isn't possible or problematic on ARM CPUs (e.g. local invocation of functions via AWS SAM).

Other info

I'm currently running the Ansible playbook from this repo against both c6i.16xlarge and c6g.16xlarge instances for x86_64 and arm64 respectively. It's taken over an hour just to clone the Chromium source (at least 44GB of data), but I'll report the issues I'm encountering on either environment below.

Status so far

• How/where is aws.tar.br built and updated?

x86_64 host

• x64 build: successfully on AL2023 with current Ansible playbook

• arm64 build: in progress

  • Requires target_cpu=["x64", "arm"] to be added to .gclient configuration

  • Requires gn to be set up like so gn gen out/Default --args="target_cpu=\"arm64\""

  • autoninja -C out/Default chrome errors:

    • Requires export LDFLAGS="-z max-page-size=0x10000" due to https://github.com/envoyproxy/envoy/issues/17100

ARM64/Gravitron host

Given the below and general notes at https://chromium.googlesource.com/chromium/src/+/main/docs/linux/chromium_arm.md, it appears that cross-compiling for ARM is the only option at present.

• https://bugs.chromium.org/p/chromium/issues/detail?id=1468386

[ec2-user@host chromium]$ gclient sync --delete_unversioned_trees --revision 905e8bdd32d891451d94d1ec71682e989da2b0a1 --with_branch_head
Updating depot_tools...
'/srv/source/depot_tools/.cipd_bin/goma_ctl.py' was not found and may not be supported on this platform!
Syncing projects: 100% (157/157), done.

________ running 'cipd ensure -log-level error -root /srv/source/chromium -ensure-file /tmp/tmpxji4sdpz.ensure' in '.'
Errors:
  failed to resolve infra/rbe/client/linux-arm64@re_client_version:0.113.0.8b45b89-gomaip (line 8): no such package: infra/rbe/client/linux-arm64
Error: Command 'cipd ensure -log-level error -root /srv/source/chromium -ensure-file /tmp/tmpxji4sdpz.ensure' returned non-zero exit status 1
Errors:
  failed to resolve infra/rbe/client/linux-arm64@re_client_version:0.113.0.8b45b89-gomaip (line 8): no such package: infra/rbe/client/linux-arm64

• https://groups.google.com/a/chromium.org/g/chromium-discuss/c/M7MPEeu4kFA

[ec2-user@host chromium]$ gclient runhooks
Updating depot_tools...
'/srv/source/depot_tools/.cipd_bin/goma_ctl.py' was not found and may not be supported on this platform!
Hook 'vpython3 -vpython-spec src/.vpython3 -vpython-tool install' took 26.87 secs
Running hooks:   3% ( 6/170) nacltools
________ running 'python3 src/build/download_nacl_toolchains.py --mode nacl_core_sdk sync --extract' in '/srv/source/chromium'
Traceback (most recent call last):
  File "/srv/source/chromium/src/build/download_nacl_toolchains.py", line 55, in <module>
    sys.exit(Main(sys.argv[1:]))
  File "/srv/source/chromium/src/build/download_nacl_toolchains.py", line 51, in Main
    return package_version.main(args)
  File "/srv/source/chromium/src/native_client/build/package_version/package_version.py", line 1290, in main
    arguments = ParseArgs(args)
  File "/srv/source/chromium/src/native_client/build/package_version/package_version.py", line 1118, in ParseArgs
    host_arch = pynacl.platform.GetArch3264()
  File "/srv/source/chromium/src/native_client/pynacl/platform.py", line 133, in GetArch3264
    assert machine in ARCH3264_DICT, "Unrecognized arch machine: %s" % machine
AssertionError: Unrecognized arch machine: aarch64
Error: Command 'python3 src/build/download_nacl_toolchains.py --mode nacl_core_sdk sync --extract' returned non-zero exit status 1 in /srv/source/chromium
Traceback (most recent call last):
  File "/srv/source/chromium/src/build/download_nacl_toolchains.py", line 55, in <module>
    sys.exit(Main(sys.argv[1:]))
  File "/srv/source/chromium/src/build/download_nacl_toolchains.py", line 51, in Main
    return package_version.main(args)
  File "/srv/source/chromium/src/native_client/build/package_version/package_version.py", line 1290, in main
    arguments = ParseArgs(args)
  File "/srv/source/chromium/src/native_client/build/package_version/package_version.py", line 1118, in ParseArgs
    host_arch = pynacl.platform.GetArch3264()
  File "/srv/source/chromium/src/native_client/pynacl/platform.py", line 133, in GetArch3264
    assert machine in ARCH3264_DICT, "Unrecognized arch machine: %s" % machine
AssertionError: Unrecognized arch machine: aarch64

[joergbaier]

Didn't check what the github actions for this repo are doing, but this could also help in the future https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

[Sparticuz]

A few notes.

• Github Actions costs too much, $15 per compile from what I've heard from others. AWS is much cheaper at a buck or two.
• Yes, I believe that graviton support should be possible now.
  • probably will need to be it's own package (@sparticuz/chromium-arm64 i guess?)
• here is my previous attempt https://github.com/Sparticuz/chrome-aws-lambda/pull/11 much of this can be pulled in
• aws.tar.br is hand generated. I basically booted up a provided.al2 docker container and copied the files out of it. This should be generated, which should probably be it's own PR. With al2023 needing it's own files, aws.tar should probably be renamed to al2.tar
• It will only support node20+

[davidjb]

Thanks for your insight, @Sparticuz. I'll aim to circle back around to looking at this -- I hadencountered issues during the compilation process (described above) and needed to push on with another way forward just for now. Cross-compiling for arm64 is the only option as I've discovered due to a lack of support in Chromium's build processes, but I also encountered issues during the cross-compile on c6i.16xlarge with AL2023.

For now I just wanted to list out the list of packages and libraries that Chromium needs at runtime at least (I'm using the chromium arm64 build from Playwright and that's working well on AL2023). I've got the following building a Layer via SAM, which is running the 'build' in a public.ecr.aws/sam/build-provided.al2023:latest-arm64 container:

dnf install -y \
  alsa-lib \
  at-spi2-atk \
  at-spi2-core \
  atk \
  avahi-libs \
  cairo \
  cups-libs \
  dbus-libs \
  expat \
  freetype \
  fontconfig \
  fribidi \
  google-noto-sans-vf-fonts \
  google-noto-serif-vf-fonts \
  graphite2 \
  gnutls \
  harfbuzz \
  libX11 \
  libXau \
  libXcomposite \
  libXcomposite \
  libXdamage \
  libXext \
  libXfixes \
  libXi \
  libXrandr \
  libXrender \
  libbrotli \
  libdatrie \
  libdrm \
  libgudev \
  libpng \
  libthai \
  libwayland-server \
  libxcb \
  libxkbcommon \
  mesa-libgbm \
  nettle \
  nspr \
  nss \
  nss-util \
  p11-kit-trust \
  pango \
  pixman \
  systemd-libs

mkdir -p "/tmp/lib"
cp -P /usr/lib64/{\
  libasound,\
  libatk-bridge-2.0,\
  libatspi,\
  libatk-1.0,\
  libavahi-client,\
  libavahi-common,\
  libcairo,\
  libcups,\
  libdbus-1,\
  libexpat,\
  libfreetype,\
  libfontconfig,\
  libfribidi,\
  libgraphite2,\
  libgnutls,\
  libharfbuzz,\
  libX11,\
  libXau,\
  libXcomposite,\
  libXdamage,\
  libXext,\
  libXfixes,\
  libXi,\
  libXrandr,\
  libXrender,\
  libbrotlicommon,\
  libbrotlidec,\
  libdatrie,\
  libdrm,\
  libudev,\
  libpng16,\
  libthai,\
  libwayland-server,\
  libxcb,\
  libxcb-render,\
  libxcb-shm,\
  libxkbcommon,\
  libgbm,\
  libnettle,\
  libhogweed,\
  libnspr4,\
  libplc4,\
  libplds4,\
  libnss3,\
  libsmime3,\
  libssl3,\
  libnssutil3,\
  libsoftokn3,\
  libfreeblpriv3,\
  libnssckbi,\
  libpango-1.0,\
  libpixman-1,\
  libsystemd\
  }.* "/tmp/lib"

set -e; \
  cd /tmp/lib; \
  tar -cf al2023.tar *; \
  brotli --best --force al2023.tar

[tuke307]

any updates on this? ARM support would be very nice

[almeidapaulooliveira]

+1

[luishgp]

+1

[anaet-aaron]

+1

[n-lam]

+1

[cjnoname]

+1

[async-lambda]

+1

[JonnyBurger]

I seem to have managed to compile a binary for ARM64 AWS Lambda https://github.com/remotion-dev/chrome-build-instructions/blob/main/V2.md

Hope this is helpful, probably some more work is required to get it working with this project.

[Sparticuz]

I seem to have managed to compile a binary for ARM64 AWS Lambda https://github.com/remotion-dev/chrome-build-instructions/blob/main/V2.md

Hope this is helpful, probably some more work is required to get it working with this project.

Awesome! I am traveling the next few weeks, but will try to look at this asap!!

[ostaplisovyj]

Any updates on this issue? I would really appreciate having Chromium available at least for local sam invocations, without x86_64 emulation.

[stephankaag]

I was able to compile Chromium for AWS Lambda on the ARM64 architecture ~~(only Amazon Linux 2 for now, will build for Amazon Linux 2023 asap).~~

Amazon Linux 2 (NodeJS 16 & NodeJS 18): Download Chromium 120 (For Playwright v1.40.1 , v1.40.0 & Puppeteer v21.8.0) Download Chromium 121 (For Playwright v1.41.2 , v1.41.1 , v1.41.0 & Puppeteer v22.1.0 , v22.0.0 , v21.11.0 , v21.10.0 , v21.9.0)

Amazon Linux 2023 (NodeJS 20): Download Chromium 120 (For Playwright v1.40.1 , v1.40.0 & Puppeteer v21.8.0) Download Chromium 121 (For Playwright v1.41.2 , v1.41.1 , v1.41.0 & Puppeteer v22.1.0 , v22.0.0 , v21.11.0 , v21.10.0 , v21.9.0)

[cjnoname]

any updates on this? :)

[davidizac]

Any updates on this @Sparticuz ?

[LorenzoRogai]

Is there any another block now for building binaries in arm64? ARM64 GitHub runners are now in GA and they cost even less compared to x86