[🚀 Feature]: Move "seleniarm" multi-arch images to the same Docker namespace as "selenium" amd64 images

[jamesmortensen]

Feature and motivation

Improving visibility of Multi-Arch Images (also known as docker-seleniarm)

We are seeking information from the community to learn how we can bring more awareness to the existence of the multiple architecture Docker container images, reduce the learning curve for those beginning to use the container images, and enable everyone to get started faster.

One idea we're considering is to move the images in the Docker Hub seleniarm namespace to the official selenium namespace.

In order to get feedback, I want to first provide more insight and transparency on how the container images are built, as well as the challenges we face in building and maintaining them. This gives us a platform for brainstorming ideas by first seeking to understand. First, let's look at the browser images supported by Selenium and Seleniarm (multi-arch). We'll also look at how we source browser binaries, the driver binaries, as well as operating systems.

Namespaces

Currently, we have two namespaces in Docker Hub:

Selenium Images - amd64/Intel only

Namespace	Image Name	Operating System	amd64	arm64	armv7l
selenium	standalone-chrome	Ubuntu 20.04.6 LTS (Focal Fossa)	✓
selenium	node-chrome	Ubuntu 20.04.6 LTS (Focal Fossa)	✓
selenium	standalone-firefox	Ubuntu 20.04.6 LTS (Focal Fossa)	✓
selenium	node-firefox	Ubuntu 20.04.6 LTS (Focal Fossa)	✓
selenium	standalone-edge	Ubuntu 20.04.6 LTS (Focal Fossa)	✓
selenium	node-edge	Ubuntu 20.04.6 LTS (Focal Fossa)	✓

Seleniarm Container Images - Multiple Architectures (amd64 / arm64 / armv7l)

Namespace	Image Name	Operating System	amd64	arm64	armv7l
seleniarm	standalone-chromium	Debian Sid (unstable)	✓	✓	✓
seleniarm	node-chromium	Debian Sid (unstable)	✓	✓	✓
seleniarm	standalone-firefox	Debian Sid (unstable)	✓	✓	✓
seleniarm	node-firefox	Debian Sid (unstable)	✓	✓	✓
seleniarm	standalone-edge	N/A	N/A	N/A	N/A
seleniarm	node-edge	N/A	N/A	N/A	N/A
seleniarm	standalone-chrome	N/A	N/A	N/A	N/A
seleniarm	node-chrome	N/A	N/A	N/A	N/A

NOTE: Google and Microsoft do not provide binaries for their browsers on ARM. Thus, we cannot provide multi-arch images for Chrome and Edge.

Browser Binaries

Let's take a moment to look at the browser binaries which are available for various architectures:

Google Chrome

Google currently does not build binaries for Chrome on Linux for ARM. Chrome is only available for Intel/x86_64 and amd64 architectures. For multi-arch images, we substitute Google Chrome with the upstream open source Chromium browser. While they share a lot of the same code and browser engine, there are slight, subtle differences in how the browsers function.

Mozilla Firefox

Although Mozilla doesn't provide official binaries for ARM on Linux, the Debian community maintains binaries for various architectures, which are available on Debian Sid (the unstable Debian). We build and maintain container images for amd64, arm64, and armv7l. Here is where binaries for Mozilla Firefox are sourced from when building container images:

Namespace	Binary Source
selenium	Mozilla Official Downloads
seleniarm	Debian Sid Snapshots (community maintained)

Chromium

Since Google does not provide binaries for ARM, and since Google Chrome is not open source (only Chromium is), the next best solution to providing a Selenium browser automation solution on ARM is to use Chromium binaries. The Debian community maintains the chromium package and keeps it up to date.

While Chromium and Google Chrome are similar and share the same browser engines, at the end of the day, they're not the same browsers. This may be fine in some use cases and problematic in others.

Browser	Binary Source
Google Chrome (amd64)	Google Linux Software Repositories
Chromium (multi-arch)	Debian Sid Snapshots (community maintained)

Edge

Currently, Microsoft does not publish binaries for ARM. Instead, open source Chromium is our next best solution.

Browser Drivers

Locating up to date browser drivers for ARM, such as chromedriver and geckodriver, is also a challenge. Up until October 2022, I maintained geckodriver binaries for arm64 until Mozilla stepped forward and took responsibility for building and supporting arm64 geckodriver binaries. At this time, I still maintain an unofficial binary for geckodriver on armv7l so that developers and testers using Raspberry Pi devices can run their Selenium Grids and run automation scripts that drive Mozilla Firefox.

Browser Driver	Binary Source
chromedriver (amd64)	Official Chromium project
chromedriver (amd64/arm64/armv7l)	Debian Sid Snapshots (community maintained)
geckodriver (amd64/arm64)	Mozilla Releases
geckodriver (armv7l)	geckodriver-arm-binaries unofficial binaries

Operating Systems

Let's also look at how the base image for docker-selenium and docker-seleniarm images differ. Selenium images are built on top of a stable Ubuntu 20.04 base image, while Seleniarm images are built on Debian Sid. Debian Sid is the unstable version of the Debian operating system. It's one of the only distributions I could find that offered both of the latest binaries.

NOTE: I may revisit this. I discovered that Chromium appears to be frozen at v85 on Ubuntu and that the latest is actually installed. They have opened a bug report to rectify this misleading information.

Although Ubuntu is based on Debian, these are different distributions of Linux. Moreover, Debian Sid is considered to be on the bleeding edge while Ubuntu 20.04 is a LTS (long term stable release) and is of course quite stable.

All of this information leads us into the next question: How do we name the images so that it's clear which flavor of the container images you're using?

How the Docker Images are used

Currently, I suggest teams who exclusively use Intel based machines use images from the "selenium" namespace due to the stability of the container images. For teams using ARM hardware, the best choice is to use images from the "seleniarm" namespace.

But one question that occasionally pops up is how teams should handle scenarios where half the team uses Intel machines and the other half uses ARM? There are two ways I've seen teams handle this:

Everyone uses seleniarm

Since the docker-seleniarm images are also compatible with Intel machines, some teams choose to have everyone use the seleniarm images. The advantage is that this ensures a single docker-compose.yml file works on all architectures. However, one potential disadvantage is it also means no one is automating Google Chrome, only Chromium. Maybe that's okay, and maybe it's not. It depends.

Intel users use selenium, M1/ARM users use seleniarm

The other option I've suggested is to detect the architecture of the machine and then load the best image for that architecture. Here's an example I've taken from an article I wrote on combining multi-arch Docker images from different sources:

// if OS is not Apple M1, then assume Intel and use selenium; otherwise, use seleniarm.
const image = !os.cpus()[0].model.includes('Apple M1')
            ? 'selenium/standalone-chrome:latest'
            : 'seleniarm/standalone-chromium:latest');

If the developer or tester uses an Intel machine, they'll pull selenium/standalone-chrome, with Ubuntu, Google Chrome, and stability. If they're using an M1, then they'll pull seleniarm/standalone-chromium, which includes Debian Sid, open source Chromium, and may involve some instability.

Moving Seleniarm to Selenium

To move Seleniarm container images to Selenium, there's a few things we need to take into consideration:

• The image names, standalone-firefox and node-firefox, are the same for both namespaces. We would either need to use different names, or there would be scenarios where one release is a mutli-arch release on Debian and another release is an amd64 release on Ubuntu.
• standalone-chome, node-chrome, and standalone-chromium and node-chromium would still be amd64-only and multi-arch respectively.
• The beta and dev tags would still reflect amd64-only, since Chromium and Mozilla do not have beta or dev channels for ARM.
• The Selenium container images are built using GitHub Actions, while the multi-arch Seleniarm images are built on CircleCI, using both arm64 and amd64 runners. The maintenance is still likely to be the same when it comes to syncing the community docker-seleniarm repo with the upstream repo (this repo).

We'd like to get feedback from the community to brainstorm some ideas for what to do. Please comment below with any suggestions on how we can proceed forward.

Usage example

The problem we're looking to solve is to make it easier to maintain multi-arch images as well as easy for the community to find and use them.

[github-actions[bot]]

@jamesmortensen, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[diemol]

Thank you for this detailed write up, @jamesmortensen!

A few doubts, comments:

The image names, standalone-firefox and node-firefox, are the same for both namespaces. We would either need to use different names, or there would be scenarios where one release is a mutli-arch release on Debian and another release is an amd64 release on Ubuntu.

An way to go could be to keep the same name but then users need to specify the platform image they want to pull, properly documented. Would that make sense?

The Selenium container images are built using GitHub Actions, while the multi-arch Seleniarm images are built on CircleCI, using both arm64 and amd64 runners. The maintenance is still likely to be the same when it comes to syncing the community docker-seleniarm repo with the upstream repo (this repo).

Can you please explain again why we need to use CircleCi? Can't I build an arm64 image on a amd64 host?

[jamesmortensen]

An way to go could be to keep the same name but then users need to specify the platform image they want to pull, properly documented. Would that make sense?

Let's say we release Firefox 112.0 on May 17th for amd64, and then on June 12th Firefox 112.0 becomes available for ARM. Currently, we tag images with the browser and driver versions. We need to figure out how would the browser tagging work. selenium/standalone-firefox:112.0 would have already been published on May 17th, so we can't really reuse the same tag. Also, the version of Selenium on May 17th and on June 12th may also be different. We would need to think through whether or not browser tagging makes sense on the images since not every image would have the same version of Firefox. Eliminating browser and driver tags would simplify the process.

We can see an example with the experiment I did where I built the image from both dockerfiles. It shows Firefox 112.0.1 for amd64 and 112.0 for arm/v7 on the same tag. It's a slight difference in this case, but it shows how there's a discrepancy:

$ docker run --rm -it --shm-size 2g --platform linux/amd64 jamesmortensen1/standalone-firefox:imagetools bash -c "firefox -v"
Mozilla Firefox 112.0.1

$ docker run --rm -it --shm-size 2g --platform linux/arm/v7 jamesmortensen1/standalone-firefox:imagetools bash -c "firefox -v"
Mozilla Firefox 112.0

Additionally, the geckodriver versions are different:

$ docker run --rm -it --shm-size 2g --platform linux/arm/v7 jamesmortensen1/standalone-firefox:imagetools bash -c "geckodriver -V"
geckodriver 0.32.2 ( 2023-02-08)

$ docker run --rm -it --shm-size 2g --platform linux/amd64 jamesmortensen1/standalone-firefox:imagetools bash -c "geckodriver -V"
geckodriver 0.33.0 (a80e5fd61076 2023-04-02 18:31 +0000)

We could convey this in the release notes, but I don't know how it would work if we attempted to tag the images with the browser and driver versions. Maybe we could outline what the release notes could look like in the case where the browser versions are different.

[jamesmortensen]

Can you please explain again why we need to use CircleCi? Can't I build an arm64 image on a amd64 host?

It's possible to build multi-arch images in GitHub Actions amd64 runners, but QEMU emulation is incredibly slow and sluggish. This creates two problems:

We can't test browsers under emulation

It's not possible to test the images because the browsers won't startup under emulation. On CircleCI, since there's both arm64 and amd64 runners natively, I can test browsers on both of those platforms.

I cannot yet test armv7l since no cloud provider has this architecture as an option. I tried downloading an armv7 VM and running in QEMU, but that didn't really work in CI

The build process is painfully slow under QEMU emulation

Building the Chromium and Firefox images from Base to Standalone takes 4 to 6 minutes to build natively, but it can take up to 20 to 40 minutes to build under emulation. The last full build of all images was the fastest ever at 17 minutes, and that was because of splitting the build process between multiple runners and executing in parallel.

If we could use a self-hosted ARM64 runner, perhaps one of Oracle's free ARM Ampere virtual machines, we could do it on GitHub Actions, but there's no guarantee Oracle will keep that as a free solution.

[jamesmortensen]

Maybe something like this:

Release Notes

selenium/standalone-firefox:4.9.1-20230517

amd64:

• Mozilla Firefox v112.0.1
• geckodriver v0.33.0
• Selenium v4.9.1

arm64:

• Mozilla Firefox v112.0
• geckodriver v0.32.2
• Selenium v4.9.1

arm/v7:

• Mozilla Firefox v112.0
• geckodriver v0.32.2
• Selenium v4.9.1

selenium/standalone-chrome:4.9.1-20230517

amd64:

• Google Chrome v113.0
• chromedriver v113.0
• Selenium v4.9.1

selenium/standalone-chromium:4.9.1-20230517

arm64:

• Chromium v113.0
• chromedriver v113.0
• Selenium v4.9.1

arm/v7:

• Chromium v113.0
• chromedriver v113.0
• Selenium v4.9.1

amd64:

• Chromium v113.0
• chromedriver v113.0
• Selenium v4.9.1

selenium/standalone-edge:4.9.1-20230517

amd64:

• Microsoft Edge v113.0
• edgedriver v113.0
• Selenium v4.9.1

[jamesmortensen]

@diemol I'm thinking of doing an experiment where I mirror all of the releases in a test namespace, in forked repos, so we can see how combined images play out in isolation. Let's iterate on this idea a little more and tease out some more questions that we may need to tackle. Please let me know if the information in the previous comments is helpful. If you have more questions, please keep them coming! :)

[jamesmortensen]

Sorry for the high volume of comments, but I discovered that the latest browser and driver binaries are indeed available on Ubuntu arm64. I just tried in an Ubuntu 20.04.6 arm64 VM to install both browser images and drivers, and it seems I got the latest versions of everything we need, except Google Chrome.

Here's the terminal output from the VM:

james@ubuntu-one:~$ arch
aarch64
james@ubuntu-one:~$ snap search chromium
Name                      Version        Publisher     Notes  Summary
chromium                  113.0.5672.63  canonical✓    -      Chromium web browser, open-source version of Chrome
boxy-svg                  3.96.0         jarek-foksa   -      Scalable Vector Graphics (SVG) editor
nativefier                50.1.1         kz6fittycent  -      Nativefier is a command-line tool to easily create a “desktop app”
restart-chromium          1              mgibbs-dfrs   -      Restart Browser
dashkiosk-client-browser  0.1            ogra          -      Chromium in Kiosk mode specifically adjusted for dashkiosk
james@ubuntu-one:~$ snap install chromium
chromium 113.0.5672.63 from Canonical✓ installed
james@ubuntu-one:~$ chromium --version
Chromium 113.0.5672.63 snap
james@ubuntu-one:~$ chromedriver -v

Command 'chromedriver' not found, but can be installed with:

sudo apt install chromium-chromedriver

james@ubuntu-one:~$ sudo apt install chromium-chromedriver
[sudo] password for james: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  chromium-browser
The following NEW packages will be installed:
  chromium-browser chromium-chromedriver
0 upgraded, 2 newly installed, 0 to remove and 31 not upgraded.
Need to get 51.0 kB of archives.
After this operation, 243 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://in.ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 chromium-browser arm64 1:85.0.4183.83-0ubuntu0.20.04.3 [48.5 kB]
Get:2 http://in.ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 chromium-chromedriver arm64 1:85.0.4183.83-0ubuntu0.20.04.3 [2,496 B]
Fetched 51.0 kB in 2s (23.9 kB/s)                 
Preconfiguring packages ...
Selecting previously unselected package chromium-browser.
(Reading database ... 146656 files and directories currently installed.)
Preparing to unpack .../chromium-browser_1%3a85.0.4183.83-0ubuntu0.20.04.3_arm64
.deb ...
=> Installing the chromium snap
==> Checking connectivity with the snap store
==> Installing the chromium snap
snap "chromium" is already installed, see 'snap help refresh'
=> Snap installation complete
Unpacking chromium-browser (1:85.0.4183.83-0ubuntu0.20.04.3) ...
Selecting previously unselected package chromium-chromedriver.
Preparing to unpack .../chromium-chromedriver_1%3a85.0.4183.83-0ubuntu0.20.04.3_
arm64.deb ...
Unpacking chromium-chromedriver (1:85.0.4183.83-0ubuntu0.20.04.3) ...
Setting up chromium-browser (1:85.0.4183.83-0ubuntu0.20.04.3) ...
Setting up chromium-chromedriver (1:85.0.4183.83-0ubuntu0.20.04.3) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
james@ubuntu-one:~$ chromedriver -v
ChromeDriver 113.0.5672.63 (0e1a4471d5ae5bf128b1bd8f4d627c8cbd55f70c-refs/branch-heads/5672@{#912})
james@ubuntu-one:~$ firefox -v
Mozilla Firefox 113.0.1
james@ubuntu-one:~$ geckodriver -v

Command 'geckodriver' not found, but can be installed with:

sudo apt install firefox-geckodriver

james@ubuntu-one:~$ sudo apt-get install firefox-geckodriver
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  firefox-geckodriver
0 upgraded, 1 newly installed, 0 to remove and 31 not upgraded.
Need to get 1,108 kB of archives.
After this operation, 3,694 kB of additional disk space will be used.
Get:1 http://in.ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 firefox-geckodriver arm64 113.0.1+build1-0ubuntu0.20.04.1 [1,108 kB]
Fetched 1,108 kB in 3s (367 kB/s)                      
Selecting previously unselected package firefox-geckodriver.
(Reading database ... 146677 files and directories currently installed.)
Preparing to unpack .../firefox-geckodriver_113.0.1+build1-0ubuntu0.20.04.1_arm64.deb ...
Unpacking firefox-geckodriver (113.0.1+build1-0ubuntu0.20.04.1) ...
Setting up firefox-geckodriver (113.0.1+build1-0ubuntu0.20.04.1) ...
james@ubuntu-one:~$ geckodriver -v
1684346901934	geckodriver	INFO	Listening on 127.0.0.1:4444
^C
james@ubuntu-one:~$ geckodriver -V
geckodriver 0.33.0

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.
james@ubuntu-one:~$ file /usr/bin/geckodriver
/usr/bin/geckodriver: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=9852c8fc4cad9cd95abd048b40f1cd814b0a39ff, for GNU/Linux 3.7.0, stripped
james@ubuntu-one:~$

I also verified, using test scripts from https://github.com/jamesmortensen/debug-tools-for-docker-selenium/tree/main/selenium-webdriver-demo-javascript that both browsers can open and be automated.

In the interests of being thorough, I should try the same experiment on an armv7l VM to make sure everything is available there too. I do know geckodriver won't be there, but I'm already building that myself. The important thing is that Chromium, Firefox, and chromedriver be there. Hope this information is helpful!

[diemol]

We could convey this in the release notes, but I don't know how it would work if we attempted to tag the images with the browser and driver versions. Maybe we could outline what the release notes could look like in the case where the browser versions are different.

I think this is good enough.

It's possible to build multi-arch images in GitHub Actions amd64 runners, but QEMU emulation is incredibly slow and sluggish. This creates two problems:

Then maybe we can move completely to Circle CI? Just the build process, and keep the release logic in GitHub Actions.

Sorry for the high volume of comments, but I discovered that the latest browser and driver binaries are indeed available on Ubuntu arm64. I just tried in an Ubuntu 20.04.6 arm64 VM to install both browser images and drivers, and it seems I got the latest versions of everything we need, except Google Chrome.

In the interests of being thorough, I should try the same experiment on an armv7l VM to make sure everything is available there too. I do know geckodriver won't be there, but I'm already building that myself. The important thing is that Chromium, Firefox, and chromedriver be there. Hope this information is helpful!

Nice 🎉