[CT-940] [CT-933] Support Apple Silicon in Container Images

[curtis-trynow-io]

Describe the feature

Currently, running the container images available from this registry like ghcr.io/dbt-labs/dbt-redshift are very slow compared to running on older Mac computers with Intel chips. When starting the container, a warning is produced by Docker such as:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

and the whole workload has to go through an emulation layer.

As a test inside a container on an M1 Mac:

root@cfeaa3b05bec:/srv# time dbt --help
...
real    0m15.608s
user    0m15.516s
sys     0m0.095s

That's 15 seconds of nothing happening just to dump the help output.

Describe alternatives you've considered

There is nothing reasonable to consider here. Installing dbt via tools like pip or similar takes way too long for reasonable development cycles or onboarding new users when a container image can be launched in seconds. Users currently deal with the sluggish behavior for the benefits of not having to maintain dbt.

Who will this benefit?

Anyone running dbt as a container and on Apple Silicon (M1, presumably M2 and all future chips). The pool of Intel chip users should dwindle with time.

Are you interested in contributing this feature?

Potentially if the build files are available for public inspection and modification.

[jtcohen6]

@curtis-trynow-io I'm going to transfer this to the dbt-core repo, since that's where our Dockerfile lives, along with the actions to build + release our Docker images for each version.

Potentially if the build files are available for public inspection and modification.

They are!

Generic Dockerfile: https://github.com/dbt-labs/dbt-core/blob/main/docker/Dockerfile

I think this bit may be relevant, since our image builder runs on ubuntu-latest only:

https://github.com/dbt-labs/dbt-core/blob/a206cfce657cf1949f40eb208ac3b69a5dd98005/.github/workflows/release-docker.yml#L57-L63

As a larger question, if this would require us building every image on every supported platform, I'm not sure if that's something we'd want to take on as part of our release process. We'd want to discuss that as a team before deciding.

[curtis-trynow-io]

https://github.com/actions/virtual-environments/issues/2187

I actually don't think this is doable right now because of GitHub Actions not supporting M1 runs-on configs outright. The roadmap item looks like it has fallen back as well, unfortunately. I was surprised to see that reading that thread since it is now more than 2.5 years old.

When this becomes doable, can this be reopened for contribution? I would be fine with closing it until then.

Also, if it somehow helps the team discussion as I understand that there may be build impact here, I think most people using the container images are probably Linux or Mac people; Windows users can lean on WSL. If the financial part of it matters, my small team of 3 pays out $150/mo for DBT Cloud, so https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates on a bottom line somewhere, we are funding up to 1875 minutes of macOS build time per month 🙂 . I imagine you have other Apple-shop customers that this would benefit too if they knew about the container images which are new themselves.

I'll leave this open for someone else to close if you you agree this is blocked by GitHub and reopening later makes sense.

[jtcohen6]

@curtis-trynow-io I'm going to close this one for now. Our official policy, for the time being, is that we can provide users with the Dockerfile to build their own images, if they need ones for a specific architecture, but we cannot provide prebuilt images for every architecture: https://docs.getdbt.com/dbt-cli/install/docker#building-your-own-dbt-docker-image