docker-node
docker-node copied to clipboard
[buildx/arm64] yarn and npm not able to download packages from internet
Hi there,
thanks for your awesome work with this project. I am experiencing issues using buildx for the arm64 arch. Everything is running smooth until yarn or npm is trying to download packages over the network.
This is an snipped from build-log running on GitHub actions while using yarn:
#22 [linux/arm64 build-deps 5/6] RUN yarn --production
#22 176.3 info There appears to be trouble with your network connection. Retrying...
#22 195.3 error An unexpected error occurred: "https://registry.yarnpkg.com/date-fns/-/date-fns-2.16.1.tgz: ESOCKETTIMEDOUT".
Running the same buildx build locally the output is a bit more concise:
=> [linux/arm64 build-deps 5/6] RUN yarn --production
=> => # Unknown QEMU_IFLA_INFO_KIND ipip
=> => # Unknown QEMU_IFLA_INFO_KIND ip6tnl
=> => # yarn install v1.22.5
=> => # Unknown QEMU_IFLA_INFO_KIND ipip
=> => # Unknown QEMU_IFLA_INFO_KIND ip6tnl
=> => # [1/4] Resolving packages...
=> => # [2/4] Fetching packages...
=> => # Unknown QEMU_IFLA_INFO_KIND ipip
=> => # Unknown QEMU_IFLA_INFO_KIND ip6tnl
=> => # info There appears to be trouble with your network connection. Retrying...
The command used to start the build:
docker buildx build --platform linux/amd64,linux/arm64 -t myrepo/myproject:latest --push client/
My Dockerfile looks like this:
(Note that I do not copy node_modules into build context. The files to copy are managed using a .dockerignore
file.)
# Stage 1 - the build process
FROM node:latest as build-deps
WORKDIR /usr/src/app
ARG REACT_APP_GRAPH_URL
ENV REACT_APP_GRAPH_URL=$REACT_APP_GRAPH_URL
COPY . ./
COPY .env.example .env
RUN yarn --production
RUN yarn build
# Stage 2 - the deployment
FROM nginx:latest
COPY --from=build-deps /usr/src/app/build /usr/share/nginx/html
COPY docker/default /etc/nginx/conf.d/default.conf
Workaround
As a workaround I currently using a .yarnrc to create an offline cache. I copy a it into the build context and install it with the yarn --offline
option. That works for now, but I do not like it very much. https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/
thank you again and best regards
ARe you sure your docker is correctly configured to allow internet access to your containers?
Hi @LaurentGoderre ,
As I wrote, the same holds true for running in GitHub Action environments. I already had this idea, so I verified the issue by running the build on other platforms. Also, in every environment amd64 builds just fine.
What do you think I can further do, to help with this issue?
Thanks and best regards
@PeterDaveHello I noticed you added the yarn label to this issue. I also checked this using npm with the exact same behavior as yarn.
Currently I am trying to run some network diagnostic inside the container. If I stumble upon something useful, I will let you know.
best regards
I think this is fairly reproducible. On my ubuntu machines, any Dockerfile
that includes RUN npm ....
will fail in this same way when either targeting an arm64
image, or running docker buildx build
when the targets include an arm64
target.
I am not able to reproduce however on my macbooks.
Let me guess – you're not building on an actual arm64 runner node which means the arm64 build is generated using QEMU. Emulation is slow, therefore there's much higher risk of running into timeouts.
It may be still possible there's an additional problem with the network connection which may be worth debugging (I do see ESOCKETTIMEDOUT
in the logs) but I'm not sure if this is related to docker-node
in any way.
While this seems plausible, I think it's the common case that projects will build against an emulated environment. Would it not be an improvement to docker-node
to perhaps update documentation in some way to call this out?
I'm having similar problem. I have also started a SO thread
got the same issue on arm instance to build amd64 image with emulator
I had the same problem, but then I found this
docker run --privileged --rm tonistiigi/binfmt --install all
in the official documentation, which solved my problem. Further explanation will be found in the docs itself.
@MunsMan I have a couple questions, it's no problem if you can't get back to me
- In what way did it solve your problem? No more
npm install
or similar timeouts? - How did your GHA yaml file look in terms of order of operations?
- Is the docker/setup-qemu-action@v2 action redundant?
@LiterallyWar Maybe I missed up the Issue, but I had a pretty similar problem. My cross-platform builds for arm always failed while running the npm install
or yarn
command.
Currently, I'm unable to recreate the error.
If I recall correctly, I didn't have any timeout related output. It felt more like a platform emulation issue.
Regarding your second point, what do you mean with GHA? I have to add that I haven't looked into it further, but thought it may help someone.
@MunsMan I was curious if how your .yml action file was organized - i.e. is it qemu -> setup buildx -> build and did you need the docker/setup-qemu-action@v2
action
We got things to work but holy cow it is too slow - $$$
@LiterallyWar I was just able to reproduce my Error:
=> [internal] load build context 0.0s
=> => transferring context: 779B 0.0s
=> [linux/amd64 2/7] WORKDIR /app 0.3s
=> [linux/amd64 3/7] COPY [package.json, package-lock.json, tsconfig.json, ./] 0.0s
=> CANCELED [linux/amd64 4/7] RUN yarn 6.1s
=> [linux/arm64 2/7] WORKDIR /app 0.3s
=> [linux/arm64 3/7] COPY [package.json, package-lock.json, tsconfig.json, ./] 0.0s
=> CANCELED [linux/arm64 4/7] RUN yarn 3.2s
=> [linux/arm/v7 2/7] WORKDIR /app 0.3s
=> [linux/arm/v7 3/7] COPY [package.json, package-lock.json, tsconfig.json, ./] 0.0s
=> ERROR [linux/arm/v7 4/7] RUN yarn 0.1s
------
> [linux/arm/v7 4/7] RUN yarn:
#0 0.128 exec /bin/sh: exec format error
------
Dockerfile:9
--------------------
7 | COPY ["package.json", "package-lock.json", "tsconfig.json", "./"]
8 |
9 | >>> RUN yarn
10 |
11 | COPY ["./src", "./src"]
--------------------
ERROR: failed to solve: process "/bin/sh -c yarn" did not complete successfully: exit code: 1
After running docker run --privileged --rm tonistiigi/binfmt --install all
it runs perfectly.
I'm using docker buildx
for building the images and takes around 2 minutes including publishing the images. I don't use docker/setup-qemu-action@v2
but I don't use GitHub Actions.
Furthermore, I run it locally, and I'm using Colima as the daemon.
awesome thank you so much for your reply and the information - it's the GHA runners, they can't emulate for crap with that action I posted or with your example straight from the docker docs
I really appreciate the help getting to the bottom of this
@LiterallyWar This error can be reproduced with these docker files & GH workflows as well.
https://github.com/frappe/frappe_docker https://github.com/frappe/frappe_docker/actions/runs/4401435320/jobs/7707642818
@Munsman I try it, but it's not resolve problem for me. Have you needed to restart docker.service ?
@HendrixNguyen I'm sorry, sadly I'm not an expert on this topic, I just posted my finding while having the problem. If I recall correctly, I didn't restart the docker.service. I ran the command and reran the build command.
I had the same problem, but then I found this
docker run --privileged --rm tonistiigi/binfmt --install all
in the official documentation, which solved my problem. Further explanation will be found in the docs itself.
- tried without cache
- tried without qemu
- tried with the docker run command mentioned above
- using yarn install
not possible to run on github workflows
edit: Without anything else, this seems to fix it on our side https://github.com/JamesonRGrieve/Agent-LLM-Frontend/pull/14
Meet the same issue on Github Actions Ci:
name: Build and Push Image to Docker hub
on:
push:
branches: ['main']
workflow_dispatch:
jobs:
docker:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v4
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: usr/app:latest
Build and push linux/amd64
image cost about 2-3 mins, but build linux/arm64
cost about more than half an hour and always in the yarn install
..... state.
any workaround stuck here same 🥹🥲
i'm stuck at yarn install too..
same issue only on arm64
#40 48.59 [1/4] Resolving packages...
#40 92.80 [2/4] Fetching packages...
#40 129.2 info There appears to be trouble with your network connection. Retrying...
#40 162.7 info There appears to be trouble with your network connection. Retrying...
#40 174.4 info There appears to be trouble with your network connection. Retrying...
We are also seeing this issue. Pretty annoying, does anyone have any workarounds for github actions or do I just need to find a different CI provider to run this build?
We are also seeing this issue. Pretty annoying, does anyone have any workarounds for github actions or do I just need to find a different CI provider to run this build?
As per the linked PRs, increase your network timeout and be extremely patient if running under buildx/QEMU via GitHub Actions. :-)
Alternatively, I guess attach your own native arm64 runner to GitHub actions, or some third party commercial service which offers arm64 runners and an easy way to bootstrap them into GitHub Actions (BuildJet, Depot.... etc). Or wait for https://github.com/actions/runner-images/issues/5631 :-)
I tried all approaches, even building on an arm64 machine without buildx or QEMU. However, I kept receiving a timeout error.
Config that worked for me to build the arm64 with yarn on GitHub actions:
Add this to your Dockerfile before RUN yarn install
:
RUN yarn config set network-timeout 300000
RUN apk add g++ make py3-pip
RUN yarn global add node-gyp
RUN yarn install
- Increase the network timeout
set network-timeout 300000
- Add additional dependencies
- Install node-gyp
It also stuck in #14 [dependencies 2/5] RUN yarn config set network-timeout 300000
for node:20-alpine
and linux/arm/v7
.
Super weird!
Same here. Less than a minute locally vs. 8 minutes on average running in a Docker build step on GitHub Actions. I confirm we are building an arm64
image with Docker buildx
official action.
Also, no way to get it working without using --network-timeout 100000
.
@mohsenasm I agree with you. Look's like this effects node:20
(lts). I have downgraded my dockerfile to node:18-alpine3.19
and yarn install
now works again.
As per the linked PRs, increase your network timeout and be extremely patient if running under buildx/QEMU via GitHub Actions. :-)
I would have NEVER imagined such a difference in build time on a Xeon V4 server:
amd64
-
npm install
--> added 1285 packages in 2m -
npm build
--> built in 29.61s
arm64 w/ QEMU
-
npm install
--> added 1285 packages in 28m -
npm build
--> built in 21m 59s
This is (or should be) unacceptable for any standards.
One way to approach this is by splitting into build stages, where the npm install and build are done only on native auth and the final image is only created with multi arch --platform
.