yarn
yarn copied to clipboard
yarnpkg
Yarn unbearably slow when ran from inside a Docker container
Do you want to request a feature or report a bug?
Hard to tell if this is a bug or a feature request
What is the current behavior? Currently when Yarn is ran from inside a docker container, the package install process is unbearably slow. After Googling around it doesn't look like there's a working solution for this, and surely having Yarn be usable from dockerized environments is important enough for this to be solved, which is why I've created this ticket.
It's the "linking dependencies" step that's especially slow and I think it has something to do with the yarn cache and/or how docker volumes work.
If the current behavior is a bug, please provide the steps to reproduce.
What is the expected behavior? Some way to run yarn from inside a docker container without it taking ages
Please mention your node.js, yarn and operating system version. Inside docker the versions are: yarn: 1.21.0 node: 12.13.1
If you're destroying the container at the end of your workflow (e.g. install dependencies with yarn, run tests, destroy the container) maybe your dependencies aren't actually being cached?
@rajivshah3 I've tried using a persistent volume for the yarn cache, but it doesn't seem to help. I think it's got something to do with yarn doing a lot ton of writing while linking dependencies which is slowed down because it happens in a volume. Not sure why npm doesn't have this problem tho
docker-sync seems to have helped, although the initial sync is extremely slow. so it looks like volume file-sync may be at fault here.
mutagen works even better, the speed of Yarn inside the container (with the Yarn cache mounted from host as a basic docker volume) is pretty much just as fast as on host
https://blog.rocketinsights.com/speeding-up-docker-development-on-the-mac/ https://mutagen.io/documentation/transports/docker
I'm trying to speed up my deploy process which needs to build a docker image. My dev process is fine, I'm running yarn start outside of docker.
same problem for me, incredibly slow in the node:12.3.1 docker image for example
I found a way, I used node:12.3.1-alpine image and now my yarn builds run 5x times quicker.
On macOS it takes ages to complete the yarn install inside a docker machine 😖
any update so far ? seems like a sync perf problem between docker hub and the container and only resolved via 3rd party sync tools like mentioned above. But man target was not use any third party and manual configuration at all.
I have also mapped volume and used different cache strategies, none of them resolved the problem.
It is a Docker issue, specific to Mac when using volume mounts because of the default consistent writes. Check out https://docs.docker.com/storage/bind-mounts/#choose-the--v-or---mount-flag specifically
The consistency option, if present, may be one of consistent, delegated, or cached. This setting only applies to Docker Desktop for Mac, and is ignored on all other platforms. If you set your volume to use delegated/cached it will yield substantial improvements
https://github.com/yarnpkg/yarn/issues/7747#issuecomment-581311418 is a good solution too, if not better
In your docker file, you can should this line,
RUN yarn install --prefer-offline --frozen-lockfile
I have just installed something using yarn install in my container (a react app) and I had to wait 40 min on a high end mac book pro.
The major pain points are installing new dependencies and using yarn test for jest. It's impossible to handle.
Done in 2444.28s
That's ≈ 40m
In your docker file, you can should this line,
RUN yarn install --prefer-offline --frozen-lockfile
In this example --prefer-offline won't do anything. This is a build step and your yarn packages won't be saved locally unless you are using buildkit and a cache. Something like this works:
ENV YARN_CACHE_FOLDER=/tmp/yarn_cache
RUN --mount=type=cache,target=/tmp/yarn_cache yarn install --prefer-offline --frozen-lockfile
You can test that it works by building once with the steps above then build again with the steps below:
ARG CACHEBUST=1
ENV YARN_CACHE_FOLDER=/tmp/yarn_cache
RUN --mount=type=cache,target=/tmp/yarn_cache yarn install --offline --frozen-lockfile
By running again with the --offline flag you can verify if all your node modules are actually stored in the cache. If they aren't then that step will fail.
This still hasn't been ideal for me, though. Even with setting up the cache properly, this is taking about 90s for only 400MB worth of node module files. To compare, I have a similar cache setup for bundler + cp -r'ing rubygems into a new folder and the time is only 30s for 670MB worth of ruby gem files. When I debug by doing my yarn install with the --verbose flag all the time seems to be from copying the files over. I'm not exactly sure why it's so much slower than just a cp -r.
Still -- I think for anyone that is running into a lot of slowdown because of every node module being pulled down and built from scratch you can instead try using --mount=type=cache and see how that works out.
That being said... it's far from a solution. yarn installs that take tens of seconds locally are taking at least 5 minutes for me during Docker build, as well.
Thanks to this issue, I actually understand that my container didn't crash, it's actually much slower than on host :sweat_smile: I'm new to yarn so I wasn't expecting a startup duration difference to be this big. I'm running this on a Linux distribution so according to previous comments, I doubt the mutagen trick will help.
So I have two questions:
- is there any solution for other OS such as Linux distributions?
- if the problem comes from Docker or another underlying dependency, is there an open issue/tracker about this that we know of? :pray:
If I may @arcanis, would you mind commenting on this issue? (Or perhaps let us know about another maintainer that could give us some visibility on the matter) If we can identify either a way to fix this, or tricks to solve the problem on major OSs, I'm sure the community would be glad to help :smile:
Thanks in advance :pray:
I've moved on from this problem and also to Rancher Desktop, but if anyone else is running into this they should try out the new experimental release of VirtioFS on Docker Desktop and report back with results.
VirtioFS is helping some for me locally, but still pretty slow if you're iterating on a Dockerfile for development purposes.
Can confirm, that enabling VirtioFS does speed up things, although it does still sort of feel a bit slow, but not as bad as without.
Macbook pro sillicon 2022.
- 14.93s outside docker
- 615.05s inside docker (no VirtioFS)
- 131.56s inside docker (with VirtioFS)
Evidently VFS can help a lot, but maybe is just doing a great job at mitigating a bad problem. Besides, I only control VFS configuration on my machine. In deployments, I use a managed build service where VFS isn't my decision (and there shouldn't be any volumes at play).
In fact, I'm now wondering if this is the cause of some prod deployment issues causing builds to run out of memory.
Also worth noting: never observed this issue with npm (using the exact same project setup with same Docker volumes etc).
Hi all,
I got the same issue with docker on WSL 2 (ubuntu 22.04).
I still trying to find the best solution, and one of them is to use Yarn 2 instead of Yarn 1.
With Yarn 2, you'll get a folder : .yarn , and inside you have a cache folder.
2 options :
- As this cache folder takes "only" barely 100Mb, you can add it to your git repo (then all your developers / CI yarn install will be very fast, but your repo will have 100Mb in more)
- You accept that all your developers will have a first build that takes 2 minutes, and then it will be fast as the yarn cache folder will be present (But your CI will still be slow, as the cache folder does not exist)
