Docker Desktop for Mac file corruption with VirtioFS with heavy disk activity?

[jmichalicek]

• [X] I have tried with the latest version of Docker Desktop
• [X] I have tried disabling enabled experimental features
• [X] I have uploaded Diagnostics
• Diagnostics ID: BC339260-6113-4CA8-8799-8A3C49B49B8B/20230118133135

Expected behavior

I do not believe this to be specific to npm install, it is just easy to replicate using npm. Running npm install or npm ci should be able to successfully install npm packages. This does work in multiple containers when using gRPC FUSE

Actual behavior

When only installing one or two things at a time, installation usually works. ie. npm install webpack, but the more things that are added, the more frequently failures occur which suggests this is related to the amount of disk activity. Every package says that it is corrupted. Because this works when using gRPC FUSE I do not believe it to be an npm issue.

With npm the issue manifests in multiple ways.

npm WARN tarball tarball data for supports-color@https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz (sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for supports-color@https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz (sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==) seems to be corrupted. Trying again.

and

npm ERR! code ENOENT
npm ERR! syscall stat
npm ERR! path /app/.npm/_cacache/content-v2/sha512/95/8e/009e79ac1167a25e2ac0200d1cf520cf6ee081abc218b8354bf0f326d0c6763c4b36c8fbd103f27ae10057b99a7b0982ab0ccfeb98dadcf7f8b728942773
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, stat '/app/.npm/_cacache/content-v2/sha512/95/8e/009e79ac1167a25e2ac0200d1cf520cf6ee081abc218b8354bf0f326d0c6763c4b36c8fbd103f27ae10057b99a7b0982ab0ccfeb98dadcf7f8b728942773'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /app/.npm/_logs/2023-01-18T15_13_37_389Z-debug-0.log

Information

• macOS Version: Ventura 13.0.1
• Intel chip or Apple chip: Apple
• Docker Desktop Version: 4.16.1

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0038: is the connection to Docker working?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?
No fatal errors detected.

Steps to reproduce the behavior

• Build an image with the Dockerfile pasted below. docker build . -t virtiofs-demo
• Add the Package.json below to the current directory
• docker run -ti -v "${pwd}":/app virtiofs-demo bash
• npm install

If it succeeds, rm -rf /app/.npm /app/node_modules and use npm install or npm ci

This is the Dockerfile I used for testing which is a very stripped down version of what I usually work with. I have tested this with multiple base docker images including the latest ubuntu and python 3.11 images. Everything works fine with gRPC FUSE and also worked perfectly using VirtioFS when it was a beta feature.

If you just do a simple npm install webpack, etc. then usually everything works. The more packages in Package.json or specified on the command line, the more failures occur.

# syntax=docker/dockerfile:1.3
ARG PYTHON_VERSION=3.11.0
ARG DISTRO=bullseye
# Currently pinned at <22.0 due to https://github.com/jazzband/pip-tools/issues/1558
# when that is resolve we can just set this to an empty string safely, I think
ARG PIP_VERSION_PIN=<22.4

FROM ubuntu AS base
ENV npm_config_cache=/app/.npm/

RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
  software-properties-common \
  sudo \
  vim \
  telnet \
  apt-transport-https \
  lsb-release \
  git-completion \
  bash-completion \
  less \
  curl \
  && apt-get autoremove && apt-get clean

# See https://nodejs.org/en/about/releases/ for picking node versions
RUN curl -sL https://deb.nodesource.com/setup_18.x | bash
RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
  nodejs \
  && apt-get autoremove && apt-get clean

RUN useradd -ms /bin/bash developer && echo "developer ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

WORKDIR /app
RUN chown developer /app
USER developer

ENV HOME=/home/developer \
  PATH=/home/developer/.local/bin:/app/app/frontend/node_modules/.bin:$PATH \
  LC_ALL=C.UTF-8 \
  LANG=C.UTF-8

Package.json

{
  "dependencies": {
    "@date-io/date-fns": "^2.16.0",
    "@testing-library/react": "^13.4.0",
    "ansi-styles": "^6.2.1",
    "jest": "^29.3.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "webpack": "^5.75.0",
    "webpack-dev-server": "^4.11.1"
  }
}