Docker Desktop for Mac file corruption with VirtioFS with heavy disk activity?

[jmichalicek]

• [X] I have tried with the latest version of Docker Desktop
• [X] I have tried disabling enabled experimental features
• [X] I have uploaded Diagnostics
• Diagnostics ID: BC339260-6113-4CA8-8799-8A3C49B49B8B/20230118133135

Expected behavior

I do not believe this to be specific to npm install, it is just easy to replicate using npm. Running npm install or npm ci should be able to successfully install npm packages. This does work in multiple containers when using gRPC FUSE

Actual behavior

When only installing one or two things at a time, installation usually works. ie. npm install webpack, but the more things that are added, the more frequently failures occur which suggests this is related to the amount of disk activity. Every package says that it is corrupted. Because this works when using gRPC FUSE I do not believe it to be an npm issue.

With npm the issue manifests in multiple ways.

npm WARN tarball tarball data for supports-color@https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz (sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for supports-color@https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz (sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==) seems to be corrupted. Trying again.

and

npm ERR! code ENOENT
npm ERR! syscall stat
npm ERR! path /app/.npm/_cacache/content-v2/sha512/95/8e/009e79ac1167a25e2ac0200d1cf520cf6ee081abc218b8354bf0f326d0c6763c4b36c8fbd103f27ae10057b99a7b0982ab0ccfeb98dadcf7f8b728942773
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, stat '/app/.npm/_cacache/content-v2/sha512/95/8e/009e79ac1167a25e2ac0200d1cf520cf6ee081abc218b8354bf0f326d0c6763c4b36c8fbd103f27ae10057b99a7b0982ab0ccfeb98dadcf7f8b728942773'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /app/.npm/_logs/2023-01-18T15_13_37_389Z-debug-0.log

Information

• macOS Version: Ventura 13.0.1
• Intel chip or Apple chip: Apple
• Docker Desktop Version: 4.16.1

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0038: is the connection to Docker working?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?
No fatal errors detected.

Steps to reproduce the behavior

• Build an image with the Dockerfile pasted below. docker build . -t virtiofs-demo
• Add the Package.json below to the current directory
• docker run -ti -v "${pwd}":/app virtiofs-demo bash
• npm install

If it succeeds, rm -rf /app/.npm /app/node_modules and use npm install or npm ci

This is the Dockerfile I used for testing which is a very stripped down version of what I usually work with. I have tested this with multiple base docker images including the latest ubuntu and python 3.11 images. Everything works fine with gRPC FUSE and also worked perfectly using VirtioFS when it was a beta feature.

If you just do a simple npm install webpack, etc. then usually everything works. The more packages in Package.json or specified on the command line, the more failures occur.

# syntax=docker/dockerfile:1.3
ARG PYTHON_VERSION=3.11.0
ARG DISTRO=bullseye
# Currently pinned at <22.0 due to https://github.com/jazzband/pip-tools/issues/1558
# when that is resolve we can just set this to an empty string safely, I think
ARG PIP_VERSION_PIN=<22.4

FROM ubuntu AS base
ENV npm_config_cache=/app/.npm/

RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
  software-properties-common \
  sudo \
  vim \
  telnet \
  apt-transport-https \
  lsb-release \
  git-completion \
  bash-completion \
  less \
  curl \
  && apt-get autoremove && apt-get clean

# See https://nodejs.org/en/about/releases/ for picking node versions
RUN curl -sL https://deb.nodesource.com/setup_18.x | bash
RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
  nodejs \
  && apt-get autoremove && apt-get clean

RUN useradd -ms /bin/bash developer && echo "developer ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

WORKDIR /app
RUN chown developer /app
USER developer

ENV HOME=/home/developer \
  PATH=/home/developer/.local/bin:/app/app/frontend/node_modules/.bin:$PATH \
  LC_ALL=C.UTF-8 \
  LANG=C.UTF-8

Package.json

{
  "dependencies": {
    "@date-io/date-fns": "^2.16.0",
    "@testing-library/react": "^13.4.0",
    "ansi-styles": "^6.2.1",
    "jest": "^29.3.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "webpack": "^5.75.0",
    "webpack-dev-server": "^4.11.1"
  }
}

[jmichalicek]

Just tested on 4.17.0, still super broken :(

[apaniel]

I have the same issue running webpck inside a docker in Mac, I can confirm 4.17.0 is still broken. gRPC FUSE does not have this problem, but it has a poor performance, a weback build with VirtioFS would take 30 seconds, while with gRPC FUSE 400 seconds.

[dy-dx]

@jmichalicek I believe I have the same issue but I cannot reproduce it using your example. I think downgrading Docker Desktop to 4.14.1 may help. Could you see if that works?

[martinml]

I can consistently reproduce in an internal project when npm is configured to store its cache in a VirtioFS shared volume. When switching to gRPC FUSE, the problem disappears.

(...)
npm WARN tarball tarball data for string-width@https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz (sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for string-width@https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz (sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for string-width@https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz (sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for string-width@https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz (sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for string-width@https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz (sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for js-yaml@https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.0.tgz (sha512-wpxZs9NoxZaJESJGIZTyDEaYpl0FKSA+FB9aJiyemKhMwkxQg63h4T1KJgUGHpTqPDNRcmmYLugrRjJlBtWvRA==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for js-yaml@https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.0.tgz (sha512-wpxZs9NoxZaJESJGIZTyDEaYpl0FKSA+FB9aJiyemKhMwkxQg63h4T1KJgUGHpTqPDNRcmmYLugrRjJlBtWvRA==) seems to be corrupted. Trying again.
npm ERR! code ENOENT
npm ERR! syscall rename
npm ERR! path /app/.npm-cache/_cacache/tmp/66556a64
npm ERR! dest /app/.npm-cache/_cacache/content-v2/sha512/b1/1b/b592970ef722ed63104abea7d37a1f4acd91303b7493c97d474fee02683cc2e87a5319884884f2338fd5ee294eca603c2769e87985c3b08f2d50b89cc13c
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, rename '/app/.npm-cache/_cacache/tmp/66556a64' -> '/app/.npm-cache/_cacache/content-v2/sha512/b1/1b/b592970ef722ed63104abea7d37a1f4acd91303b7493c97d474fee02683cc2e87a5319884884f2338fd5ee2964eca603c2769e87985c3b08f2d50b89cc13c'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

• Docker Desktop 4.17.0
• MacBook Pro (13-inch, M1, 2020) + Ventura 13.2.1
• Image node:18.14-alpine (node 18.14.2 + npm 9.5.0)

[whisper0077]

I experienced this bug with Docker Desktop 4.17.0 + VirtioFS, but it did not occur with version 4.18.0 Please try upgrading to version 4.18.0

[jmichalicek]

I missed the activity going on here, but I just tested with Docker Desktop 4.19.0 + VirtioFS and my first run of heavy disk usage worked great.

[cdbennett]

Not sure if this is the same issue you all are talking about, but when I run the Spring Cloud Config Server in a Docker container (using virtiofs), and I modify some config files on the host (which are mounted into the config server container), the container randomly sees corrupted and partial content files for a while. Even waiting a few seconds is not enough for the container to see the correct file content.

Discussed in https://github.com/docker/roadmap/issues/7#issuecomment-1641132358

If I disabled the virtiofs support in Docker Desktop, the problem goes away and things work correctly. (Though many other common operations are slower of course, due to the inefficient file sync needed.)

[fredericdalleau]

@cdbennett if you still see the problem, could you provide a simplified reproduction with more details, in a separate issue please?

[cdbennett]

If I get a chance, I'll try to create a reproducible test case.

[cdbennett]

@fredericdalleau - I tried to write a minimal reproduction case but wasn't able to break it with that. So far only the full Docker Compose for our proprietary application causes the issue with virtiofs (it is 100% reproducible in this case). If I ever am able to make a minimal test case I'll follow up with a new issue.

[nrhope]

I also experienced this problem but instead of npm with a mount in the container to a directory on host mac book built using maven, with VirtioFS and Docker 4.30.0. As VirtioFS is currently the default for fresh Docker installs, maybe the default should change to gRPC FUSE (at least for Mac Intel) until this issue is fixed?

[redochka]

same problem here.

My docker-compose file is:

x-defaults:
  network: &network
    networks:
      - net

services:
  apache:
    container_name: apache
    image: httpd:2.4
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - my-volume-www:/usr/local/apache2/htdocs:delegated
      - my-volume-www-opencart:/var/www/opencart
      - ./apache-fpm/vhosts:/usr/local/apache2/conf/vhosts
    depends_on:
      - php-fpm82
    configs:
      - source: apache-httpd-conf
        target: /usr/local/apache2/conf/httpd.conf
    <<: *network

  php-fpm74:
    container_name: "php74-fpm"
    image: php74-fpm
    ports:
      - "9074:9000"
    volumes:
      - my-volume-www:/usr/local/apache2/htdocs
      - my-volume-www-opencart:/var/www/opencart
    <<: *network

  php-fpm82:
    container_name: "php82-fpm"
    image: php82-fpm
    ports:
      - "9082:80"
    volumes:
      - my-volume-www:/usr/local/apache2/htdocs
      - my-volume-www-opencart:/var/www/opencart
    # not working...
    x-develop:
      watch:
        - action: sync
          path: /Users/reda/docker/data/www/opencart/demo_oc4023
          target: /var/www/opencart/demo_oc4023
    <<: *network

  db:
    container_name: mariadbContainer
    image: mariadb:latest
    ports:
      - "3306:3306"
    environment:
      - MYSQL_USER=xxx
      - MYSQL_PASSWORD=xxx
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
      - MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=yes
    volumes:
      - my-volume-mariadb:/var/lib/mysql
    <<: *network

networks:
  net:

configs:
  apache-vhosts:
    file: apache/vhosts.apache.conf
  apache-httpd-conf:
    file: apache-fpm/httpd.my.conf

volumes:
  my-volume-www:
    external: true
  my-volume-www-opencart:
    external: true
  my-volume-mariadb:
    external: true