docker
grpc: failed to unmarshal the received message: string field contains invalid UTF-8
After upgrading to buildx version v0.19.2, the following error is reported when building the image
[+] Building 3.2s (5/20) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 829B 0.0s
=> [internal] load metadata for nexus.internal.com/base-image:linux-v2.1.18 0.1s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CANCELED [ 1/16] FROM nexus.internal.com/base-image:linux-v2.1.18@sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b 2.8s
=> => resolve nexus.internal.com/base-image:linux-v2.1.18@sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b 0.1s
=> => sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b 2.85kB / 2.85kB 0.0s
=> => sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc 76.10MB / 76.10MB 1.9s
=> => sha256:777c67696248a1e3d74fb3a1be418d3a393beb8bb362370faa659a282a336006 5.35MB / 5.35MB 0.4s
=> => sha256:d6bfbb7dc215c3a340544c4c87042210e647b140d51a6f210541801fce8274fd 7.78kB / 7.78kB 0.0s
=> => sha256:1ff5115dcffa2e271b232c9dde16c0f7f399773da441c440b497381167582a5f 59.59MB / 142.52MB 2.9s
=> => sha256:11006e65265e13aa0d3f6d56ce11dee5dca992cdcb8a9760431a109aaa79e375 15.73MB / 38.09MB 2.9s
=> => sha256:374999033e8d93a2d24cb2b13d62c8240959ab05c848b8fef1de101a600e7adb 20.97MB / 98.93MB 2.9s
=> => sha256:977e16a77d624fa14338325b88ad9f2aa3059564fa2daac0dce60f8dee15f0dc 2.64MB / 2.64MB 1.1s
=> => sha256:b257af7807da588a476e41081426372189b78526e7c371de2680606c3c02822f 315.64kB / 315.64kB 0.5s
=> => sha256:7e226f4e7f678e9028d1b76c8c0d56d6900058be170ce69baa63ba5567f8f53a 14.68MB / 901.58MB 2.9s
=> => sha256:08be1ee72a529c98177917e0c10bbbc2713519220fad9b3f810b691249a2faa9 193B / 193B 0.5s
=> => sha256:91e577162b472fd8eacc65cf3641be7d736a6d285185d878119e7ae53e4079cd 124B / 124B 0.5s
=> => sha256:8cc69cd3c0537c8a40164bff8c41e624703873f6e31d529d6316ada963d6fa2b 60.82MB / 172.92MB 2.9s
=> => sha256:a941cf7ebf1a67f1c76848a42b49d08eaec0fc9b3d62e4133f2e9d547f1d44f9 15.37MB / 15.37MB 2.4s
=> => extracting sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc 0.9s
=> ERROR [internal] load build context 2.5s
=> => transferring context: 121.49MB 2.4s
------
> [internal] load build context:
------
ERROR: failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
3057857 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/session/filesync.recvDiffCopy
/root/build-deb/engine/vendor/github.com/moby/buildkit/session/filesync/diffcopy.go:104
github.com/moby/buildkit/session/filesync.FSSync
/root/build-deb/engine/vendor/github.com/moby/buildkit/session/filesync/filesync.go:236
github.com/moby/buildkit/source/local.(*localSourceHandler).snapshot
/root/build-deb/engine/vendor/github.com/moby/buildkit/source/local/source.go:264
github.com/moby/buildkit/source/local.(*localSourceHandler).Snapshot
/root/build-deb/engine/vendor/github.com/moby/buildkit/source/local/source.go:153
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).Exec
/root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/ops/source.go:108
github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
/root/build-deb/engine/vendor/github.com/moby/buildkit/solver/jobs.go:1100
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
/root/build-deb/engine/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:122
sync.(*Once).doSlow
/usr/local/go/src/sync/once.go:74
sync.(*Once).Do
/usr/local/go/src/sync/once.go:65
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1695
3066149 v0.19.2 /usr/libexec/docker/cli-plugins/docker-buildx buildx build --debug --network host -t test:test -f ../Dockerfile .
google.golang.org/grpc.(*ClientConn).Invoke
google.golang.org/[email protected]/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
github.com/moby/[email protected]/api/services/control/control_grpc.pb.go:88
github.com/moby/buildkit/client.(*Client).solve.func2
github.com/moby/[email protected]/client/solve.go:269
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
runtime/asm_amd64.s:1695
3057857 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/solver.(*edge).execOp
/root/build-deb/engine/vendor/github.com/moby/buildkit/solver/edge.go:966
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction[...].func2
/root/build-deb/engine/vendor/github.com/moby/buildkit/solver/internal/pipe/pipe.go:78
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1695
3066149 v0.19.2 /usr/libexec/docker/cli-plugins/docker-buildx buildx build --debug --network host -t test:test -f ../Dockerfile .
github.com/moby/buildkit/client.(*Client).solve.func2
github.com/moby/[email protected]/client/solve.go:285
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/[email protected]/errgroup/errgroup.go:78
version info:
# docker buildx version
github.com/docker/buildx v0.19.2 1fc5647
# docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default* docker
\_ default \_ default running v0.17.3 linux/amd64 (+3), linux/386
# docker info
Client: Docker Engine - Community
Version: 27.4.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.19.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.31.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 6
Running: 6
Paused: 0
Stopped: 0
Images: 6
Server Version: 27.4.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
runc version: v1.2.2-0-g7cb3632
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.10.0-26-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.88GiB
........
After upgrading to buildx version v0.19.2,
Upgrading from what version?
Please post reproduction steps.
Hello, we get the same problem since an upgrade this WE. 27.3.1 -> 27.4.1 end result for a build that worked fine before:
#0 building with "default" instance using docker driver
[...]
#6 [application 3/15] RUN [...redacted...]
#6 CACHED
#7 [application internal] load build context
#7 transferring context: 848.24kB done
#7 ERROR: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
#8 [application 4/15] RUN apt-get update
#8 0.334 runc run failed: container process is already dead
#8 CANCELED
------
> [application internal] load build context:
------
failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
version that works with the same build:
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.22
GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
version that FAILS with the same build:
Client: Docker Engine - Community
Version: 27.4.1
API version: 1.47
Go version: go1.22.10
Git commit: b9d17ea
Built: Tue Dec 17 15:45:56 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.4.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.10
Git commit: c710b88
Built: Tue Dec 17 15:45:56 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.24
GitCommit: 88bf19b2105c8b17560993bee28a01ddc2f97182
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb3632
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.19.3
Path: /usr/libexec/docker/cli-plugins/docker-buildx
if we can help pinpoint the problem, please ask
some tests i did:
- changed to base image in the pipeline => same error
- changed the layer just before the problem (by saving a file somewhere), just to try to avoid a layer cache issue
- on the same OS release, same dockerfile, i didn't have the issue. only difference there: i didn't build the same Dockerfile before the os upgrade (i mean with a lower docker version)
- split "docker compose --build" in a separate docker build call, with the --no-cache flag => same issue
so for me, it's only happening while docker compose build is running under the gitlab-ce runner hood, on a debian bookworm host with latest docker-ce packages (as of today)
again, if i can test something else, just ask
My best guess is that you have something in the build context that makes the paths or symlink fields in https://github.com/tonistiigi/fsutil/blob/master/types/stat.proto#L10-L17 contain values that are not strings. Not sure what case would do that but likely that our validation that these are actual strings and not non-printable bytes is more strict now.
My best guess is that you have something in the build context that makes the paths or symlink fields in https://github.com/tonistiigi/fsutil/blob/master/types/stat.proto#L10-L17 contain values that are not strings. Not sure what case would do that but likely that our validation that these are actual strings and not non-printable bytes is more strict now.
After checking, it was found that some file names were indeed not encoded in UTF-8, but in GB2312.
i confirm it's related to a filename encoding. As a workaround we have excluded some folders from the image (.dockerignore) and it builds fine now.
Do you have a practical testcase for this, so we can look if there is a way to mitigate with workaround or improve error.
Any updates?
The same error after update to the latest version for Ubuntu 20.04:
$ docker -v
Docker version 28.1.1, build 4eba377
$ docker build -t test .
ERROR: failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
How to find what files exactly break a building process?
