Cannot run the `containerexec` example: Invalid chunk size

[mre]

• Crate version: 0.4.0 from master https://github.com/softprops/shiplift/commit/eb98b1916c0220e44e2d0f3c869c01a2dd037f60
• OS: macOS 10.14.1
• Output of running docker version on the command line:

Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     true

I might well be overlooking something, but I was trying to run the containerexec example from the repository and I get an error message:

/p/t/shiplift ❯❯❯ docker run -d centos:latest sleep infinity
Unable to find image 'centos:latest' locally
latest: Pulling from library/centos
a02a4930cb5d: Pull complete
Digest: sha256:184e5f35598e333bfa7de10d8fb1cebb5ee4df5bc0f970bf2b1e7c7345136426
Status: Downloaded newer image for centos:latest
33ad7bfd9b46f7217fa16f4c8e9070f1cae75267b361159f621f5c9a6607b75c
/p/t/shiplift ❯❯❯ cargo run --example containerexec -- 33ad7bfd9b46f7217fa16f4c8e9070f1cae75267b361159f621f5c9a6607b75c
    Finished dev [unoptimized + debuginfo] target(s) in 0.56s
     Running `target/debug/examples/containerexec 33ad7bfd9b46f7217fa16f4c8e9070f1cae75267b361159f621f5c9a6607b75c`
Error: Docker Error: Docker Error: error reading a body from connection: Invalid chunk size line: Invalid Size

A normal docker exec from the commandline works as expected:

docker exec 33ad7bfd9b46f7217fa16f4c8e9070f1cae75267b361159f621f5c9a6607b75c echo "hello world"
hello world

Any idea why this might be happening? Found some issues regarding chunked encoding in hyper (https://github.com/hyperium/hyper/issues/1146), but I'm not sure how to troubleshoot this. Maybe there's a way to curl the Docker API directly and inspect the headers somehow.

[mre]

Alright I think that's a leftover task from moving to an async API. https://github.com/softprops/shiplift/pull/128#issuecomment-432869752 and https://github.com/softprops/shiplift/blob/master/src/lib.rs#L452 Maybe @abusch has an idea on how to fix that.

[abusch]

Damn, I have no memories of why I put that TODO there :( I wonder if this could be the same kind of issue that was fixed in https://github.com/softprops/shiplift/pull/154 ?

[softprops]

I just pushed a new release with that pr published a new release including this pr https://github.com/softprops/shiplift/blob/master/CHANGELOG.md#050. can you verify if this is still a problem

[keirlawson]

Just encountered this exact same issue running the example with 0.5.0, mac OS 10.14.4, docker version:

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:39 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:13:06 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[tjtelan]

I only encounter this issue on macOS. In Linux, the example runs fine. I'm unfamiliar with how to dig in further.

docker version from macOS host

 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:26:49 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:32:21 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker version from Linux host

 Version:           19.03.1-ce
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        74b1e89e8a
 Built:             Fri Aug 16 14:00:42 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.1-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       74b1e89e8a
  Built:            Fri Aug 16 13:58:02 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.8.m
  GitCommit:        a4bc1d432a2c33aa2eed37f338dceabb93641310.m
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

macOS example

$ uname
Darwin
$ docker run -d centos:latest sleep infinity
Unable to find image 'centos:latest' locally
latest: Pulling from library/centos
d8d02d457314: Pull complete
Digest: sha256:307835c385f656ec2e2fec602cf093224173c51119bbebd602c53c3653a3d6eb
Status: Downloaded newer image for centos:latest
98eb5d732950d33bae45c9765162f700ba4a0810002dd7be1fc33c7322d7d806
$ cargo run --example containerexec -- 98eb5d732950d33bae45c9765162f700ba4a0810002dd7be1fc33c7322d
7d806
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/examples/containerexec 98eb5d732950d33bae45c9765162f700ba4a0810002dd7be1fc33c7322d7d806`
Error: Docker Error: Docker Error: error reading a body from connection: Invalid chunk size line: Invalid Size

Linux example

$ uname
Linux
$ docker run -d centos:latest sleep infinity
Unable to find image 'centos:latest' locally
latest: Pulling from library/centos
d8d02d457314: Pull complete 
Digest: sha256:307835c385f656ec2e2fec602cf093224173c51119bbebd602c53c3653a3d6eb
Status: Downloaded newer image for centos:latest
c37421ba8ce4f0c8656b2345bc84eeca1582541407163de124b794ddcc17a020
$ cargo run --example containerexec -- c37421ba8ce4f0c8656b2345bc84eeca1582541407163de124b794ddcc17a020
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/examples/containerexec c37421ba8ce4f0c8656b2345bc84eeca1582541407163de124b794ddcc17a020`
Stdout: echo VAR=value on stdout
Stderr: echo VAR=value on stderr

[D4nte]

Same issue here, sorry I could not investigate more.

[shadaj]

https://github.com/hyperium/hyper/commit/cb71d2cdbd22e538663e724916dc343430efcf29 appears to fix this issue. The issue before was that Docker on macOS returns the logs as Transfer-Encoding: identity, which hyper wasn't handling properly. Was able to get things running locally by manually applying the diff in the commit to the version of hyper that shiplift uses.

[tjtelan]

• Crate 0.6.0 on master 6cd1d7f93bd6f150341582a1b54087cefffdbf87..4b399f576e219c8b407aaba7049e87f11beac716

Since #229 updated hyper, I thought it was about time to check back in on this issue on both Linux and MacOS.

At least from my manual testing, it appears to be fixed in master.

(However #229 mistakenly changed the eprintln!() stderr output to say stdout, which we'll see in the example output.) https://github.com/softprops/shiplift/blob/4b399f576e219c8b407aaba7049e87f11beac716/examples/containerexec.rs#L34

Linux:

$ uname
Linux
$ docker run -d centos:latest sleep infinity
9c7bcd54ecbba385b35011d29ea9910c712f1278554b8b7a5cbe3cd15591a315
$ cargo run --example containerexec -- 9c7bcd54ecbba385b35011d29ea9910c712f1278554b8b7a5cbe3cd15591a315
     Running `target/debug/examples/containerexec 9c7bcd54ecbba385b35011d29ea9910c712f1278554b8b7a5cbe3cd15591a315`
Stdout: echo VAR=value on stdout
Stdout: echo VAR=value on stderr

MacOS:

$ uname
Darwin
$ docker run -d centos:latest sleep infinity
099c5ec6663289f9d041ad0c890b2ca87b2bbf66001cf18235835308244fc363
$ cargo run --example containerexec -- 099c5ec6663289f9d041ad0c890b2ca87b2bbf66001cf18235835308244fc363
     Running `target/debug/examples/containerexec 099c5ec6663289f9d041ad0c890b2ca87b2bbf66001cf18235835308244fc363`
Stdout: echo VAR=value on stderr
Stdout: echo VAR=value on stdout