sudo rkt fetch does not work

[dulltz]

Issue Report

We use Container Linux 2135.4.0, then notice that sudo rkt fetch does not work in some cases.

Bug

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2135.4.0
VERSION_ID=2135.4.0
BUILD_ID=2019-06-24-2257
PRETTY_NAME="Container Linux by CoreOS 2135.4.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Windows10 Hyper-V

Expected Behavior

We can fetch a container image quay.io/cybozu/chrony:3.3 by sudo rkt fetch.

$ sudo rkt fetch quay.io/cybozu/chrony:3.3
pubkey: prefix: "quay.io/cybozu/chrony"
key: "https://quay.io/aci-signing-key"
gpg key fingerprint is: BFF3 13CD AA56 0B16 A898  7B8F 72AB F5F6 799D 33BC
        Quay.io ACI Converter (ACI conversion signing key) <[email protected]>
Are you sure you want to trust this key (yes/no)?
yes
Trusting "https://quay.io/aci-signing-key" for prefix "quay.io/cybozu/chrony" after fingerprint review.
Added key for prefix "quay.io/cybozu/chrony" at "/etc/rkt/trustedkeys/prefix.d/quay.io/cybozu/chrony/bff313cdaa560b16a8987b8f72abf5f6799d33bc"
Downloading signature: [=======================================] 473 B/473 B
sha512-4327b4a010bd581a1a8b02fdd9d18935

Actual Behavior

We exec sudo rkt fetch quay.io/cybozu/chrony:3.3 and wait, but never complete it.

$ sudo rkt fetch quay.io/cybozu/chrony:3.3
pubkey: prefix: "quay.io/cybozu/chrony"
key: "https://quay.io/aci-signing-key"
gpg key fingerprint is: BFF3 13CD AA56 0B16 A898  7B8F 72AB F5F6 799D 33BC
        Quay.io ACI Converter (ACI conversion signing key) <[email protected]>
Are you sure you want to trust this key (yes/no)?
yes
Trusting "https://quay.io/aci-signing-key" for prefix "quay.io/cybozu/chrony" after fingerprint review.
Added key for prefix "quay.io/cybozu/chrony" at "/etc/rkt/trustedkeys/prefix.d/quay.io/cybozu/chrony/bff313cdaa560b16a8987b8f72abf5f6799d33bc"
Downloading signature: [=======================================] 473 B/473 B

Reproduction Steps

1. Exec sudo rkt fetch quay.io/cybozu/chrony:3.3 on Container Linux 2135.4.0

Other Information

sudo rkt fetch quay.io/cybozu/chrony:3.3 succeed in Container Linux 2079.0.4

core@localhost ~ $ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2079.4.0
VERSION_ID=2079.4.0
BUILD_ID=2019-05-15-0808
PRETTY_NAME="Container Linux by CoreOS 2079.4.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

[lucab]

Thanks for the report. It looks like either some regression by a new Go toolchain, or a network issue. On the same machine, can you please try the pre-built upstream binary and check if that one works?

[dulltz]

@lucab We tried the upstream rkt binary, and it works.

core@localhost ~/rkt-v1.30.0 $ sudo ./rkt fetch quay.io/cybozu/chrony:3.3
pubkey: prefix: "quay.io/cybozu/chrony"
key: "https://quay.io/aci-signing-key"
gpg key fingerprint is: BFF3 13CD AA56 0B16 A898  7B8F 72AB F5F6 799D 33BC
        Quay.io ACI Converter (ACI conversion signing key) <[email protected]>
Are you sure you want to trust this key (yes/no)?
yes
Trusting "https://quay.io/aci-signing-key" for prefix "quay.io/cybozu/chrony" after fingerprint review.
Added key for prefix "quay.io/cybozu/chrony" at "/etc/rkt/trustedkeys/prefix.d/quay.io/cybozu/chrony/bff313cdaa560b16a8987b8f72abf5f6799d33bc"
Downloading signature: [=======================================] 473 B/473 B
Downloading ACI: [=============================================] 43 MB/43 MB
image: signature verified:
  Quay.io ACI Converter (ACI conversion signing key) <[email protected]>
sha512-4327b4a010bd581a1a8b02fdd9d18935
core@localhost ~/rkt-v1.30.0 $ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2135.4.0
VERSION_ID=2135.4.0
BUILD_ID=2019-06-24-2257
PRETTY_NAME="Container Linux by CoreOS 2135.4.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
core@localhost ~/rkt-v1.30.0 $ ./rkt version
rkt Version: 1.30.0
appc Version: 0.8.11
Go Version: go1.8.3
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL

[dulltz]

And after sudo ./rkt fetch quay.io/cybozu/chrony:3.3 succeeded, sudo rkt fetch quay.io/cybozu/chrony:3.3 also succeeded.

[lucab]

Good to know, that seems to confirm my previous comment. For reference, the second fetch with the host-binary is almost a no-op, as the image is already fetched and cached locally at that point (but still good point that it doesn't hang that way).