INTERNAL_ERROR building image with Kaniko

[coryan]

Error in Kaniko step:

INFO[0047] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: stream error: stream ID 95; INTERNAL_ERROR

Looks like 3 builds affected last night:

https://pantheon.corp.google.com/cloud-build/builds;region=global/47becb5f-2556-41d4-8b67-e0ee160a68f4;step=0?project=cloud-cpp-testing-resources https://pantheon.corp.google.com/cloud-build/builds/af84399a-58de-4e62-a2fb-84991c1fd9d2;step=0?project=cloud-cpp-testing-resources https://pantheon.corp.google.com/cloud-build/builds;region=global/cbb56412-49cd-4b9d-b3f3-610b65b19d9e;step=0?project=cloud-cpp-testing-resources

[coryan]

https://pantheon.corp.google.com/cloud-build/builds/07dea1f5-b889-43c0-84e0-0e84f9fd9c9f;step=0?project=cloud-cpp-testing-resources

Step #0: INFO[0040] No files changed in this command, skipping snapshotting. 
Step #0: INFO[0040] RUN /var/tmp/ci/install-cloud-sdk.sh         
Step #0: INFO[0040] Found cached layer, extracting to filesystem 
Step #0: error building image: error building stage: failed to execute command: extracting fs from image: stream error: stream ID 93; INTERNAL_ERROR
Finished Step #0

[devbww]

https://console.cloud.google.com/cloud-build/builds;region=us-east1/42a7f779-924d-4533-bb50-e4350f41fbe7?project=cloud-cpp-testing-resources

Step #0: time="2021-07-21T21:25:21Z" level=info msg="RUN /var/tmp/ci/install-cloud-sdk.sh"
Step #0: time="2021-07-21T21:25:21Z" level=info msg="Found cached layer, extracting to filesystem"
Step #0: error building image: error building stage: failed to execute command: extracting fs from image: stream error: stream ID 3; INTERNAL_ERROR
Finished Step #0
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:v1.6.0-debug" failed: step exited with non-zero status: 1

[coryan]

https://console.cloud.google.com/cloud-build/builds;region=us-east1/66c3d69a-7c8e-441d-8150-1f18fa8ab38d?project=936212892354

and

https://console.cloud.google.com/cloud-build/builds;region=us-east1/02a28e28-7899-4cdd-9730-442b3b3cf98c?project=936212892354

[dbolduc]

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/69a2d3d6-8356-47a7-ab0d-f80081c1e460?project=cloud-cpp-testing-resources

[coryan]

kaniko:1.7.0 is out, it supports retries when extracting the filesystem:

https://github.com/GoogleContainerTools/kaniko/pull/1685 https://github.com/GoogleContainerTools/kaniko/releases/tag/v1.7.0

A simple upgrade did not work. Seems like some configuration is needed.

[dbolduc]

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/7770/79662afb6d35baabcd7645103517b0b3aaa068d9/fedora-34-ubsan/log-9df4d9fb-dff0-4d1d-aef1-2eb1189cd722.txt https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/7770/79662afb6d35baabcd7645103517b0b3aaa068d9/fedora-34-coverage/log-a200cc61-c10c-466e-8003-37d322249c27.txt

[dbolduc]

(internal link) https://console.cloud.google.com/cloud-build/builds;region=us-east1/b8452f2f-03c9-4dc7-b147-525d759feed6?project=936212892354

[dbolduc]

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/8393/c2c30f3ea1fa2eff0d0b4a61febfd3ee449c6448/demo-ubuntu-focal-demo-install/log-34251808-f6b6-4b9e-849b-58c928cf1176.txt

[dbolduc]

hmmm another demo-ubuntu-focal....

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/811ece9fc33e5c5752a9f5ddc4946d8cc62e39ed/demo-ubuntu-focal-demo-install/log-a17ff190-ec48-4c51-a6e5-a2377ae9afb0.txt

[devbww]

(internal link) https://console.cloud.google.com/cloud-build/builds;region=us-east1/60659dd4-1530-4eba-9e82-734f0e439933?project=936212892354

[coryan]

No repeats in 90d, and probably fixed by #8558

[dbolduc]

(internal link) https://console.cloud.google.com/cloud-build/builds;region=us-east1/1d174b5a-4a85-45dd-b374-4f7a43287e94?project=936212892354

[coryan]

No repeats in 90d, but it seems it happens on longer cycles. I will give it another chance.

[dbolduc]

https://storage.cloud.google.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/8290acb981944ce53e58e09db2df2e3a2e1420ec/ubuntu-focal-quickstart-cmake/log-6a7e6d8b-7802-4b67-adfa-c6e4425ce385.txt

[dbolduc]

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/17b0de962ef2735fd33882f6b556b322af4229a0/fedora-37-cmake-check-api/log-50e153ab-c3e6-404e-ab0b-36694048a286.txt

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/17b0de962ef2735fd33882f6b556b322af4229a0/ubuntu-focal-quickstart-cmake/log-6f371eb1-86ec-4b82-ab99-35446ec2b4fe.txt

[coryan]

No repeats in 90d, closing for now.