Sometimes fails to restore cache due to crcmod installation isn't using the module's C extension

[adam-ashored]

Affected builder image

(e.g., gcr.io/cloud-builders-community/cache)

Expected Behavior

It should restore cache

Actual Behavior

It fails to restore cache

Additional Info

I have two very similar triggers - one builds an NodeJS/ExpressJS application and the other one builds a React SPA. I've set up save_cache and restore_cache images in both triggers. The NodeJS/ExpressJS application builds, saves, and restores cache fine, but the exact same steps in the React SPA couldbuild.yaml give me this:

Step #0 - "restore_cache": Restoring cache from file ./2257534182-3654919205.tgz...
Step #0 - "restore_cache": could allow data corruption to go undetected during uploading/downloading.
Step #0 - "restore_cache": NOTE: It is strongly recommended that you not disable integrity checks. Doing so
Step #0 - "restore_cache": 
Step #0 - "restore_cache": checks, see the "check_hashes" option in your boto config file.
Step #0 - "restore_cache": To download regardless of crcmod performance or to skip slow integrity
Step #0 - "restore_cache": 
Step #0 - "restore_cache": installing the extension, please see "gsutil help crcmod".
Step #0 - "restore_cache": hash computation will likely throttle download performance. For help
Step #0 - "restore_cache": but your crcmod installation isn't using the module's C extension, so the
Step #0 - "restore_cache": Downloading this composite object requires integrity checking with CRC32c,
Step #0 - "restore_cache": CommandException: 
Step #0 - "restore_cache": Downloading cache file: gs://XXXXXXXXXXX/2257534182-3654919205.tgz...

[bertschneider]

We have the same problem with restore_cache. It looks like the CommandException prevents the download and later on commands fail because the file is not present.

Here is another log:

Step #0 - "restore-m2-cache": Downloading cache file: gs://codebuild-cache/resources-4255399772.tgz...
Step #0 - "restore-m2-cache": CommandException: 
Step #0 - "restore-m2-cache": Downloading this composite object requires integrity checking with CRC32c,
Step #0 - "restore-m2-cache": but your crcmod installation isn't using the module's C extension, so the
Step #0 - "restore-m2-cache": hash computation will likely throttle download performance. For help
Step #0 - "restore-m2-cache": installing the extension, please see "gsutil help crcmod".
Step #0 - "restore-m2-cache": 
Step #0 - "restore-m2-cache": To download regardless of crcmod performance or to skip slow integrity
Step #0 - "restore-m2-cache": checks, see the "check_hashes" option in your boto config file.
Step #0 - "restore-m2-cache": 
Step #0 - "restore-m2-cache": NOTE: It is strongly recommended that you not disable integrity checks. Doing so
Step #0 - "restore-m2-cache": could allow data corruption to go undetected during uploading/downloading.
Step #0 - "restore-m2-cache": Restoring cache from file ./resources-4255399772.tgz...
Step #0 - "restore-m2-cache": tar (child): ./resources-4255399772.tgz: Cannot open: No such file or directory
Step #0 - "restore-m2-cache": tar (child): Error is not recoverable: exiting now
Step #0 - "restore-m2-cache": tar: Child returned status 2
Step #0 - "restore-m2-cache": tar: Error is not recoverable: exiting now
Step #0 - "restore-m2-cache": rm: cannot remove './resources-4255399772.tgz': No such file or directory

[bertschneider]

It should be installed in the base image https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile.slim

[adam-ashored]

@bertschneider good to know it's not just me. It is unclear to me why this is happening with one trigger and not the other.

[sezna]

I've been unable to fix this, I tried rebuilding the base image and also rebuilding this one, but to no avail. Should I just add pip3 and crcmod to the dockerfile for Dockerfile-restore?

[f-loris]

I experienced the same issue. As quick a workaround you can configure check_hashes here while downloading: https://github.com/GoogleCloudPlatform/cloud-builders-community/blob/da89cfe04155209312c255cc79a5063a661e065b/cache/restore_cache#L70 For example: gsutil -q -o GSUtil:check_hashes=always cp "${REMOTE_CACHE_FILE}" "${SRC_DIR}" Other options for check_hasheswould be never or if_fast_else_skip

Note: if fallback key is used, then the following line needs to be modified as well: https://github.com/GoogleCloudPlatform/cloud-builders-community/blob/da89cfe04155209312c255cc79a5063a661e065b/cache/restore_cache#L95