[custom-images] deprecate gsutil command

[cjac]

This PR includes recent changes I've made as I've iterated on the examples in the example/secure-boot/ directory as well as a patch to fix issue #111

README.md custom_image_utils/args_parser.py tests/test_args_parser.py

• returned default disk size to 30GB. It could be 20GB since the base image only occupies 18GB, but the source disk image is already set to 30GB, so we'll maintain that. If we need a large disk, we can specify the size at the time of image generation.
• You can see the current disk usage statistics per tensorflow (tf) image here: https://github.com/GoogleCloudDataproc/custom-images/blob/19d386eb8e9582739a0087472b8f53ffdbf95da0/examples/secure-boot/pre-init.sh#L149-L157

custom_image_utils/shell_script_generator.py

• Remove some uses of execute_with_retries
• deprecate gsutil ; prefer gcloud
• only print red/green for a short section, not the entire line
• include signing key material in the metadata attributes
• reduce noise from dd

examples/secure-boot/build-current-images.sh examples/secure-boot/create-key-pair.sh examples/secure-boot/dask.sh examples/secure-boot/env.json.sample examples/secure-boot/install_gpu_driver.sh examples/secure-boot/pre-init.sh examples/secure-boot/rapids.sh

• iterative changes in the example scripts

scripts/customize_conda.sh startup_script/run.sh

• deprecate gsutil ; prefer gcloud

[cjac]

All of my tests are passing, and I'm prepared to submit this change with customer's confirmation that the updates do not cause their workflow to fail.

[cjac]

I've pushed these changes up to the 2025.02 branch of this repo if you'd like to test them. Here is an example of how one might exercise the script on all supported images:

git clone -b 2025.02 [email protected]:GoogleCloudDataproc/custom-images.git
cd custom-images
grep -B3 'bash .*/build-current-images.sh' examples/secure-boot/README.md
cp examples/secure-boot/env.json.sample env.json
vi env.json
docker build -t dataproc-dask-rapids-pre-init:latest .
docker run -it dataproc-dask-rapids-pre-init:latest /bin/bash examples/secure-boot/build-current-images.sh

[cjac]

Thank you Chris, I'm still running my suite of tests, so a bit of buffer is helpful to me.