GoogleCloudDataproc
[custom-images] install_gpu_driver.sh does not configure spark when run as startup script with custom-images #110
Problem
Although the gpu driver installer completes successfully when used as an image customization script, it does not create or alter the spark defaults file to include gpu parameters. This is because at the time of the run of the startup script, bdutil has not yet created the files.
See configure_yarn_resources[1] for example
This is a duplicate of the same issue[2] in the custom-images repository
Proposed solution:
Detect whether the script is being run as an image customization. If custom image environment not detected, run as normal; if custom image environment is detected, delay execution by installing a service to run the configuration update phase on first boot.
Scripts to be run after first boot can be placed into a directory such as /etc/rc.local.d/
An entry could be added to /etc/rc.local to
- check for the existence of files in this directory
- execute them
- upon successful execution, remove them
[1] https://github.com/GoogleCloudDataproc/initialization-actions/blob/7e875225563b21680623ffc8bd04d8888aaef626/gpu/install_gpu_driver.sh#L1436 [2] https://github.com/GoogleCloudDataproc/custom-images/issues/110
