gapic-generator-java icon indicating copy to clipboard operation
gapic-generator-java copied to clipboard

chore: secure hermetic-build docker image

Open diegomarquezp opened this issue 1 year ago • 3 comments

This PR improves the Hermetic Build Docker image in a few different ways:

  • Use secure base images
  • Use a secure repository for Python dependencies. This was done by pointing to the Airlock Python registries, which requires a couple of special configuration files and a build-time secret.
    • The build-time secret must be configured before building the image locally (instructions added in the development guide). Cloud Build automatically injects the secrets and no special setup is needed.
    • A few dependencies were submitted for approval and they were automatically added to the staging repository. Once they become part of the "trusted" repository, we will use the "safest" trusted repository.
  • Use multiple stages where only the relevant binaries are copied into the final image
    • The library_generation python package and its dependencies are built in a separate stage
  • Use Alpine-based images
    • Since this kind of distro is not compatible with glibc-based binaries, a special compatibility enablement step was added
    • Since the UNIX tools are from FreeBSD, the -d flag in the rm command is not supported. This is why we removed its usage in the scripts
  • Reduction of image size to ~1 GB. Most space is used by:
    • OpenJDK: ~300MB
    • GLIBC compatibility: ~250MB
  • Reduction of build time to 150 seconds (parallel stage builds via buildkit).

Pending items

  • [x] Discuss a way to pass the Application Default Credentials to the now (now failing - see job output) hermetic_library_generation workflow in sdk-platform-java (other repos don't need to build the image). We can add a secret in the repo or migrate this action to Cloud Build.
  • [x] Discuss a strategy for the integration tests now that we need a secret to passed it to the image building workflow. The current approach is to use Cloud Build instead of GitHub Actions since it has out-of-the box compatibility with APPLICATION_DEFAULT_CREDENTIAL secrets such as the one declared in the Dockerfile. In the meantime, I left a build workflow in .cloudbuild to replace the integration tests and removed the integration tests portion of the existing GitHub Action. Another possible approach is to have a repository secret and keep using the GitHub Action.
    • [ ] If approach is validated, create the Cloud Build trigger

diegomarquezp avatar Sep 12 '24 17:09 diegomarquezp

I think this PR has done multiple things, change base image and optimize image building.

Can we modify the base image first and create a new PR for optimization?

JoeWang1127 avatar Sep 17 '24 14:09 JoeWang1127