appengine-plugins icon indicating copy to clipboard operation
appengine-plugins copied to clipboard

Gradle parallel build support

Open czp3009 opened this issue 1 year ago • 1 comments

When building project with multiple sub-projects, gradle parallel builds are key to improving CI speed.

In this case, gradle will try to build all sub-projects at the same time.

If there are multiple sub-projects to deploy (multiple services), these sub-projects will all have the task downloadCloudSdk.

This is a file download task, if gradle start downloading at the same time, the task will fail.

fail message:

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':default-service:downloadCloudSdk'.
> java.nio.file.FileAlreadyExistsException: /root/.cache/google-cloud-tools-java/managed-cloud-sdk/downloads/google-cloud-sdk.tar.gz

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':example-service:downloadCloudSdk'.
> java.nio.file.FileAlreadyExistsException: /root/.cache/google-cloud-tools-java/managed-cloud-sdk/downloads/google-cloud-sdk.tar.gz

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

There are several workaround, but none of them are perfect:

  1. Not using parallel builds. This results in very slow builds, especially in large projects.
  2. Use an image with gcloud already installed as a CI build image. This may result in the gcloud not being up to date, and the build image containing the compiler needed for the build, which may not been created by the CI user.

For tasks that require requests to external resources, such as the file system, gradle provides an API to avoid execute multi time: https://docs.gradle.org/current/userguide/build_services.html

I don't know much about gradle plugins, can you please see if you can use build service to solve this problem.

czp3009 avatar Aug 08 '22 06:08 czp3009