woodpecker icon indicating copy to clipboard operation
woodpecker copied to clipboard

Use multiple agents if matrix platform is used

Open JulianGro opened this issue 2 years ago • 2 comments

Clear and concise description of the problem

When using the matrix system for building on multiple platforms as described in: https://woodpecker-ci.org/docs/usage/matrix-pipelines#example-matrix-pipeline-using-multiple-platforms Woodpecker will start building on one of the platforms and wait till that build is finished before it starts on the second platform.

The result is drastically increased overall build time.

Suggested solution

Run all platforms at the same time.

Alternative

No response

Additional context

In our case, if we were to fully switch to WoodpeckerCI, a PR CI build would take upwards of 5 hours if it would go through platforms one by one instead of running them all at once.

For clarification: If we define

matrix:
  platform:
    - 'linux/amd64'
    - 'linux/arm64'

platform: ${platform}

I would want linux/amd64 and linux/arm64 to run at the same time.

Validations

  • [X] Read the Contributing Guidelines.
  • [X] Read the docs.
  • [X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

JulianGro avatar Jul 31 '22 20:07 JulianGro

You have multiple agents? And what's your WOODPECKER_MAX_PROCS amount?

anbraten avatar Aug 01 '22 05:08 anbraten

Yes, I have an agent on aarch64 and one on amd64.

My WOODPECKER_MAX_PROCS is set to standard (which I assume is 1), since the documentation lists it as an "agent configuration options". My understanding is that this would enable one agent to build multiple pipelines simultaniously, but I want both my agents to work on one pipeline (since there is a matrix that requires both platforms)

JulianGro avatar Aug 01 '22 06:08 JulianGro