Unable to build depending on platform/parallelism param

[wiardvanrij]

Hi, for Thanos we have been using [email protected] (no shame) but I was working to upgrade this due to the awesome parallelism feature :)

I'm not going to lie, I've spend multiple hours figuring out why our crossbuild did not work. I could eventually pin-point it back to version 0.11.1 where it gets broken. However I absolutely could not find a reason on why that specific version. Further more I also figured out it only break on 2 of the 5 architectures/platforms;

- linux/amd64
- **darwin/amd64**
- linux/arm64
- windows/amd64
- **freebsd/amd64**

So, when I removed freebsd & darwin, it works fine for the other platforms.

The error is:

>> building Thanos binary in .build/darwin-amd64
: not foundmu-v0.13.0: 1: ����
/go/bin/promu-v0.13.0: 2: Syntax error: ")" unexpected
make: *** [Makefile:151: build] Error 2
!! exit status 2

Things I tried;

• checked if the promu bin was actually there + correct right (yes on both).
• build with -p darwin to rule out some regex behavior, but it gives exactly the same error.
• build it on GitHub, but it's the same issue.
• Tried to basically clean up my Makefile, just to make sure there was nothing there and only ran the Makefile with just the PHONY's build & crossbuild.
• I also checked Prometheus usage of Promu and I saw no discrepancies in either the configuration or usage. I think we basically do the same but it just fails here (:

Some other info

Local;

• Docker version 20.10.12, build e91ed57
• Pop!_OS 21.10
• go version go1.17.5 linux/amd64

To reproduce;

I guess you could checkout https://github.com/wiardvanrij/thanos/tree/feature/testing-parallelism-circle-local and do a make crossbuild Also see the build here: https://github.com/wiardvanrij/thanos/actions/runs/1612950003

Any help/pointers would be appreciated!

[wiardvanrij]

I've been trying again, so what I noticed was:

Doing a build without parallelism (--parallelism 1 --parallelism-thread 0) gives me the following:

  thanos git:(feature/parallel-build) make crossbuild
(re)installing /home/wiard/go/bin/promu-v0.13.0
go: downloading github.com/prometheus/promu v0.13.0
go: downloading github.com/google/go-github/v25 v25.1.3
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
>> crossbuilding all binaries defined in .promu.yml
/home/wiard/go/bin/promu-v0.13.0 crossbuild -v --parallelism 1 --parallelism-thread 0
Using config file: .promu.yml
Running command: crossbuild [-v --parallelism 1 --parallelism-thread 0]
> running the base builder docker image
 +  docker create -t --name promu-crossbuild-base1640290228-0 quay.io/prometheus/golang-builder:1.17-base -i github.com/thanos-io/thanos -p linux/amd64 linux/arm64 windows/amd64
eec33bbabc9c9623e61fa5711970a4bbdff409a276f0f92d28d5ca799cfc7a51
 +  docker cp /home/wiard/Projects/thanos/. promu-crossbuild-base1640290228-0:/app/
 +  docker start -a promu-crossbuild-base1640290228-0

...SNIP...

# linux-arm64
>> building Thanos binary in .build/linux-arm64
 >   thanos
# windows-amd64
>> building Thanos binary in .build/windows-amd64
 >   thanos.exe
go: downloading github.com/elastic/go-windows v1.0.1
 +  docker cp -a promu-crossbuild-base1640290228-0:/app/.build/. /home/wiard/Projects/thanos/.build
 +  docker rm -f promu-crossbuild-base1640290228-0
promu-crossbuild-base1640290228-0

So, successful.

Doing it with parallelism (--parallelism 2 --parallelism-thread 1).

    thanos git:(feature/parallel-build) make crossbuild CI_PARALLELISM=2 CI_PARALLELISM_THREAD=1
    >> crossbuilding all binaries defined in .promu.yml
    /home/wiard/go/bin/promu-v0.13.0 crossbuild -v --parallelism 2 --parallelism-thread 1
    Using config file: .promu.yml
    Running command: crossbuild [-v --parallelism 2 --parallelism-thread 1]
    > running the base builder docker image
    +  docker create -t --name promu-crossbuild-base1640290396-1 quay.io/prometheus/golang-builder:1.17-base -i github.com/thanos-io/thanos -p linux/arm64 windows/amd64
    3c93cd3f61ccb2ae80a3beda891b5fc092646d72ef9648f230f90560f50699ea
    +  docker cp /home/wiard/Projects/thanos/. promu-crossbuild-base1640290396-1:/app/
    +  docker start -a promu-crossbuild-base1640290396-1

    ...SNIP...

    > building Thanos binary in .build/linux-arm64
    /go/bin/promu-v0.13.0: 1: Syntax error: word unexpected (expecting ")")
    make: *** [Makefile:151: build] Error 2
    !! exit status 2
    The base builder docker image exited unexpectedly
    github.com/prometheus/promu/cmd.runCrossbuild
        /home/wiard/go/pkg/mod/github.com/prometheus/[email protected]/cmd/crossbuild.go:160
    github.com/prometheus/promu/cmd.Execute
        /home/wiard/go/pkg/mod/github.com/prometheus/[email protected]/cmd/promu.go:131
    main.main
        /home/wiard/go/pkg/mod/github.com/prometheus/[email protected]/main.go:20
    runtime.main
        /usr/local/go/src/runtime/proc.go:255
    runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1581
    make: *** [Makefile:163: crossbuild] Error 1

[wiardvanrij]

I went into the build container and just checked the params.

The way it was now, did not work for me: (mind the -p, just spaces)

root@c022d9d43f23:/app# ../builder.sh -i github.com/thanos-io/thanos -p linux/arm64 windows/amd64
Error: Unknown option: windows/amd64
Usage: builder.sh [args]
  -i,--import-path arg  : Go import path of the project
  -p,--platforms arg    : List of platforms (GOOS/GOARCH) to build separated by a space
  -T,--tests            : Go run tests then exit

Quoted works

root@c022d9d43f23:/app# ../builder.sh -i github.com/thanos-io/thanos -p "linux/arm64 windows/amd64" 
# linux-arm64

Comma separated as well

root@c022d9d43f23:/app# ../builder.sh -i github.com/thanos-io/thanos -p linux/arm64,windows/amd64  
# linux-amd64

Look, I'll be honest I have no idea why it does work for Prometheus or Promu itself. But if I use it for Thanos, for some reason at some Promu version this is "broken". I found it really, really hard to debug this due to my lack of knowledge on Promu and the fact that it downloads Promu in the build container again (which is where something goes bad).

So I have tested this against promu and Thanos. If this change makes sense, then: great. However, it would make me feel better if someone else could actually confirm where it is going bad and if this is actually the correct fix via #229

edit: I see the build for my PR actually breaks, so perhaps this isn't solution although I have no idea why it does work locally :D

[roidelapluie]

Ooh in prometheus we repeat the -p as far as I know

../builder.sh -i github.com/thanos-io/thanos -p linux/arm64 -p windows/amd64

[roidelapluie]

https://github.com/prometheus/prometheus/blob/41d0183d16e8b2fc731272d23f3fec8fc67dbf45/.github/workflows/ci.yml#L106

[simonpasquier]

I think that it can be closed: the failure is because the Thanos build pipeline reinstalls promu from source (instead of the release tarball for Prometheus) but the GOOS/GOARCH variables aren't properly set.

https://github.com/thanos-io/thanos/pull/6824#issuecomment-1820917358