Cross-compilation support

[surendrabaid]

Description

I'm facing same problem now, not able to build for linux . I'm building on my mac.

skbaid$ make release OS=linux ARCH=amd64 VERSION=2.00.18 GOPATH=/Users/skbaid/GoglandProjects/abc-service-adapter GOARCH="amd64" GOOS="linux" go build -ldflags "-w -s -X main.Version=2.00.18 -X main.Name=abc-service-adapter" -o bin/abc-service-adapter src/abc/main.go go build github.com/confluentinc/confluent-kafka-go/kafka: build constraints exclude all Go files in /Users/skbaid/GoglandProjects/abc-service-adapter/src/github.com/confluentinc/confluent-kafka-go/kafka make: *** [release] Error 1

has anyone found solution for this ?

[edenhill]

Cross compilation is not currently supported

[surendrabaid]

Could you please elaborate? when i use command 'make release OS=#darwin ARCH=amd64 VERSION=2.00.18' , build is successful but when i run 'make release OS=# linux ARCH=amd64 VERSION=2.00.18' , build is failing.

[edenhill]

We currently do not support building for another OS or ARCH than you are currently on: in your case you try to build a Linux binary on an OSX machine.

[nkristianto]

hi @edenhill does this case solved on the new version of confluent-kafka-go?

[edenhill]

@nkristianto Sorry no, cross builds are currently not supported.

Having said that, cross compilation is most likely possible but it will take a fair bit of environment-specific work to get working. Please share your success.

[yctn]

Would be nice to get this fixed in the future

[darkdefender27]

Definitely need cross compilation. :/

[Smadarl]

I was able to get this to work using XGo: https://github.com/karalabe/xgo. I created a simple docker image based off their generic image (5GB), using this docker file:

FROM karalabe/xgo-latest
RUN git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure && make && make install && cd .. && rm -rf librdkafka

Then, I was able to compile my program like this: xgo --targets=linux/amd64 -image build-go-kafka ./consumers/actions Worked like a charm.

[mikekamornikov]

@edenhill I'm trying to use confluent-kafka-go in a monorepo driven by bazel. The thing is most of our devs use OSX for daily work while the final producer/consumer services are dockerized. And it means we need to cross-compile anyway. I wonder:

• How hard it is to make it possible to cross-compile librdkafka?
• Is it super-hard or just low priority because no one is asking for it?
• How do devs in the wild use confluent-kafka-go? Especially what is the recommended way to handle librdkafka now? Should I prepare a docker base with everything ready OOB? Should I prepare cc toolchain somehow to cross-compile? Please advice

[jubyvictor]

Trying to use v 1.4.2, go build for linux fails on OSX. Assuming its a cross-compilation issue ? go build for osx succeeds.

` ebuspub git:(master) ✗ GOOS=linux GOARCH=amd64 go build

ebuspub/ebusnative

ebusnative/ebusnative.go:21:20: undefined: kafka.Producer ebusnative/ebusnative.go:53:23: undefined: kafka.NewProducer ebusnative/ebusnative.go:53:42: undefined: kafka.ConfigMap ebusnative/ebusnative.go:83:37: undefined: kafka.Message ebusnative/ebusnative.go:97:9: undefined: kafka.Message `

[shivakumarss]

This is very much required. Please prioritise this ticket.

[edenhill]

We recommend that you use a docker container to build non-native binaries, if possible.

[michaelcoder33]

This is still a issue!?

[NewestUser]

I have a similar issue. I try to build the project inside a docker container. The build fails with:

../kfka/producer.go:11:12: undefined: kafka.NewProducer
../kfka/producer.go:11:31: undefined: kafka.ConfigMap
../kfka/producer.go:23:5: undefined: kafka.Producer
../kfka/producer.go:32:21: undefined: kafka.Message
ERROR: Service 'my-app' failed to build : The command '/bin/sh -c go mod download && cd cmd && go build -o my-app' returned a non-zero code: 2

At the same time when I build it on my osx laptop it succeeds. Here is the docker file that I use to run the build in:

FROM golang:1.14.6-alpine3.12

# create a build dir
RUN mkdir /build
WORKDIR /build

# it seems gcc is not available on plain alpine
ENV CGO_ENABLED=0

# copy module dependencies
COPY go.mod .
COPY go.sum .

# copy the project itself
COPY . .

RUN go mod download && cd cmd && go build -o my-app

CMD ["./cmd/my-app"]

Here is my go.mod file

module github.com/newestuser/my-app

go 1.14

require (
	github.com/confluentinc/confluent-kafka-go v1.4.2
	github.com/go-playground/universal-translator v0.17.0 // indirect
	github.com/gorilla/mux v1.8.0
	github.com/leodido/go-urn v1.2.0 // indirect
	github.com/stretchr/testify v1.6.1
	go.mongodb.org/mongo-driver v1.4.2
	gopkg.in/go-playground/assert.v1 v1.2.1 // indirect
	gopkg.in/go-playground/validator.v9 v9.31.0
)

[dotmodusgeorge]

@NewestUser

Check out https://github.com/confluentinc/confluent-kafka-go#librdkafka.

When building on Alpine it requires additional flags to be set.

[NewestUser]

@dotmodusgeorge thanks for pointing this out. I confirm that if I remove ENV CGO_ENABLED=0 from my Dockerfile the code compiles successfully.

[mejedi]

I wonder why it is not fixed in 2 years. Our project has recently switched from another Kafka connector, and cross compilation stopped working. This is a serious usability issue.

@edenhill Could you please share what are the issues preventing cross-compilation?

[edenhill]

The standard Go toolchain does not support cross-compilation of CGo out of the box, we recommend using docker for this purpose. This looks promising: https://github.com/gythialy/golang-cross

[mejedi]

@edenhill Wasn't aware that the standard Go toolchain does not support cross-compilation of CGo out of the box. Thank you for the info.

[cturner-confluent]

I just came across this article and seems like it might offer a solution other than compile inside docker haven't fully tested it myself but reading through it sounds promising - https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho. just sharing for anyone else who may be running into this issue still.

[aca]

CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CC="zig cc -target x86_64-linux-musl" CXX="zig c++ -target x86_64-linux-musl" go build -tags musl

musl working fine

[jiawei666]

is this support now?

[jiawei666]

is this support now? @edenhill

[sagikazarmark]

Cross-compilation works with static linking on Alpine with the prebuilt libraries:

FROM --platform=$BUILDPLATFORM tonistiigi/xx:1.2.1@sha256:8879a398dedf0aadaacfbd332b29ff2f84bc39ae6d4e9c0a1109db27ac5ba012 AS xx

FROM --platform=$BUILDPLATFORM golang:1.20.4-alpine3.18@sha256:0a03b591c358a0bb02e39b93c30e955358dadd18dc507087a3b7f3912c17fe13 AS builder

COPY --from=xx / /

RUN apk add --update --no-cache ca-certificates make git curl clang lld

ARG TARGETPLATFORM

RUN xx-apk --update --no-cache add musl-dev gcc

RUN xx-go --wrap

WORKDIR /usr/local/src/openmeter

ARG GOPROXY

ENV CGO_ENABLED=1

COPY go.mod go.sum ./
RUN go mod download

COPY . .

# See https://github.com/confluentinc/confluent-kafka-go#librdkafka
# See https://github.com/confluentinc/confluent-kafka-go#static-builds-on-linux
RUN go build -ldflags '-linkmode external -extldflags "-static"' -tags musl .
RUN xx-verify YOUR_BINARY

Make sure you are not building for ARMv7, because that is not supported.

Also, using the APK packaged version does not seem to work with cross-compilation.

See https://github.com/confluentinc/confluent-kafka-go/issues/595#issuecomment-778795349

[amigooy]

It works for me on my Mac(Intel).

1. Install musl-cross

brew install FiloSottile/musl-cross/musl-cross

2. Compile

CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CC=x86_64-linux-musl-gcc go build -o main-linux --ldflags '-linkmode external -extldflags "-static"' -tags musl

3. On Linux, verify whether the executable file has dynamic dependencies

$ ldd main-linux
	not a dynamic executable