go-duckdb Documentation on cross-compilation

trafficstars

Today, I'd like to ask for a small feature request, which – I think – could be useful to more people than just me.

It's about cross-compiling Go programs which shall use DuckDB. While building a binary for the platform you're developing on is pretty simple, doing the same with cross-compilation is not.

Given that there are three major operating systems these days (macOS, Linux, Windows), and two CPU architectures (x86-64 and ARM), you end up with quite a number of combinations: You essentially have six combinations as development base, and you have six combinations you might want to build.

E.g.:

Build for macOS / ARM on a Linux / x86-64 machine.
Build for Windows / x86-64 on a macOS / arm machine.
And so on …

Effectively, this is 36 combinations in total.

Now here's my question: Since I find it pretty difficult to figure out how to cross-compile, would it maybe be possible to add information on this to the documentation of this module? I think that it doesn't need an example per combination, but having some general guidelines or tips would already be helpful (or even a link to a page which explains it in more detail).

What do you think of this?

PS: If someone is able to provide the commands, I'd be more than happy to work on the documentation and submit a PR. It's just that I don't know how to do it myself.

Sep 22 '24 12:09 goloroden

@goloroden Thanks for bringing this up. Cross-compiling is a total pain. What could make sense is an issue where we first collect all the cross-compiling combinations we have working so far. Then we can start generating a kind of matrix from it with source and destination and which toolchain to use.

Starting with macOS we could use the approach I've commented in an earlier issue you have opened.

With linux as source we could use the compile chain from the Github runners (need to figure out what the have pre-installed) and the commands from the Makefile.

From and to Windows: @taniabogatsch is currently working on a Windows build. But I have no access to Windows to test anything. And I'm still confused about Mingw32, Msys2, Cygwin...

Sep 24 '24 05:09 marcboeker

Okay, I'll start collecting things here. This is my test program:

package main

import (
	"database/sql"
	"fmt"

	_ "github.com/marcboeker/go-duckdb"
)

func main() {
	db, err := sql.Open("duckdb", "")
	if err != nil {
		panic(err)
	}
	defer db.Close()

	var value int
	row := db.QueryRow("SELECT 23 AS value")
	row.Scan(&value)
	fmt.Println(value)
}

The goal is to compile and run the binary afterwards in the appropriate environment. If the file size is around 40 MByte, it apparently includes DuckDB statically linked; and if it runs and prints 23, it seems to work.

This list can – sooner or later – maybe be merged into the official documentation, to make things easier for other people who want to do cross-compilation with DuckDB and Go. If you want to add an entry to this list, please leave a comment below, outlining on which OS/Arch you are compiling, for which OS/Arch you are compiling, what the required steps are, and whether it was verified to work.

So far, the following combinations have been reported and successfully verified to work:

✅ On Linux (ARM) for Linux (ARM)

(Verified by @goloroden)

# Compile
$ go build .

❌ On Linux (ARM) for Linux (x86)

❌ On Linux (ARM) for macOS (ARM)

❌ On Linux (ARM) for macOS (x86)

❌ On Linux (x86) for Linux (ARM)

✅ On Linux (x86) for Linux (x86)

(Verified by @goloroden)

# Compile
$ go build .

❌ On Linux (x86) for macOS (ARM)

❌ On Linux (x86) for macOS (x86)

✅ On macOS (ARM) for Linux (ARM)

(Verified by @goloroden)

# Install the cross-compiler components for Linux (ARM)
$ brew install messense/macos-cross-toolchains/aarch64-unknown-linux-gnu

# Cross-compile
$ CC="aarch64-linux-gnu-gcc" CXX="aarch64-linux-gnu-g++" CGO_ENABLED=1 GOOS=linux GOARCH=arm64 go build .

✅ On macOS (ARM) for Linux (x86)

(Verified by @goloroden)

# Install the cross-compiler components for Linux (x86)
$ brew install messense/macos-cross-toolchains/x86_64-unknown-linux-gnu

# Cross-compile
$ CC="x86_64-linux-gnu-gcc" CXX="x86_64-linux-gnu-g++" CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build .

✅ On macOS (ARM) for macOS (ARM)

(Verified by @goloroden)

# Compile
$ go build .

✅ On macOS (ARM) for macOS (x86)

(Verified by @goloroden)

# Cross-compile
$ CGO_ENABLED=1 GOOS=darwin GOARCH=amd64 go build .

❌ On macOS (x86) for Linux (ARM)

❌ On macOS (x86) for Linux (x86)

❌ On macOS (x86) for macOS (ARM)

❌ On macOS (x86) for macOS (x86)

Oct 01 '24 17:10 goloroden

@marcboeker I have collected what I have so far, do you maybe have other ones you can contribute?

Oct 02 '24 19:10 goloroden

I am trying to build it on windows 10:

go build -tags="no_duckdb_arrow"
# win-duckdb
C:\Program Files\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/Yusuf-I9/go/pkg/mod/github.com/marcboeker/[email protected]/deps/windows_amd64\libduckdb.a(ub_duckdb_main_extension.cpp.obj):ub_duckdb_main_extension.cpp:(.rdata$_ZTVN14duckdb_httplib8DataSink19data_sink_streambufE[_ZTVN14duckdb_httplib8DataSink19data_sink_streambufE]+0x38): undefined reference to `std::basic_streambuf<char, std::char_traits<char> >::seekpos(std::fpos<_Mbstatet>, std::_Ios_Openmode)'
collect2.exe: error: ld returned 1 exit status

I tried different things but it doesn't work. I am able to build duckdb from the source. But somehow I can not build your example.

Oct 19 '24 05:10 yusufozturk

@yusufozturk What happens if you simply run:

$ go build .

Does this create an executable for Windows, which includes DuckDB?

Oct 19 '24 06:10 goloroden

@goloroden no, same error.

I tried a lot of different things, but nothing worked for me.

@taniabogatsch I dont think #248 solves all the issues. Or maybe @JAicewizard can share a step by step process to help us :)

Oct 19 '24 11:10 yusufozturk

@yusufozturk Anyway: Thanks for trying it out and giving instant feedback 😊

Oct 19 '24 11:10 goloroden

Okay somehow it works now. I can build and debug on VS Code, but only with -tags="no_duckdb_arrow". But I have no idea what fixed the problem because I tried a lot of different things.

I will try to replicate the success on a clean Windows machine and I will create a documentation for my team. I can share the same documentation here if I can make it again.

Edit: VS Code debug works perfectly. Building is also okay. Binary is generated. But binary is crashing when it tries to load DuckDB. At this line:

db, err := sql.Open("duckdb", "")

There is no error or panic. Just stops working at this line.

Oct 19 '24 21:10 yusufozturk

I am trying to build it on windows 10:

go build -tags="no_duckdb_arrow"
# win-duckdb
C:\Program Files\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/Yusuf-I9/go/pkg/mod/github.com/marcboeker/[email protected]/deps/windows_amd64\libduckdb.a(ub_duckdb_main_extension.cpp.obj):ub_duckdb_main_extension.cpp:(.rdata$_ZTVN14duckdb_httplib8DataSink19data_sink_streambufE[_ZTVN14duckdb_httplib8DataSink19data_sink_streambufE]+0x38): undefined reference to `std::basic_streambuf<char, std::char_traits<char> >::seekpos(std::fpos<_Mbstatet>, std::_Ios_Openmode)'
collect2.exe: error: ld returned 1 exit status

I tried different things but it doesn't work. I am able to build duckdb from the source. But somehow I can not build your example.

I can reproduce this on linux with mingw, I don't know how to fix it though. One possibility is that the gcc version on github CI is different from that on your (and my) machine, leading to different stdc++ being used (https://stackoverflow.com/questions/73261554/sfml-undefined-reference-to-stdbasic-streambuf). I don't know how to fix this easily, besides setting your mingw version to be the same as on github CI. You could also try running deps.windows.amd64 to create the dependencies specifically for your system and see if that works.

Oct 20 '24 12:10 JAicewizard

Edit: VS Code debug works perfectly. Building is also okay. Binary is generated. But binary is crashing when it tries to load DuckDB. At this line: db, err := sql.Open("duckdb", "") There is no error or panic. Just stops working at this line.

I remember having seen this before in one of the issues... 🤔

I dont think https://github.com/marcboeker/go-duckdb/pull/248 solves all the issues.

I agree, the shipped Windows build currently provides a solution for one(?) workflow/mingw version? But I don't have enough experience with Windows to know how to fix this for the pre-build library...

Also, it might be worth bundling the different issues and solutions on Windows into a discussion. Similar to this issue being a collection of knowledge on cross compilation.

Oct 21 '24 08:10 taniabogatsch

I managed to reproduce a successfull build on a semi-clean windows. basic steps:

install msys64
install ucrt mingw (pacman -S mingw264-ucrt-x86_64-gcc in a ucrt64 shell. Not sure if the shell really matters, as long as it is msys)
make sure this gcc is in path (c:/msys64/ucrt64/bin)

I am 99% sure that the build errors relating to std::basic_streambuf are because of some CRT version mismatches, having an older version might not work. (Arch linux is on mingw 11, which does not work, msys has version mingw 12).

Oct 27 '24 14:10 JAicewizard

I hope this helps anyone, I will open a PR with instructions in a minute. Also if anyone can confirm that using mingw 12 on linux allows for cross-compiling, that would be really cool. I don't feel like upgrading the arch packages myself ATM, but it shouldn't be much work.

Oct 27 '24 14:10 JAicewizard

on linux amd64 for windows amd64

OS: debian

system requirements:

binutils-mingw-w64
gcc-mingw-w64-ucrt64
g++-mingw-w64-ucrt64

CGO_ENABLED=1 \
CC=x86_64-w64-mingw32ucrt-gcc  CXX=x86_64-w64-mingw32ucrt-g++ \
GOOS=windows GOARCH=amd64
go build -ldflags "-linkmode external -extldflags '-static'" -o x.exe .

Mar 27 '25 09:03 obarisk

Whipped up a script to run a sweep, should work on most *nix systems.

EDIT: Updated it with some setup-* functions that perform the required setup per-env. Hopefully we eventually have such a setup function for each host->target pair!

With these setups I now am able to xcompile Linux (x86 -> ARM) and Darwin (ARM -> x86).

#!/bin/bash
set -euo pipefail

cd $(mktemp -d)
echo "working in $(pwd)"

cat >main.go <<EOF
package main

import (
	"database/sql"
	"fmt"

	_ "github.com/marcboeker/go-duckdb/v2"
)

func main() {
	db, err := sql.Open("duckdb", "")
	if err != nil {
		panic(err)
	}
	defer db.Close()

	var value int
	row := db.QueryRow("SELECT 23 AS value")
	row.Scan(&value)
	fmt.Println(value)
}
EOF

go mod init example.com/go-duckdb-xcc/v2
go mod tidy

MY_MACHINE=$(uname -m)
MY_SYSTEM=$(uname -s)

TARGET_MACHINES=(arm64 amd64)
TARGET_SYSTEMS=(linux darwin windows)

setup-Linux-x86_64-linux-arm64 () {
  apt-get update -y
  apt-get install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
  export CC="aarch64-linux-gnu-gcc"
  export CXX="aarch64-linux-gnu-g++"
  export CGO_ENABLED=1
}

setup-Darwin-arm64-linux-arm64 () {
  brew install messense/macos-cross-toolchains/aarch64-unknown-linux-gnu
  export CC="aarch64-linux-gnu-gcc"
  export CXX="aarch64-linux-gnu-g++"
  export CGO_ENABLED=1
}

setup-Darwin-arm64-linux-amd64 () {
  brew install messense/macos-cross-toolchains/x86_64-unknown-linux-gnu
  export CC="x86_64-linux-gnu-gcc"
  export CXX="x86_64-linux-gnu-g++"
  export CGO_ENABLED=1
}

setup-Darwin-arm64-darwin-amd64 () {
  export CGO_ENABLED=1
}

for TARGET_SYSTEM in "${TARGET_SYSTEMS[@]}"; do
  for TARGET_MACHINE in "${TARGET_MACHINES[@]}"; do
    SETUP_CMD="setup-$MY_SYSTEM-$MY_MACHINE-$TARGET_SYSTEM-$TARGET_MACHINE"
    if command -v "$SETUP_CMD"
    then
      $SETUP_CMD
    else
      export CC= CXX= CGO_ENABLED=
    fi

    GOOS="$TARGET_SYSTEM" GOARCH="$TARGET_MACHINE" go build -o "main-$TARGET_SYSTEM-$TARGET_MACHINE" || true
  done
done

printf '\n--------\n\n'

echo 'Using markboeker/go-duckdb/v2 version:'
grep 'github.com/marcboeker/go-duckdb/v2' go.mod
echo ''

for TARGET_MACHINE in "${TARGET_MACHINES[@]}"; do
  for TARGET_SYSTEM in "${TARGET_SYSTEMS[@]}"; do
    msg="Building for ($TARGET_SYSTEM/$TARGET_MACHINE) from ($MY_SYSTEM/$MY_MACHINE): "
    if [ -f "main-$TARGET_SYSTEM-$TARGET_MACHINE" ]; then
      msg+="OK"
    else
      msg+="FAIL"
    fi

    echo "$msg"
  done
done

Output on my machine (macos/arm64) (truncated):

Using markboeker/go-duckdb/v2 version:
require github.com/marcboeker/go-duckdb/v2 v2.3.2

Building for (linux/arm64) from (Darwin/arm64): FAIL
Building for (darwin/arm64) from (Darwin/arm64): OK
Building for (windows/arm64) from (Darwin/arm64): FAIL
Building for (linux/amd64) from (Darwin/arm64): FAIL
Building for (darwin/amd64) from (Darwin/arm64): OK
Building for (windows/amd64) from (Darwin/arm64): FAIL

Output on our CI server (linux/amd64) (truncated):

Using markboeker/go-duckdb/v2 version:
require github.com/marcboeker/go-duckdb/v2 v2.3.2

Building for (linux/arm64) from (Linux/x86_64): OK
Building for (darwin/arm64) from (Linux/x86_64): FAIL
Building for (windows/arm64) from (Linux/x86_64): FAIL
Building for (linux/amd64) from (Linux/x86_64): OK
Building for (darwin/amd64) from (Linux/x86_64): FAIL
Building for (windows/amd64) from (Linux/x86_64): FAIL

Jun 25 '25 05:06 antonysouthworth-halter

Hi, wondering if there's any updates/success on this? We are trying to build our app's binaries with DuckDB but the cross-compilation from linux targetting darwin and windows failed.

Aug 13 '25 18:08 Yuan325

👋 Thanks for opening this issue!

This issue has been moved to https://github.com/duckdb/duckdb-go/issues/20, and I'm closing it here.

Please see https://github.com/marcboeker/go-duckdb/issues/565 for more details.

Oct 17 '25 14:10 mlafeldt

go-duckdb go-duckdb copied to clipboard

Documentation on cross-compilation

✅ On Linux (ARM) for Linux (ARM)

❌ On Linux (ARM) for Linux (x86)

❌ On Linux (ARM) for macOS (ARM)

❌ On Linux (ARM) for macOS (x86)

❌ On Linux (x86) for Linux (ARM)

✅ On Linux (x86) for Linux (x86)

❌ On Linux (x86) for macOS (ARM)

❌ On Linux (x86) for macOS (x86)

✅ On macOS (ARM) for Linux (ARM)

✅ On macOS (ARM) for Linux (x86)

✅ On macOS (ARM) for macOS (ARM)

✅ On macOS (ARM) for macOS (x86)

❌ On macOS (x86) for Linux (ARM)

❌ On macOS (x86) for Linux (x86)

❌ On macOS (x86) for macOS (ARM)

❌ On macOS (x86) for macOS (x86)

go-duckdb
go-duckdb copied to clipboard