We need CGO to not be disabled in order for netcgo to work properly.

[ncabatoff]

Fixes https://github.com/hashicorp/vault/issues/12012.

[ncabatoff]

@archoversight I haven't had time to work on this lately, but I will come back to it soon. In the meantime, a colleague pointed out that your test with GO_TAGS=netcgo make dev was invalid because the dev make target uses BUILD_TAGS rather than GO_TAGS. So we haven't yet entirely settled the question as to whether CGO_ENABLED must be set to 1.

Really I need to make time to figure out how to test this locally. If you have any tips that don't require setting up openvpn I'm interested.

[archoversight]

@ncabatoff Alright, so based upon your statement I tried compiling with no CGO_ENABLED=1 set in the environment, and setting BUILD_TAGS to netcgo:

% BUILD_TAGS=netcgo make dev
==> Checking that build is using go version >= 1.17.5...
==> Using go version 1.17.6...
go: downloading github.com/hashicorp/consul/api v1.12.0
go: downloading github.com/hashicorp/vault-plugin-auth-kubernetes v0.7.1-0.20220207145307-c9fa6acdfe0e
go: downloading github.com/hashicorp/vault-plugin-secrets-azure v0.11.3
go: downloading github.com/hashicorp/vault-plugin-secrets-gcp v0.11.2
go: downloading github.com/denisenkom/go-mssqldb v0.12.0
go: downloading github.com/hashicorp/serf v0.9.6
go: downloading github.com/golang-sql/sqlexp v0.0.0-20170517235910-f1bb20e5a188
go: downloading github.com/hashicorp/mdns v1.0.4
go: downloading github.com/miekg/dns v1.1.41
==> Removing old directory...
==> Building...
Number of parallel builds: 11

-->    darwin/amd64: github.com/hashicorp/vault

==> Results:
total 352784
-rwxr-xr-x  1 c5309377  staff   172M Feb 10 12:08 vault

Attempting to use it against an instance that is across split DNS:

% vault login -method=oidc
Error authenticating: Put "https://vault.example.internal/v1/auth/oidc/oidc/auth_url": dial tcp: lookup vault.example.internal on 172.16.108.111:53: no such host
% vault --version
Vault v1.10.0-dev ('dce88412d5815f235664e3cf2fa2c4a686023e14')

Running otool against it:

% otool -L ~/go/bin/vault
/Users/c5309377/go/bin/vault:
	/usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)

Recompiling with CGO_ENABLED=1:

% CGO_ENABLED=1 BUILD_TAGS=netcgo make dev
==> Checking that build is using go version >= 1.17.5...
==> Using go version 1.17.6...
==> Removing old directory...
==> Building...
Number of parallel builds: 11

-->    darwin/amd64: github.com/hashicorp/vault

==> Results:
total 372424
-rwxr-xr-x  1 c5309377  staff   182M Feb 10 12:14 vault
% vault login -method=oidc
Complete the login via your OIDC provider. Launching browser to:

[...]

success
% otool -L ~/go/bin/vault
/Users/c5309377/go/bin/vault:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60157.60.19)
% vault --version
Vault v1.10.0-dev ('dce88412d5815f235664e3cf2fa2c4a686023e14') (cgo)

No matter what flags I set, or how I set them, I have to enable CGO_ENABLED=1 to get Go to link against IOKit.framework and to use the macOS system DNS resolution rather than the built-in Go one which does not understand macOS's split DNS infrastructure.

That is whether I use make dev or just plain old make:

% make GO_TAGS=netcgo CGO_ENABLED=1
==> Checking that build is using go version >= 1.17.5...
==> Using go version 1.17.6...
==> Removing old directory...
==> Building...
Number of parallel builds: 11

-->    darwin/amd64: github.com/hashicorp/vault

==> Results:
total 372416
-rwxr-xr-x  1 c5309377  staff   182M Feb 10 12:20 vault
% otool -L ~/go/bin/vault
/Users/c5309377/go/bin/vault:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60157.60.19)
C02CG59EMD6M ~/Projects/3rdparty/vault % vault --version
Vault v1.10.0-dev ('dce88412d5815f235664e3cf2fa2c4a686023e14') (cgo)

Broken:

% make dev BUILD_TAGS=netcgo
==> Checking that build is using go version >= 1.17.5...
==> Using go version 1.17.6...
==> Removing old directory...
==> Building...
Number of parallel builds: 11

-->    darwin/amd64: github.com/hashicorp/vault

==> Results:
total 352784
-rwxr-xr-x  1 c5309377  staff   172M Feb 10 12:18 vault
% otool -L ~/go/bin/vault
/Users/c5309377/go/bin/vault:
	/usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)

% make GO_TAGS=netcgo
==> Checking that build is using go version >= 1.17.5...
==> Using go version 1.17.6...
==> Removing old directory...
==> Building...
Number of parallel builds: 11

-->    darwin/amd64: github.com/hashicorp/vault

==> Results:
total 352784
-rwxr-xr-x  1 c5309377  staff   172M Feb 10 12:17 vault
% otool -L ~/go/bin/vault
/Users/c5309377/go/bin/vault:
	/usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)
% vault --version
Vault v1.10.0-dev ('dce88412d5815f235664e3cf2fa2c4a686023e14')

[archoversight]

In the context of this particular issue, what was observed(1) was that setting CGO_ENABLED=1 on MacOS forces go into an external linker mode (2), and will try to use ld64 from the Apple Developer Tools. This became an issue when targeting ARM. Since we don't have MacOS ARM build hosts, we still have to cross compile those binaries and getting go to use the correct version of the developer tools was not easy (I forget the details now 😞), but we don't actually need the system linker. The go linker is perfectly capable of linking darwin/arm from darwin/amd64. We only care that it is able to load system libraries so that it can use native dns resolution at runtime, which is the behavior with CGO_ENABLED=1 or unset.

Do note that terraform fixed this by building on macOS, and since the toolchain contains the headers/information for both x86_64 and arm64 they were able to build both:

https://github.com/hashicorp/terraform/pull/30300

Could the same technique not be used here for vault?

[archoversight]

Is there anyway to get this work prioritized/completed so that binaries on macOS are able to resolve using the system resolver and thus can use split DNS?

[maxb]

I wonder if, as part of finalizing this journey, it might make sense to revisit whether to continue setting the netcgo build tag, as well?

According to the big comment in https://go.googlesource.com/go/+/go1.20/src/net/net.go, Go has good reasons to prefer its native DNS implementation when it can, and makes efforts to automatically transition to the cgo resolver if needed... and in particular, it already automatically defaults to cgo DNS resolution when running on darwin anyway!

    // Darwin pops up annoying dialog boxes if programs try to do
    // their own DNS requests. So always use cgo instead, which
    // avoids that.
    if runtime.GOOS == "darwin" || runtime.GOOS == "ios" {
        confVal.forceCgoLookupHost = true
        return
    }

(from Go src/net/conf.go)