Issue with DNS name resolution?

[Silvanoc]

I'm having issues with ORAS on my company network and I suspect that it can be related with name resolution (DNS).

Environment

Network setup

Working in the office

Directly connected to my company intranet.

Working from home

VPN enables connectivity with the company intranet based on routing.

Container Registry and Authorization Service

Both reachable from both company intranet and Internet. Thanks to internal DNS the names of both services get resolved to IP addresses routed through the VPN when working from home.

Pushing only allowed from the intranet. A gatekeeper blocks it when coming from the Internet.

Issue

Pushing from the intranet works. In this case everything gets routed to the intranet.

Pushing from home fails being blocked by the gatekeeper. Meaning that the request has been routed through the Internet instead of the VPN.

Pushing with Skopeo from home works like a charm. Meaning that connectivity is not an issue.

Question

Can it be that ORAS is somehow resolving the server names for my container registry and authorization service without respecting the configured internal DNS?

Please remember that Skopeo is pushing successfully, so internal DNS and VPN routing are working properly.

After some debugging with the service provider I cannot identify any other reason for the issues...

[shizhMSFT]

ORAS CLI uses golang built-in http package for HTTP/HTTPS connections with the default net.Dialer.

[shizhMSFT]

@Silvanoc Can you provide more information so that we can help you to troubleshoot as it may be a general golang issue?

Information required to be collected are:

• CPU Architecture
• OS Name and Version
• Method to obtain the oras binary
  • If the oras binary is built by yourself, please check the value of CGO_ENABLED
• Output of oras version

[shizhMSFT]

There is a known issue at https://github.com/golang/go/issues/12524.

Since oras CLI binaries in the release page are built with cgo disabled, it will have the same issue in macOS.

[Silvanoc]

CPU Architecture: x86-64 OS Name and Version: Debian Stable/Bullseye/11 Method to obtain the oras binary: downloaded from GitHub project release 0.14.1 Output of oras version:

Version:        0.14.1
Go version:     go1.19
Git commit:     a27b2d2ba151a39e29a227256ce1e2432d1e60f5
Git tree state: clean

[shizhMSFT]

@Silvanoc Thanks for the environment information.

I did some analysis on Debian 11. I tried skopeo installed using apt.

$ skopeo --version
skopeo version 1.2.2
$ ldd `which skopeo`
        linux-vdso.so.1 (0x00007fffff3ef000)
        libgpgme.so.11 => /lib/x86_64-linux-gnu/libgpgme.so.11 (0x00007f2068ccb000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2068ca9000)
        libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007f2068c3c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2068a67000)
        libassuan.so.0 => /lib/x86_64-linux-gnu/libassuan.so.0 (0x00007f2068a51000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f2068a2b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2068d30000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f20689fd000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f20689d5000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2068891000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f20687f9000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f20687f3000)
$ GODEBUG=netdns=2 skopeo list-tags docker://ghcr.io/oras-project/oras > /dev/null
go package net: dynamic selection of DNS resolver
go package net: hostLookupOrder(localhost) = files,dns
go package net: hostLookupOrder(ghcr.io) = files,dns
go package net: hostLookupOrder(ghcr.io) = files,dns
go package net: hostLookupOrder(ghcr.io) = files,dns

As shown above, skopeo from apt is dynamically linked and is using dynamic selection of DNS resolver.

For the oras binary from our release page, we have

$ oras version
Version:        0.14.1
Go version:     go1.19
Git commit:     a27b2d2ba151a39e29a227256ce1e2432d1e60f5
Git tree state: clean
$ ldd `which oras`
        not a dynamic executable
$ GODEBUG=netdns=2 oras manifest fetch ghcr.io/oras-project/oras:v0.14.1 > /dev/null
go package net: confVal.netCgo = false  netGo = true
go package net: built with netgo build tag; using Go's DNS resolver
go package net: hostLookupOrder(ghcr.io) = files,dns

As shown above, oras is statically linked and is using Go's DNS resolver.

@Silvanoc I think the DNS resolver makes the different in your scenarios. Can you try build oras from source with CGO_ENABLED=1? For example,

GOARCH=amd64 CGO_ENABLED=1 GOOS=linux go build -v oras.land/oras/cmd/oras

With the above binary, oras is the using the same DNS resolver as skopeo and should resolve this issue.

$ ldd oras
        linux-vdso.so.1 (0x00007ffc0cede000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4832f44000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4832d6f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4832f7a000)
$ GODEBUG=netdns=2 ./oras manifest fetch ghcr.io/oras-project/oras:v0.14.1 > /dev/null
go package net: confVal.netCgo = false  netGo = false
go package net: dynamic selection of DNS resolver
go package net: hostLookupOrder(ghcr.io) = files,dns

[Silvanoc]

@shizhMSFT that's it! Thanks!

I've built ORAS with CGO_ENABLED=1 and it works! If I disable the cgo resolver (go native one gets used) with GODEBUG=netdns=go (or GODEBUG=netdns=go+2 to see the debug output confirming that the go resolver is being used), then it stops working.

So as ORAS is built, it cannot work in my network environment. Since I'm only in an experimentation phase, that's not a big deal for me, I can simply build it myself. But IMHO ORAS might want to consider this issue. Therefore I'll leave it up to you to close this issue or leave it open to track the problem.

I wonder if it's a bug on Go's resolver. And if it'll affect other Go tools using the native resolver in my environment.

[shizhMSFT]

Thank @Silvanoc for confirmation.

@FeynmanZhou @yizha1 We need to think about distributing software like oras, notation via package managers like apt, brew, etc. so that we can release Linux/Unix distribution specific binaries for native support.

Let's leave this issue open for tracking.

[Silvanoc]

I've found out that I had an erroneous resolver configuration on my machine. Apparently the Glibc resolver was capable of somehow workaround the error, what the Go native resolver couldn't. Therefore Skopeo was working, but ORAS wasn't.

I'd say that there's no bug on ORAS on the Go native resolver, but it's a bit less robust than the Glibc one. Of course by using the Glibc resolver ORAS wouldn't be fully statically linked anymore. I don't know if you want to pay that price... I personally don't have a strong opinion on that.

[FeynmanZhou]

Thank @Silvanoc for confirmation.

@FeynmanZhou @yizha1 We need to think about distributing software like oras, notation via package managers like apt, brew, etc. so that we can release Linux/Unix distribution specific binaries for native support.

Let's leave this issue open for tracking.

I think ORAS has provided such a binary on Homebrew. We can release the binaries with native support on apt, yum, dpkg, etc. after ORAS CLI v1.0.0.

[shizhMSFT]

Closing as the issue has been resolved.