GO Modules Proxy support

[owenhaynes]

Is there an existing issue for this?

• [X] I have searched the existing issues

Feature description

JFrog, GCP Artificat registry all provide ways for private go modules. Currently the Dependabot docs say that private registries are supported but no corresponding documentation is available.

Private go modules registries can be use in all sorts of way especially for code generation pipelines.

Currently dependabot fails because it can not resolve the packages for example generated.internal/foo

so we need a way of providing GOPROXY and GONOSUMDB to dependabot.

I seen past issues about GOPRIVATE but all seem to lead to dead ends.

[jurre]

Thanks for opening. The current support assumes using private (git) repositories, but we don't currently have great support for private module proxies. We're tracking this internally, but there isn't a public facing issue. I'll link this to our internal issue 👍

[kurtmc]

How is no one using dependabot and artifactory? @jurre Can we get a timeline for implementing the ability to set GOPROXY?

[kurtmc]

@jurre Any update on this?

[abscondment]

Bump on this :)

Our use case: our applications use a mixture of public and private go modules. The private ones are hosted in an internal Github Enterprise (git.example.com). The public ones are hosted by the usual suspects (github.com, golang.org, gopkg.in, etc).

To insulate our build pipeline from public modules becoming unavailable, we proxy all dependencies through a nexus repository.

All our build environments need configuration like this to go mod * successfully:

go env -w GOPROXY=https://nexus.example.com/repository/goproxy/
go env -w GONOPROXY=none
go env -w GONOSUMDB="*.example.com"

Dependabot is missing both of our goals here: it can't find the private modules hosted on git.example.com, and it isn't accessing the public modules through our proxy. The first issue is the real blocker, but solving it correctly would also resolve the second issue.

[bgentry]

@jonjanego if I can be of any help testing a solution here (no matter how beta it is), please let me know. I recently launched River Pro which is installable as a private Go module using GOPROXY and GONOSUMDB envs. I have multiple private repos using this dependency where I'd like to set up Dependabot, but can't because it's unable to access a private or custom GOPROXY server.

This also doesn't just affect me, it affects any customers of River Pro (there are already several, all of which are GitHub customers too).

Excited to see this soon :pray:

[promiseofcake]

This is also important to us.

[joshua-temple]

The general ability to define environment variables for Dependabot would absolve this issue, no? I know, classic end user assumption.

Our org needs GOPROXY GOPRIVATE GONOPROXY and GONOSUMDB

[bgentry]

@abdulapopoola if you need anybody to help you test this in the real world, I'm your guy! I have a couple of projects set up with the River Pro private GOPROXY and I'm more than happy to help you test in any way I can, please lmk.

[paulbes]

Just wanted to add that this is also important for us, as we recently started using River pro, and would like to keep our dependencies up to date.

[mfridman]

Wanted to share another use case. At Buf, we run a Go module proxy for our Generated SDKs. Many of our customers work with private code they need to keep updated, and they frequently mention the lack of dependabot support for private Go modules.

[abdulapopoola]

@bgentry , this is a feature request and unfortunately we don't currently have this on our roadmap. However, this is something I can bring to the team.

[bgentry]

@abdulapopoola gotcha, I was under the impression that it was already being scoped for work in the near future based on your recent updates:

[abdulapopoola]

Yes, it's something we definitely want to do in the near future :), we've had a few more urgent asks come in. It's still in our plans (just not immediate short term).

[dmitris]

we need a way of providing GOPROXY and GONOSUMDB to dependabot.

@owenhaynes I would suggest to change this to "providing GOPROXY and GOPRIVATE" since per https://go.dev/ref/mod#environment-variables, GOPRIVATE is a default value for GONOPROXY and GONOSUMDB.

[alhss]

We've released our new Go Private Registry Support feature and you can start using it today, though please note it's currently in preview and your feedback is welcomed. To set up Dependabot's Go Proxy Support, you have two configuration options. Option 1: Create a go.env file in your repository root to configure Go module behavior (GOPROXY=https://your-registry.com/path and GOPRIVATE=your-company.com/*). This tells Go where to look for modules but doesn't handle authentication. Option 2: Configure credentials in your dependabot.yml file by adding a registries section with your private registry details (type: goproxy-server, URL, username, and password), then reference it in your updates configuration.

[pierrre]

Do we need to store credentials in a Git repository ? There is no "secret" configuration ?

[Nishnha]

@pierrre you can set up a goproxy-server registry in your dependabot.yml with a registry url, username, and password. The password will be something like

password: ${{secrets.MY_GOPROXY_PASSWORD}}

where MY_GOPROXY_PASSWORD is a Dependabot secret

[bgentry]

Amazing @alhss 🎉 Thank you for shipping this.

I can confirm that this config works to allow Dependabot to function even when using private modules on River's private registry.

It was able to generate a dependabot PR on my test repo for the first time in many months. Though there were no updates available for the private deps, it did successfully update other deps and got 200 responses from my proxy sever. I'll be sure to post back if I find anything that doesn't work, will be releasing a new private module version soon and will verify that.

go.env

GONOSUMDB=riverqueue.com/riverpro

Required because this is a private proxy serving private modules.

.github/dependabot.yaml

version: 2
registries:
  golang-proxy:
    type: goproxy-server
    url: https://proxy.golang.org
    username: ""
    password: ""
  riverpro-proxy:
    type: goproxy-server
    url: https://riverqueue.com/goproxy
    username: river
    password: ${{secrets.RIVER_PRO_SECRET}}
updates:
  - package-ecosystem: "gomod"
    directory: "/" # Location of package manifests
    groups:
      go-dependencies:
        update-types:
          - "minor"
          - "patch"
    registries:
      - golang-proxy
      - riverpro-proxy
    schedule:
      interval: "weekly"

[bgentry]

There are a couple things I found surprising or noteworthy about this after further testing.

1. If you specify just a single custom registry (say riverpro-proxy in the example above and set the registries: "*" value, it appears to send all proxy requests through that custom registry first, but then quietly falls back to the proxy.golang.org proxy for anything that fails or 404s in the sole custom registry.

2. You can override this to produce a GOPROXY=proxy.golang.org,riverqueue.com/goproxy style preference cascade by specifying two custom registries as my updated example code above now does. You then specify them individually in the registries list like to produce a desired order:

       registries:
         - golang-proxy
         - riverpro-proxy
   

3. What was particularly surprising/confusing is that even when specifying the proxy.golang.org registry, I was required to provide username and password values even though that proxy does not require them. Empty string values worked fine.

Anyway, once I landed on the above configuration, I got the desired behavior equivalent to GOPROXY=proxy.golang.org,riverqueue.com/goproxy where only the private deps that aren't found on proxy.golang.org are attempted to be fetched on the secondary private registry. I believe you'll want to highlight these issues in the docs (here?) including an example of how to specifically achieve the fall back proxy setup documented in the Go module reference.

[jurre]

@bgentry would you say that order of public first and then private is desirable? My concern with making this the default behavior is you end up leaking these private package names to the public proxy.

I believe you can achieve the desired behavior for your scenario by committing a go.env file to your repo with the GOPROXY value that you want the go toolchain to use.

[bgentry]

@jurre I would say it highly depends! I'm actually writing a blog post on this as we speak 😄 The privacy section in the docs cover it fairly well though.

I think the point is it should be straightforward to control exactly which proxies are used and in which order. If you're saying go.env is the best way to do that, that could be ok, but I'm not clear how that'd play together with authentication defined in the dependabot.yaml.

At the very least it's important that you clearly document how to achieve the various setups with Dependabot and i.e. guarantee that certain private module names aren't ever sent to a proxy that shouldn't see them. If I specify just a single private registry in my registries list, would I be ensured that Dependabot will never attempt to resolve my modules via a public proxy? Maybe the issue with my first attempt was just the usage of * and that resulting in an automatic fallback to the public proxy and/or direct mode.

Again my point (1) above is the most confusing one IMO. I set just a single private registry (albeit specifying registries: "*") and yet Dependabot was still able to resolve modules that 404'd at that registry. Which GOPROXY value actually gets used in that scenario? Is it GOPROXY=https://private-registry.com,direct, or GOPROXY=https://private-registry.com,https://proxy.golang.org,direct? This needs to be clear to avoid accidental leaks or sending requests to the wrong place.

[bgentry]

HI all, I saw that you officially announced this feature this week! However I noticed that some key doc sections have still not been updated, particularly the supported private registry list: https://docs.github.com/en/enterprise-cloud@latest/code-security/dependabot/working-with-dependabot/configuring-access-to-private-registries-for-dependabot#supported-private-registries

I'm also still unclear on the exact behavior / equivalent GOPROXY value that's used when I specify a single private registry with registries: "*" and would appreciate clarification on this. I think it's important to know exactly how to be certain that requests aren't accidentally sent to the wrong registry for cases where that matters (I elaborated on such cases in this blog post on private module proxies).

Thanks!

[alhss]

@bgentry

I noticed that some key doc sections have still not been updated

It should be updated today. Thanks for reminding us

Nice blog post! We will try to clarify those questions.

[alhss]

@bgentry If you have 2 registries set up, and then have registries set up to '*' under update. your GOPROXY will look like 'privateReg1,privateReg2,direct' . here's the line of code referring to it. The 'direct' is a fallback and it means it will fetch directly from the source of the module. So overall no public proxy is ever used but it does fallback to direct. Does that answer your question of requests not being sent to the wrong registry?

You can also configure the goprivate variable in a go.env file in your repository, and Dependabot will respect it. We've included this info in our docs at https://docs.github.com/en/code-security/dependabot/working-with-dependabot/configuring-access-to-private-registries-for-dependabot#goproxy-server