[epic] Remove GOPATH for go modules formula

[chenrui333]

Historically, golang builds need to happen under the GOPATH, as more and more formula has moved to go-modules, we can deprecate the GOPATH usage.

[jonchang]

Looks like there are 291 formula that still use GOPATH.

• [x] acmetool
• [ ] ahoy
• [x] akamai
• [x] algernon
• [x] aliyun-cli
• [x] alp
• [ ] amazon-ecs-cli
• [x] annie
• [x] antibody
• [x] anycable-go
• [ ] apache-brooklyn-cli
• [x] apm-server
• [ ] aptly
• [x] armor
• [x] assh
• [x] atlantis
• [x] auditbeat
• [x] aurora
• [x] aws-es-proxy
• [x] aws-iam-authenticator
• [x] aws-okta
• [x] benthos
• [x] bettercap
• [x] bitrise
• [x] borg
• [x] buildkit
• [x] c14-cli
• [x] calicoctl
• [x] cayley
• [x] certstrap
• [ ] cf-tool
• [x] cfssl
• [x] chamber
• [x] charm
• [x] chronograf
• [x] cig
• [x] circleci
• [x] cli53
• [x] clipper
• [x] cointop
• [x] collector-sidecar
• [x] consul-backinator
• [x] consul-template
• [x] consul
• [x] container-diff
• [x] convox
• [ ] corectl
• [x] cql
• [x] ctop
• [ ] cwlogs
• [x] dashing
• [x] dcos-cli
• [x] deis
• [x] deisctl
• [ ] dep
• [ ] devd
• [x] devspace
• [x] direnv
• [x] dive
• [x] dnscontrol
• [x] dnscrypt-proxy
• [ ] docker-credential-helper-ecr
• [ ] docker-credential-helper
• [x] docker-gen
• [x] docker-ls
• [ ] docker-machine-driver-hyperkit
• [ ] docker-machine-driver-vmware
• [ ] docker-machine-driver-vultr
• [ ] docker-machine-parallels
• [ ] docker-machine
• [ ] docker-swarm
• [ ] docker
• [ ] docker2aci
• [ ] dockerize
• [ ] dockviz
• [ ] dockward
• [x] doctl
• [x] drone-cli
• [ ] dvm
• [x] elvish
• [ ] emp
• [x] envconsul
• [x] etcd
• [x] exercism
• [x] faas-cli
• [x] fabio
• [x] filebeat
• [ ] fleetctl
• [ ] flint-checker
• [x] fluxctl
• [x] fn
• [ ] forego
• [x] fork-cleaner
• [x] fortio
• [x] frpc
• [x] frps
• [x] frugal
• [x] fsql
• [x] gauge
• [ ] gdm
• [x] gdrive
• [x] geoipupdate
• [x] ghr
• [ ] git-appraise
• [x] git-sizer
• [x] git-town
• [x] github-markdown-toc
• [ ] github-release
• [x] gitlab-runner
• [ ] glide
• [ ] glooctl
• [ ] go-bindata
• [x] go-jira
• [x] go-md2man
• [ ] go
• [ ] [email protected]
• [ ] [email protected]
• [ ] [email protected]
• [ ] [email protected]
• [ ] goad
• [x] gobuster
• [ ] gocryptfs
• [ ] gofabric8
• [x] gollum
• [x] gomplate
• [x] goose
• [ ] gopass
• [x] gor
• [x] goreman
• [x] gost
• [x] gotags
• [ ] govendor
• [x] gowsdl
• [ ] gox
• [x] grafana
• [x] gron
• [ ] grv
• [ ] gx-go
• [x] gx
• [x] heartbeat
• [x] helm
• [ ] helm@2
• [x] helmsman
• [x] hey
• [x] hivemind
• [x] hostess
• [x] hub
• [x] iamy
• [x] immortal
• [ ] influxdb
• [ ] infrakit
• [x] inlets
• [ ] ipfs
• [x] iron-functions
• [ ] ironcli
• [ ] istioctl
• [ ] jabba
• [x] jd
• [x] jfrog-cli-go
• [x] jid
• [ ] jp
• [x] juju
• [x] jump
• [x] jvgrep
• [x] k3d
• [x] k6
• [x] kapacitor
• [x] karn
• [ ] kedge
• [x] kompose
• [ ] kops
• [x] kube-aws
• [x] kubeaudit
• [x] kubebuilder
• [x] kubeless
• [x] kubeprod
• [ ] kubernetes-cli
• [x] kubernetes-service-catalog-client
• [x] kyma-cli
• [ ] landscaper
• [x] lean-cli
• [x] leaps
• [x] lego
• [x] lf
• [x] linkerd
• [ ] lxc
• [x] mage
• [ ] mailhog
• [ ] massren
• [x] megacmd
• [x] metricbeat
• [x] micro
• [x] minio-mc
• [x] minio
• [x] mmark
• [ ] modd
• [ ] mpdviz
• [x] nats-server
• [x] nats-streaming-server
• [x] node_exporter
• [x] nomad
• [x] nsq
• [x] oauth2_proxy
• [x] octant
• [x] opa
• [ ] openshift-cli
• [ ] operator-sdk
• [x] overmind
• [x] packer
• [x] peco
• [ ] perkeep
• [x] pgweb
• [x] piknik
• [ ] pilosa
• [x] prest
• [x] protoc-gen-go
• [x] prototool
• [ ] pulumi
• [x] pumba
• [ ] pup
• [ ] qpm
• [ ] rack
• [x] rancher-cli
• [ ] rancher-compose
• [x] rke
• [x] s-search
• [x] scc
• [x] scw
• [x] serve
• [x] shellz
• [x] shfmt
• [x] ship
• [ ] sift
• [ ] skaffold
• [x] skopeo
• [x] slackcat
• [x] slacknimate
• [x] smimesign
• [ ] snag
• [ ] snap-telemetry
• [x] sonobuoy
• [x] spaceinvaders-go
• [x] srclib
• [x] ssh-vault
• [x] ssllabs-scan
• [ ] step
• [x] stolon
• [ ] stout
• [x] syncthing-inotify
• [ ] syncthing
• [x] td
• [ ] teleconsole
• [x] telegraf
• [x] teleport
• [ ] termshare
• [x] terraform-docs
• [x] terraform-inventory
• [x] terraform
• [ ] [email protected]
• [x] terraformer
• [x] terragrunt
• [x] terrahelp
• [ ] textql
• [x] the_platinum_searcher
• [x] traefik
• [x] traefik@1
• [x] triangle
• [x] ultralist
• [ ] uru
• [ ] vault
• [x] vegeta
• [x] velero
• [ ] vert
• [x] virgil
• [x] virustotal-cli
• [x] vultr
• [x] warp
• [x] websocketd
• [x] wego
• [x] wellington
• [x] wsk
• [x] wskdeploy
• [x] wu
• [x] yq

[alebcay]

While we're here - would it be worth standardizing some of the flags that we use for building Go formulae? Specifically, I would advocate for always including

• -ldflags "-s -w", unless it breaks build
• -trimpath: "remove all file system paths from the resulting executable. Instead of absolute file system paths, the recorded file names will begin with either "go" (for the standard library), or a module path@version (when using modules), or a plain import path (when using GOPATH)."

[jonchang]

It looks like Debian sets -trimpath but doesn't strip binaries with -s -w. Would it be useful to add something like std_go_args (like we have for CMake builds)?

[alebcay]

Would it be useful to add something like std_go_args (like we have for CMake builds)?

I think it would be useful. I've already seen some formulae floating around with stripped binaries, which is why I've brought this up. From my limited testing with the PRs above, there's a reduction of 15-20% in total formula size. I personally don't think it would be a bad idea to strip binaries, seeing as Go binaries are all statically linked.

[SMillerDev]

Maybe you want to wait @chenrui333 and do gopath at the same time as the go_std_args?

[chenrui333]

Would it be useful to add something like std_go_args (like we have for CMake builds)?

+1 for std_go_args idea.

We can definitely pipeline the work between std_go_args and gopath removal though. After that we can refine our template for easy bootstrap golang formula.

[FiloSottile]

You can use the GOFLAGS environment variable to always set -trimpath. I'd recommend against stripping DWARF, as people might want to use a debugger or a profiler on their binaries.

[SMillerDev]

That's not what's suggested here, this is about GOPATH

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[chenrui333]

checkin, give some more time for this issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[chenrui333]

checkin, give some more time for this issue.

[carlocab]

Looks like many of failures in go1.16 (#71289) need fixing by removing GOPATH. These are the formula that have failed to build on both Mojave and Catalina:

• [x] ahoy #71507
• [x] akamai #71487
• [x] amazon-ecs-cli #71480
• [x] apache-brooklyn-cli #71603
• [x] aptly #71606
• [x] bitrise #71607
• [x] carina #71608
• [x] cf-tool #71609
• [x] collector-sidecar #71610
• [x] confd #71610
• [x] cql #71610
• [x] cwlogs #71657
• [x] devd #71668
• [x] docker-credential-helper #71669
• [x] docker-gen #71670
• [x] docker-machine-driver-hyperkit #71671
• [x] docker-machine-driver-vmware #71671
• [x] docker-machine-driver-vultr #71671
• [x] dockerize #71471
• [x] dockviz #71573
• [x] dockward #71672
• [x] dvm #71673
• [x] emp #71674
• [x] flint-checker #71474
• [x] fsql #71675
• [x] gdm #71569
• [x] gdrive #71472
• [x] git-appraise #71570
• [x] git-sizer #71527
• [x] github-release #71473
• [x] gollum #71676
• [x] gx-go #71679
• [x] hivemind #71711
• [x] ironcli #71711
• [x] jabba #71688
• [x] jp #71711
• [x] kedge #71659
• [x] leaps #71700
• [x] mailhog #71711
• [x] massren #71456
• [ ] microplane #71458
• [x] modd #71700
• [x] mpdviz #71700
• [x] path-extractor #71604
• [x] pgweb #71705
• [x] piknik #71705
• [x] pup #71683
• [x] qpm #71705
• [x] rack #71706
• [x] s-search #71706
• [x] shellz #71684
• [x] sift #71687
• [x] snag #71685
• [x] snap-telemetry #71663
• [x] spaceinvaders-go #71706
• [x] ssh-vault #71708
• [x] stout #71664
• [x] td #71708
• [x] teleconsole #71708
• [x] termshare #71709
• [x] terraform-inventory #71409
• [x] textql #71709
• [x] traefik@1 #71709
• [x] uru #71710
• [x] vert #71710
• [x] vultr #71408
• [x] wego #71710

[chenrui333]

it is also a huge opportunity to disable/remove go formulae now.

we can definitely temporarily to have some go formulae build with [email protected] though.

[nklmilojevic]

I have started from the bottom of the list, for now wego, vert and uru don't have go modules in their repos.

[SMillerDev]

Could you make sure to make an upstream issue for those (mentioning that this will break go 1.16 compatibility) and link them here? That way we can keep track of the status.

[nklmilojevic]

^ Will do it one by one

[nklmilojevic]

@SMillerDev it seems that all of these formulas that are left contain repos that don't have go.mod at all, so that is why there is GOPATH still there for install command. I think these formulas will have to update or keep depending on [email protected].

[MikeMcQuaid]

it is also a huge opportunity to disable/remove go formulae now.

Let's try and only do this for Go formulae that aren't widely used. Remember we're primarily a binary package manager for our users; it doesn't matter to them if we can't build from source.

[vvvvv]

regarding mailhog:
There is already a ticket but the maintainer seems to be pretty unresponsive as the last contribution is from jun 2020

[vvvvv]

regarding termshare:
Last update was 7 years ago, maintainer doesn't answer issues or pr.
The project seems to be pretty much dead so I'd suggest removing it.

[Bo98]

GO111MODULE=auto is probably an acceptable fix that can be done quite quickly for all of the above (can be batch scripted).

I'm quite happy 1.16 is making this change though. Formulae that are not built using Go modules have never had reproducible builds, and have had a history of breaking.

Going forward, we should probably not accept any new formula that uses GO111MODULE=auto/off (adding an audit to prevent it), but that's a debate for later.

[vvvvv]

regarding modd:
HEAD builds just fine as upstream already uses go modules in master not in the latest version.
I don't know what to do in this case, just change the version to commit 8983974 ? There's also already a ticket regarding arm support https://github.com/cortesi/modd/issues/95

[vvvvv]

regarding mpdviz:
Last update was 7 years ago.
Project seems to be dead.

[vvvvv]

regarding path-extractor:
Project seems to be dead.
@chenrui333 filed an issue on 29 Jul 2020 noting the absence of any license and maintainer didn't responde.

[Bo98]

For path-extractor, that is a valid reason to disable the formula. disable! because: :no_license.

[TimothyGu]

I see that people have started to add GO111MODULE=auto to a bunch of packages. However, according to https://blog.golang.org/go116-module-changes, that won't work in Go 1.17:

We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate.

Instead of adding GO111MODULE everywhere, which is only a temporary band-aid, I think it would be better to properly convert formulae to not use GOPATH at all.

[Bo98]

Instead of adding GO111MODULE everywhere, which is only a temporary band-aid, I think it would be better to properly convert formulae to not use GOPATH at all.

For pretty much all of the affected formulae, they don't use Go modules and only work in GOPATH. This requires upstream to release updates to correct. We've already put in a lot of effort to migrate to Go modules and the last group are out of our control.

GO111MODULE is a reasonable workaround so that Go 1.16 can ship this weekend. Otherwise, we're going to have to wait a lot longer while we wait for various updates from upstream.

Once Go 1.16 is shipped, we then have a known timeframe (until 1.17) for the affected formulae to be updated to use Go modules. If upstream do not issue an update to support this, then we will disable those formulae, marking as unmaintained or similar.

[roopakv]

@Bo98 would you be able to update the comment https://github.com/Homebrew/homebrew-core/issues/47627#issuecomment-780944517 to reflect the state of the world :)

more of these have been opened/fixed and if I'm going to be honest I'm confusing myself 😂

[carlocab]

Whoops; I got it. Give me a few minutes.