Golang environment in Docker

[dhaavi]

trafficstars

Prerequisites

• [x] Have you tried launching atom . from the terminal in your project's directory?
• [x] Have you verified the output from go env is correct? If it is, please include the output in this issue.
• [x] Have you updated Atom to the latest version?
• [ ] Have you tried using Atom Beta, which can be run side-by-side with Atom Stable? Is the behavior the same, or different?
• [x] Have you updated your Atom packages to the latest versions?
• [x] Have you read the FAQ?
• [x] Have you searched the issues to see if others are experiencing the same issue?

Description

I am trying to run the complete golang environment in docker, while integrating atom/go-plus into it.

Host: MacOS Mojave Container: ubuntu:latest

I imagine this to work as follows:

• go-plus searches for the go binary or other go tools in PATH.
• go-plus finds what it is looking for and executes the tool.
• But the tool is just a wrapper script that runs the command in an already running docker container.
• The docker container is completely set up with the go environment and everything works fine in the container.
• The project paths outside/inside container are the same and the network is shared with the host for maximum compatibility (for the beginning).

The problem:

go-plus does not pick up the wrapper scripts in the PATH, it gives me the "Missing Go Tool" error.

I added the directory with the wrapper scripts to the atom environment by adding

process.env.PATH = [process.env.PATH, '/Users/[REDACTED]/fakego/bin'].join(':')

to init.coffee.

I also verified that the PATH is correct in the Atom Dev Console:

> process.env.PATH
"/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:[REDACTED]/fakego/bin"

The directory contains wrapper scripts for addr2line, api, asm, buildid, cgo, compile, cover, dist, doc, fix, go, godoc, gofmt, link, nm, objdump, pack, pprof, test2json, tour, trace, vet and yacc.

I also tried to set GOPATH and/or GOROOT there, as well as in the package config interface, to no avail.

Output from atom -v && apm -v

On host:

Atom    : 1.32.2
Electron: 2.0.9
Chrome  : 61.0.3163.100
Node    : 8.9.3
apm  2.1.2
npm  6.2.0
node 8.9.3 x64
atom 1.32.2
python 2.7.10
git 2.17.2

Output From go env

Ran on host, proxied into container:

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/.go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build094912619=/tmp/go-build -gno-record-gcc-switches"

Steps to Reproduce

Will write if necessary, as it's rather complex.

Expected Behavior

go-plus to find the go wrapper script.

Actual Behavior

go-plus gives me the "Missing Go Tool" error.

Related Issues

https://github.com/joefitzgerald/go-plus/issues/786 https://github.com/joefitzgerald/go-plus/issues/523

[dhaavi]

I now also tried to link /usr/local/go or /usr/local/go/bin to my faked environment. No success. I'd like to understand how go-plus finds/validates the go binary and tools.

[zmb3]

What you're doing should work, but you'll probably have to set GOOS=linux as well.

For the go tool, go-plus searches in the following order:

• $GOROOT/bin
• Atom configuration
• $PATH
• Well known install locations (C:\Go\bin, /usr/local/go, /usr/local/bin, etc.)

Once we've found a Go tool, we must be able to execute both go version and go env -json successfully in order to consider it a viable option.

Have you tried setting your $PATH when you invoke Atom instead of in init.coffee?

GOOS=linux PATH=/Users/[REDACTED]/fakego/bin:$PATH atom .

For other tools, we look at $GOPATH/bin first, and check $PATH if nothing is found.

[dhaavi]

Interesting, it then really should work. I just tried (on MacOS host):

$ GOOS=linux PATH=/Users/[REDACTED]/fakego/bin:$PATH atom .
$ # no success, try another way and check commands:
$ export PATH=/Users/[REDACTED]/fakego/bin:$PATH
$ go version
go version go1.10.4 linux/amd64
$ go env -json
{
	"CC": "gcc",
	"CGO_CFLAGS": "-g -O2",
	"CGO_CPPFLAGS": "",
	"CGO_CXXFLAGS": "-g -O2",
	"CGO_ENABLED": "1",
	"CGO_FFLAGS": "-g -O2",
	"CGO_LDFLAGS": "-g -O2",
	"CXX": "g++",
	"GCCGO": "gccgo",
	"GOARCH": "amd64",
	"GOBIN": "",
	"GOCACHE": "/home/user/.cache/go-build",
	"GOEXE": "",
	"GOGCCFLAGS": "-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build453410605=/tmp/go-build -gno-record-gcc-switches",
	"GOHOSTARCH": "amd64",
	"GOHOSTOS": "linux",
	"GOOS": "linux",
	"GOPATH": "/home/user/.go",
	"GORACE": "",
	"GOROOT": "/usr/lib/go-1.10",
	"GOTMPDIR": "",
	"GOTOOLDIR": "/usr/lib/go-1.10/pkg/tool/linux_amd64",
	"PKG_CONFIG": "pkg-config"
}
$ export GOOS=linux
$ atom .

What happened when I opened Atom:

• Immediately got the Missing Go Tool error. (It reopened a .go file)
• I focused the tab with the .go file:

what.js? [sm]:135 missing guru tool

• I saved to .go file:

formatter.js? [sm]:194 skipping format, could not find tool goimports
builder.js? [sm]:69 Uncaught (in promise) Error: cannot find go tool
    at Builder.build (builder.js? [sm]:61)
    at <anonymous>
tester.js? [sm]:342 Uncaught (in promise) Error: cannot find go executable
    at Tester.runTests (tester.js? [sm]:396)
    at <anonymous>

Tested command availability (continuing session from before):

$ guru
Run 'guru -help' for more information.
$ goimports #blocks
^C
$

Also tried:

• just updated to go-plus 5.9.1
• reinstalled go-plus again

---

I then grew curious of other sources of trouble and tried to execute the commands from within Atom. By trying that, I solved a first problem that may have to do with all of this: I had to change my wrapper scripts to not use the -ti flag for docker exec, but only -i. For reference, the full wrapper command now is:

docker exec -i --user=$(id -u):$(id -g) -w /home/user $container go $@

Check that commands work (ish) in Atom Dev Console:

> const execSync = require('child_process').execSync;
> execSync('go version', { encoding: 'utf-8' });
"go version go1.10.4 linux/amd64
"
> execSync('go env -json', { encoding: 'utf-8' });
"{
	"CC": "gcc",
	"CGO_CFLAGS": "-g -O2",
	"CGO_CPPFLAGS": "",
	"CGO_CXXFLAGS": "-g -O2",
	"CGO_ENABLED": "1",
	"CGO_FFLAGS": "-g -O2",
	"CGO_LDFLAGS": "-g -O2",
	"CXX": "g++",
	"GCCGO": "gccgo",
	"GOARCH": "amd64",
	"GOBIN": "",
	"GOCACHE": "/home/user/.cache/go-build",
	"GOEXE": "",
	"GOGCCFLAGS": "-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build321977714=/tmp/go-build -gno-record-gcc-switches",
	"GOHOSTARCH": "amd64",
	"GOHOSTOS": "linux",
	"GOOS": "linux",
	"GOPATH": "/home/user/.go",
	"GORACE": "",
	"GOROOT": "/usr/lib/go-1.10",
	"GOTMPDIR": "",
	"GOTOOLDIR": "/usr/lib/go-1.10/pkg/tool/linux_amd64",
	"PKG_CONFIG": "pkg-config"
}
"
> execSync('guru', { encoding: 'utf-8' });
child_process.js:656 Uncaught Error: Command failed: guru
[...]
> execSync('guru --help', { encoding: 'utf-8' });
child_process.js:656 Uncaught Error: Command failed: guru --help
Run 'guru -help' for more information.
Go source code guru.
Usage: guru [flags] <mode> <position>
[...]
> execSync('goimports', { encoding: 'utf-8' }); // does not block, exits after a short duration (~1s)
""
> execSync('goimports --help', { encoding: 'utf-8' });
child_process.js:656 Uncaught Error: Command failed: goimports --help
usage: goimports [flags] [path ...]
[...]

After having gone through that, the Missing Go Tool error has disappeared and various others have appeared... so yay! progress!: https://www.commitstrip.com/en/2018/05/09/progress/

I will post an update as soon as I have resolved all issues or have reached another roadblock.

PS: Missing Go Tool has been a bit deceptive - as it just failed in that specific environment.

[zmb3]

Thanks for all this good info. I will try to reproduce locally as well.

FYI: I would not necessarily expect execSync('gomiports') to work, as that's bypassing go-plus' locator.

[dhaavi]

I put together a small repo to help you reproduce the issues: https://github.com/dhaavi/go-in-docker

At least I got goimports to work (Go code is formatted after save), so the basics seem fine. Interestingly though, I did not get the orange-ish error popups, as in my original environment, but the errors should be the same.

Will investigate further tomorrow.

[zmb3]

That helps a lot. Found a few issues so far:

1. Our logic for finding the go tool was a little silly. We found the tool, then executed go env to find its $GOROOT, and then checked $GOROOT/bin for the go tool. This is an easy fix that I'll get in shortly.
2. We run go install -i . on save, but the container has its own $GOPATH (which doesn't match that of the host). I think you'll need to update the scripts to preserve the host's setting (using go env GOPATH to also pick up the default, unset GOPATH).
3. go-plus sets the working directory to the project directory for all commands that it runs, but wrapper scripts hard-code the working directory to /home/user, so most commands fail. The working directory needs to be preserved inside the container.
4. As of Go 1.10+, Go uses a build cache. The go install command failed as it can't create a directory for this cache. You may need to override GOCACHE to point to a writable location inside the container.

"go: disabling cache (/.cache/go-build) due to initialization failure: mkdir /.cache: permission denied go install github.com/pkg/errors: mkdir /Users/zmb3/go/pkg: permission denied"

[dhaavi]

Thank you for your input.

1. Our logic for finding the go tool was a little silly. We found the tool, then executed go env to find its $GOROOT, and then checked $GOROOT/bin for the go tool. This is an easy fix that I'll get in shortly.

Great! Thanks!

2. We run go install -i . on save, but the container has its own $GOPATH (which doesn't match that of the host). I think you'll need to update the scripts to preserve the host's setting (using go env GOPATH to also pick up the default, unset GOPATH).

As I understand it, the $GOPATH env variable should never be used on the Host, except you use $GOPATH directly in go-plus. Anyway, I executed go install -i . in the project directory on the host, and it perfectly ran inside the container.

3. go-plus sets the working directory to the project directory for all commands that it runs, but wrapper scripts hard-code the working directory to /home/user, so most commands fail. The working directory needs to be preserved inside the container.

Fixed.

4. As of Go 1.10+, Go uses a build cache. The go install command failed as it can't create a directory for this cache. You may need to override GOCACHE to point to a writable location inside the container.

Fixed.

---

With these changes, something changed:

• I do not get the Missing Go Tool error anymore
• When I save a file, it does not format it anymore - interestingly, it did before...
• 10 seconds after a save (atom is unresponsive during that time), these errors pop up, but I can't get any value out of them - I think it wants to run go test and goimports:

tester.js? [sm]:398 Uncaught (in promise) Error
    at Tester.runTests (/Users/[user]/.atom/packages/go-plus/lib/test/tester.js:398)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188)
builder.js? [sm]:105 Uncaught (in promise) Error
    at Builder.build (builder.js? [sm]:99)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188)

• Also, in the "info" tab of go-plus, go version and go env show an empty output:

$ go version

$ go env

• In the container there are multiple instances of goimports and gocode - maybe idling around or abondoned?

• On an attempt to trigger autocomplete, this gocode error appeared:

flag provided but not defined: -unimported-packages

Am I maybe using a wrong version/package? But other than that, this indicates that the gocode daemon is - kind of - successfully running inside docker and that communication works. (?)

(I have not yet update the go-in-docker repo with the new changes, will do so tomorrow.)

[zmb3]

As I understand it, the $GOPATH env variable should never be used on the Host, except you use $GOPATH directly in go-plus.

I'm not sure what you mean here. GOPATH must absolutely be preserved else the code won't build inside the container.

10 seconds after a save (atom is unresponsive during that time), these errors pop up, but I can't get any value out of them

Likely related to the lack of a GOPATH in the container.

Also, in the "info" tab of go-plus, go version and go env show an empty output:

I am seeing this as well, will continue to look into it.

In the container there are multiple instances of goimports and gocode - maybe idling around or abondoned?

Not sure what's going on here, but could also be explained by the lack of a GOPATH.

flag provided but not defined: -unimported-packages

This flag was recently (re)added to gocode. Please update the gocode tool inside the container: go get -u github.com/mdempsky/gocode

If you ping here when you've updated the go-in-docker repo I'm happy to take another look. That was super helpful in allowing me to investigate what's going on here :-)

[dhaavi]

I monkey-patched your fix for finding the go tool and made some progress.

• goimports seems to work fine, it formats the code on save (that was my bad)
• saving no longer makes atom hang for 10 seconds (also my bad)
• gocode works a little more: it gives me suggestions for packages and built-ins, but not for anything else, no errors.
• go test seems to work, with one exception: the temporary path created for "coverage.out" of course is not the same on the host and the container. Providing a custom, shared path via the go-plus settings does not work, as -coverprofile= is then supplied twice.

So only two errors are left at the moment:

• gocode does not suggest package contents
• go test needs a shared path to write/read coverage.out

About the $GOPATH: What I meant, is, that $GOPATH is only ever needed inside the container. I had it only set in the container (to /home/user/go). I now also set it on the host to that same value, even though that path does not exist.

If you check out the go-in-docker repo, please re-follow all the steps, as many small things changed.

[dhaavi]

I played around with a couple different packages now. gocode does work in other circumstances. Will further update this issue with more details when I get an understanding of the circumstances that cause gocode not to work.

[dhaavi]

Fixed another issue in https://github.com/dhaavi/go-in-docker

[dhaavi]

Pull request https://github.com/joefitzgerald/go-plus/pull/835 in combination with "Additional test args" set to -coverprofile=/tmp/dockenv/go/coverage.out fixes coverage. Atom now correctly shows test coverage.

[dhaavi]

I am further updating https://github.com/dhaavi/go-in-docker as I find and fix issues. It already works pretty well.

There is still an issue with gocode: It still does not do suggests for other internal packages (ie. the same github path), sometimes suggesting things from another, unimported package.

The second issue I solved with https://github.com/joefitzgerald/go-plus/pull/835. It would be nice to have a final solution here, as I don't want to monkey-patch this every time go-plus updates.