"filename or extension is too long" when building many source files

[pedxyuyuko]

What version of Garble and Go are you using?

$ garble version
mvdan.cc/garble v0.7.0

Build settings:
       -compiler gc
     CGO_ENABLED 0
          GOARCH amd64
            GOOS windows
         GOAMD64 v1
$ go version
go version go1.18.2 windows/amd64

What environment are you running Garble on?

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Administrator\AppData\Local\go-build
set GOENV=C:\Users\Administrator\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\Storage\Caches\gopath\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\Storage\Caches\gopath
set GOPRIVATE=
set GOPROXY=https://goproxy.cn,direct
set GOROOT=D:\Storage\Go-1.18.2\go1.18.2
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\Storage\Go-1.18.2\go1.18.2\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18.2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=D:\GoProject\Test\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\ADMINI~1\AppData\Local\Temp\go-build464808010=/tmp/go-build -gno-record-gcc-switches

What did you do?

I tried to complie a project with lib github.com/gotd/td but there is a error occure.

go build github.com/gotd/td/tg: D:\Storage\Caches\gopath\bin\garble.exe: fork/exec D:\Storage\Caches\gopath\bin\garble.exe: The filename or extension is too long.

What did you expect to see?

Compile successfully

What did you see instead?

Error above

I notice that the command works fine in Linux. Then I found out that the error The filename or extension is too long will be throw when a command has more than 32k byte exec in Windows. This probably is Windows' fault but I still want to post this here and try to find solution together!

Any help would be much appreciated! English is not my first language, sorry for that.

[mvdan]

Thanks for reporting. I think this is because we replace paths to Go source with temporary paths, so the command

/GOROOT/pkg/tool/linux_amd64/compile [flags] /your/pwd/foo.go /your/pwd/bar.go /your/pwd/baz.go

becomes

/GOROOT/pkg/tool/linux_amd64/compile [flags] /tmpdir/garble-shared12345678/package/import/path/foo.go /tmpdir/garble-shared12345678/package/import/path/bar.go /tmpdir/garble-shared12345678/package/import/path/baz.go

You can see how this can balloon the size of each invocation of the compiler, which is likely how you are running into a command size limit of 32KiB.

It seems like some regular Go builds can already run into this, like https://github.com/golang/go/issues/47235.

I'm not sure if we have a good and easy fix for this. We don't want to modify the source files in-place, because that could easily lead to data loss or races with other processes. We likely don't want to os.Chdir into the temporary directory, as that might break the Go toolchain, but perhaps we could try.

[mvdan]

Ah, upstream Go does support "response files" as arguments to the toolchain, which are just lists of arguments written to a single file to sidestep the limit. See https://github.com/golang/go/issues/18468.

We're running into the issue because go build will only use these if the argument list reaches ~30KiB. But if go build invokes compile with ~20KiB of arguments it won't choose to use a response file, and our replacing of paths could expand those ~20KiB to be large enough to go over the system's limit.

[mvdan]

I think our best bet is to copy upstream's code to replace argument lists with response files. I can look into it when I have the time.

[mvdan]

Sorry, what do you mean? This is about go build -toolexec not supporting very long argument lists. This requires a bug fix in upstream Go.