go.work without go.mod in workspace fails with `failed to teardown GOPATH: unlinkat ...`

[betamos]

A typical go.work setup (Go 1.18+ Workspaces) causes the build to fail in the teardown step, when running with BP_TARGETS="./sub".

It appears the builder only checks for go.mod (and not go.work), so the builder assumes a pre-module (legacy) GOPATH app.

Workaround

My deployment works by adding an empty wrapper go.mod in the project root: go mod init wrapper

This tricks the buildpack into assuming the (modern) go module build, which coincidentally fixes the build without breaking the go build command (in my case).

Expected Behavior

Build should succeed.

Current Behavior

latest: Pulling from paketobuildpacks/builder
Digest: sha256:efdf0c991248fbbf949bf33ff8752b1ba22828d5a0f7c4481608172d0011ee1d
Status: Image is up to date for paketobuildpacks/builder:latest
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:960420f8e05845e9acbc4e04de77dc852ca0823bbdf640d9e9f4131a89f8d1fe
Status: Image is up to date for paketobuildpacks/run:base-cnb
0.15.3: Pulling from buildpacksio/lifecycle
Digest: sha256:e2a5fa066b2070fa9dc0837cd09a15b2e404adbac3e2f94c8ea99a8cbe0045d3
Status: Image is up to date for buildpacksio/lifecycle:0.15.3
===> ANALYZING
[analyzer] Restoring data for SBOM from previous image
===> DETECTING
[detector] 3 of 8 buildpacks participating
[detector] paketo-buildpacks/ca-certificates 3.5.1
[detector] paketo-buildpacks/go-dist         2.2.4
[detector] paketo-buildpacks/go-build        2.0.9
===> RESTORING
[restorer] Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
[restorer] Restoring metadata for "paketo-buildpacks/go-dist:go" from cache
[restorer] Restoring metadata for "paketo-buildpacks/go-build:targets" from app image
[restorer] Restoring metadata for "paketo-buildpacks/go-build:gocache" from cache
[restorer] Restoring data for "paketo-buildpacks/go-dist:go" from cache
[restorer] Restoring data for "paketo-buildpacks/go-build:gocache" from cache
[restorer] Restoring data for SBOM from cache
===> BUILDING
[builder] 
[builder] Paketo Buildpack for CA Certificates 3.5.1
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Launch Helper: Reusing cached layer
[builder] Paketo Buildpack for Go Distribution 2.2.4
[builder]   Resolving Go version
[builder]     Candidate version sources (in priority order):
[builder]       <unknown> -> ""
[builder] 
[builder]     Selected Go version (using <unknown>): 1.18.10
[builder] 
[builder]   Executing build process
[builder]     Installing Go 1.18.10
[builder]       Completed in 4.059s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_go-dist/go
[builder]       Completed in 0s
[builder] 
[builder] Paketo Buildpack for Go Build 2.0.9
[builder]   Executing build process
[builder]     Running 'go build -o /layers/paketo-buildpacks_go-build/targets/bin -buildmode pie -trimpath ./sub'
[builder]       Completed in 7.42s
[builder] 
[builder] failed to teardown GOPATH: unlinkat /tmp/gopath4127721619/pkg/mod/github.com/gorilla/[email protected]/example_route_test.go: permission denied
[builder] ERROR: failed to build: exit status 1

Possible Solution

Perhaps add a check for presence of go.work in the path manager.

Steps to Reproduce

go.work in app root:

go 1.19

use ./sub

The sub directory contains the simple sample app (which has its own go.mod file, see samples/go/mod)

Motivations

This took an unreasonable amount of time to track down, perhaps because I'm unused to buildpacks. I was deploying with fly.io and thought it was a problem on their end until I finally came down to the buildpack itself. Even then I wasn't sure if it was a permission error with my files. It would be helpful if the diagnostics could help differentiate between internal errors in the buildpack vs external errors (which I assume are much more common day-to-day).

Also, I still don't know how to test the fix locally. How can I do that?

[ryanmoran]

I'm interested to hear what your use-case is for using workspaces when you deploy your code. My understanding was that the workspaces feature was mostly to help out when you need to debug or patch dependencies locally, but that you'd switch to a replace directive when you really needed to deploy, maybe also pulling in a patched dependency once the change was in upstream.

Can you provide some more context on why this works best for you?

[betamos]

My understanding was that the workspaces feature was mostly to help out when you need to debug or patch dependencies locally

That's certainly one use-case, but deploying can be done to dev/staging environments as well. Are buildpacks not designed for local dev? I had planned to use Docker Compose to spin up multiple services locally, for higher fidelity and ease of lifecycle management.

you'd switch to a replace directive when you really needed to deploy

IIUC using go.work and using replace-directives are orthogonal - you can use replace within a go.work file or directly in the go.mod file.

Can you provide some more context on why this works best for you?

I have a mixed-language monorepo with some local modules, and some public. Git subtree is used to sync the public modules to repos as regular modules with their own go.mod files. That said, I don't expect anything different than what the go toolchain provides out of the box.

Here's the rationale from the design doc. Monorepos are mentioned here.

[phroggyy]

I'm running into this issue as well. I have services that import from other services (to get their gRPC service definition), a go.work in the root, and replace directives pointing to sibling directories. In order to pass the proper build context to buildpack (also doing it through fly, same as @betamos), the builder has to run in the monorepo root, as only the built service's codebase gets sent to the Docker context otherwise.

TL;DR: I need to pass multiple service directories into the docker build context in order for the build to work. The most straightforward way I've found (there may be options I've missed) is to mount the whole monorepo (though inefficient) into the context and build in a directory.