Feature request: Support Golang multi-module workspaces

[Nr18]

Describe your idea/feature/enhancement

When using Golang for your Lambda functions it's typical to setup a workspace, see: https://go.dev/doc/tutorial/workspaces. What you can do with workspaces is reference to for example a module that might contain some shared logic. Extremely useful for your models that you might use in multiple lambda functions.

Proposal

When you perform sam build the dependencies are downloaded. But when you are referencing a module in the same repository the build will fail. While if you would compile it from the workspace the go.work file is considered before downloading any external dependencies.

Things to consider:

1. Building using go build ./functions/sample-function does work:

   GOOS=linux CGO_ENABLED=0 go build  -o .aws-sam/build/SampleFunctionFunction/bootstrap ./functions/sample-function
   

   However, using the command in a Makefile:

   build-SampleFunctionFunction:
       cd ../../ && GOOS=linux CGO_ENABLED=0 go build -o $(ARTIFACTS_DIR)/bootstrap ./functions/sample-function
   

   Would fail in:

   Building codeuri: /Users/nr18/workspace/prive/golang-sample-multiple/functions/sample-function runtime: provided.al2 metadata: {'BuildMethod': 'makefile'} architecture: arm64 functions:                   
   SampleFunctionFunction                                                                                                                                                                                             
   SampleFunctionFunction: Running CustomMakeBuilder:CopySource                                                                                                                                                       
   SampleFunctionFunction: Running CustomMakeBuilder:MakeBuild                                                                                                                                                        
   SampleFunctionFunction: Current Artifacts Directory : /Users/nr18/workspace/prive/golang-sample-multiple/.aws-sam/build/SampleFunctionFunction                                                              
   cd ../../ && GOOS=linux CGO_ENABLED=0 go build -o /Users/nr18/workspace/prive/golang-sample-multiple/.aws-sam/build/SampleFunctionFunction/bootstrap ./functions/sample-function
   
   Build Failed
   Error: CustomMakeBuilder:MakeBuild - Make Failed: go: go.mod file not found in current directory or any parent directory; see 'go help modules'
   make: *** [build-SampleFunctionFunction] Error 1
   

Additional Details

Template

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: |-
  Sample Serverless Application Model using golang with shared modules
Resources:
  SampleFunctionFunction:
    Type: AWS::Serverless::Function
    Properties:
      Architectures: [ arm64 ]
      Runtime: provided.al2
      CodeUri: ./functions/sample-function
      Handler: bootstrap
      Timeout: 300

go.work

go 1.21.1

use (
	./models
	./functions/sample-function
)

models/go.mod

module example.com/models

go 1.21.1

models/main.go

package models

type Person struct {
	Name string
}

func New(name string) *Person {
	return &Person{Name: name}
}

functions/sample-function/go.mod

module example.com/functions/sample-function

go 1.21.1

require (
	github.com/aws/aws-lambda-go v1.41.0 // indirect
	github.com/aws/aws-sdk-go-v2 v1.21.1 // indirect
	github.com/aws/aws-sdk-go-v2/config v1.18.44 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.13.42 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.12 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.42 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.36 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.44 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.36 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.15.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.17.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.23.1 // indirect
	github.com/aws/smithy-go v1.15.0 // indirect
	github.com/sirupsen/logrus v1.9.3 // indirect
	golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
)

functions/sample-function/main.go

package main

import (
	"context"
	"example.com/models"
	"fmt"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	log "github.com/sirupsen/logrus"
)

func main() {
	log.SetFormatter(&log.JSONFormatter{})
	log.SetLevel(log.DebugLevel)

	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		log.Printf("error: %v", err)
		return
	}

	lambda.Start(New(cfg).Handler)
}


func New(config aws.Config) *Lambda {
	return &Lambda{
		config: config,
	}
}

type Lambda struct {
	ctx    context.Context
	config aws.Config
}

type Request struct{}
type Response struct{}

// Handler of the AWSLambda function.
func (x *Lambda) Handler(ctx context.Context, request Request) (Response, error) {
	person := models.New("John Doe")
	fmt.Printf("Name: %s\n", person.Name)
	return Response{}, nil
}

Build output

Building codeuri: /Users/nr18/workspace/prive/golang-sample-multiple/functions/sample-function runtime: provided.al2 metadata: {} architecture: arm64 functions: SampleFunctionFunction                     
SampleFunctionFunction: Running CustomMakeBuilder:CopySource                                                                                                                                                       
SampleFunctionFunction: Running CustomMakeBuilder:MakeBuild                                                                                                                                                        
SampleFunctionFunction: Current Artifacts Directory : /Users/nr18/workspace/prive/golang-sample-multiple/.aws-sam/build/SampleFunctionFunction                                                              
GOOS=linux CGO_ENABLED=0 go build -o bootstrap

Build Failed
Error: CustomMakeBuilder:MakeBuild - Make Failed: lambda.go:5:2: no required module provides package example.com/models; to add it:
        go get example.com/models
make: *** [build-SampleFunctionFunction] Error 1

[lucashuy]

Thanks for opening this feature request. This might be a result of how the Makefile workflow works as of right now (since it copies the CodeUri into a temporary folder and runs commands from there). Would changing the runtime from provided.al2 to go1.x work for the time being? The go1.x currently builds in the source folder without having to copy files.

[atcherry]

Are there any updates to this? I am not using Makefile as the custom build system nor am I using provided.al2, but I am trying to use go work spaces to no avail.

[KlemenKozelj]

🎉 I believe we successfully resolved this issue using the provided.al2 runtime.

In our repository, we maintain a Go workspace containing 3 Lambdas and 1 module that houses shared code among them:

.
├── go.work
├── go.work.sum
├── samconfig.toml
├── template.yaml
└── lambdas
    ├── commons
    │   ├── aws.go
    │   ├── go.mod
    │   ├── kinesis.go
    │   └── types.go
    ├── lambda-A
    │   ├── Makefile
    │   ├── go.mod
    │   ├── go.sum
    │   ├── main.go
    │   ├── main_test.go
    │   └── kinesis.go
    ├── lambda-B
    │   ├── Makefile
    │   ├── dynamo.go
    │   ├── go.mod
    │   ├── go.sum
    │   ├── main.go
    │   ├── main_test.go
    │   └── dynamo.go
    └── lambda-C
        ├── Makefile
        ├── go.mod
        ├── go.sum
        ├── main.go
        ├── main_test.go
        └── sqs.go

In the go.work file:

go 1.21.6

use (
	./lambdas/commons
	./lambdas/lambda-A
	./lambdas/lambda-B
	./lambdas/lambda-C
)

Here's a snippet from template.yaml, where Lambdas are built using the makefile build method:

LambdaA:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: makefile
    Properties:
      CodeUri: lambdas/lambda-A/
      Environment:
        Variables:
          KINESIS_STREAM_NAME: !Ref KinesisStream
      Events:
        Api:
          Type: Api
          Properties:
            Path: /somepath
            Method: POST
            RestApiId: !Ref ApiGateway
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action: kinesis:PutRecord*
              Resource: !GetAtt KinesisStream.Arn

As mentioned by @lucashuy in a previous response, the makefile copies content from CodeUri into a temporary directory. To address this, we modified the makefile for each Lambda:

build-LambdaA:
	rm -rf `pwd`/*
	cp $(PWD)/go.* `pwd`
	cp -R $(PWD)/thryve `pwd`
	GOOS=linux go build -o bootstrap ./lambdas/lambda-A/
	cp ./bootstrap $(ARTIFACTS_DIR)/.

pwd refers to the current temporary directory where SAM executes the build, and PWD refers to the root of the project where template.yaml is located.

We haven't explored the solution with the go1.x runtime as it has been deprecated. More details can be found here.

I hope others facing a similar issue find this helpful! Let me know if you have any further questions.

Best regards, Klemen 🚀

[Nr18]

Tried your option and did not get it to work that easy... Do you have a working sample on GitHub by any chance that I can look at?