EC2 service is too big to compile on smaller machines

[tenorwill]

Describe the bug

I'm experiencing a problem building a simple getec2 program that describes ec2 instances. Using the go build -v main.go results in this:

github.com/aws/aws-sdk-go-v2/service/ec2: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed

Sometimes the signal is not killed and it simply hangs. If I don't interrupt with ctrl-c the program consumes all memory on the host and it dies. It only happens with the ec2 service.

Build

go build -v .
github.com/aws/aws-sdk-go-v2/service/ec2

After waiting at the terminal in a second window running top/htop, i slowly see the memory and CPU creep to 100% utilization causing the server to crash.

Build Server OS

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)

Go Version (installed using yum)

go version go1.20.7 linux/amd64

Here's the code I'm using:

File: main.go

package main

import (
    "context"
    "fmt"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/ec2"
)

func main() {

    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        fmt.Println("Error loading AWS configuration:", err)
        return
    }


    client := ec2.NewFromConfig(cfg)


    input := &ec2.DescribeInstancesInput{}


    result, err := client.DescribeInstances(context.TODO(), input)
    if err != nil {
        fmt.Println("Error describing EC2 instances:", err)
        return
    }


    for _, reservation := range result.Reservations {
        for _, instance := range reservation.Instances {
            fmt.Printf("Instance ID: %s\n", *instance.InstanceId)
            fmt.Printf("Instance Type: %s\n", *instance.InstanceType)
            fmt.Printf("Public IP: %s\n", *instance.PublicIpAddress)
            fmt.Printf("Private IP: %s\n", *instance.PrivateIpAddress)
            fmt.Printf("State: %s\n", instance.State.Name)
            fmt.Println("--------")
        }
    }
}

Expected Behavior

The program builds as expected. It seems to only fail when building on an AWS EC2 instance - local building works on MacOS or Windows. I also tried from a different distribution (Ubuntu 22.04 for example). Similarly, fails on the EC2 instance running that OS. Interestingly, it does NOT fail when running it on a local VMware ESXi VM. It only seems to occur on an AWS EC2 instance and only recently.

Current Behavior

The program hangs indefinitely, causes CPU and memory to spike, crashes the server

Using the go build -v main.go results in this:

github.com/aws/aws-sdk-go-v2/service/ec2: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed

Reproduction Steps

• Start an EC2 instance
• install Go using yum: 'yum install go -y'
• use the provided code in a single main.go file (or similar to describe ec2 instances)
• build the program using go build -v main.go
• observe the error github.com/aws/aws-sdk-go-v2/service/ec2: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed

Possible Solution

unknown

Additional Information/Context

Until recently, this program worked perfectly. It's odd that it only appears to be service / ec2 and not other AWS SDK services.

AWS Go SDK V2 Module Versions Used

module getec2

go 1.20

require (
	github.com/aws/aws-sdk-go-v2 v1.21.0
	github.com/aws/aws-sdk-go-v2/config v1.18.42
	github.com/aws/aws-sdk-go-v2/service/ec2 v1.122.0
)

require (
	github.com/aws/aws-sdk-go-v2/credentials v1.13.40 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.11 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.41 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.35 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.43 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.35 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.14.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.17.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.22.0 // indirect
	github.com/aws/smithy-go v1.14.2 // indirect
	github.com/jmespath/go-jmespath v0.4.0 // indirect
)

Compiler and Version used

go version go1.20.7 linux/amd64

Operating System and version

Amazon Linux Version 2

[RanVaknin]

Hi @tenorwill ,

Thanks for opening this issue.

I tested it with a t2.micro and ran into the same issue. After changing the instance type to something beefier (m5.large) this was built quickly.

Until recently, this program worked perfectly. It's odd that it only appears to be service / ec2 and not other AWS SDK services.

Your observation is valid. One possible reason I can think of, is that ec2 released more features hence the source code grew, thats why you were having a hard time building this with smaller instance type.

Thanks for bringing this to our attention, I will review this with the team. Ran~

[tenorwill]

Hi @RanVaknin,

Awesome, thank you kindly for verifying/validating. My experience was similar on local workstations - a MBP with loads of RAM/CPU or a local VM with more than 2GB RAM resulted in the build succeeding. But anything 2GB or less would eventually fail or result in a system crash. Hopefully it's a quick fix for you and the team.

[RanVaknin]

Hi @tenorwill,

We acknowledge the difficulty you’re facing with the compilation of the EC2 service in the Go SDK on smaller EC2 instances. We’re actively discussing how to address the growing complexity and size of the EC2 service model across multiple SDKs. There have been similar experiences with other language SDKs, and we’re investigating several strategies to mitigate this.

This is a complex issue that would require an involved coordination between the EC2 and SDK team, and will take time. For now, we recommend using an instance with higher resources for the compilation as a temporary workaround. We apologize for any inconvenience this may have caused and thank you for your patience as we work towards a long-term solution.

Thanks again, Ran~

[tenorwill]

Thanks @RanVaknin! I will change my workflow to compile my program on a more powerful machine first, then transfer the binary file to my server

[FlorianSW]

@RanVaknin hi there 👋 Just wanted to slide into this discussion as well. With one of the recent SDK Updates to the EC2 service we encounter several of our CodeBuild projects failing with the same error message as well (only if EC2 is one of the vendored SDK services). We used to use a ARM BUILD_GENERAL1_SMALL compute type for our projects. The repositories itself (without deps) are small to medium sizes, nothing too fancy I would say. With the latest version of the EC2 service SDK we need at least the BUILD_GENERAL1_MEDIUM compute type for it to succeed.

Just wanted to let you know here that it now also affects the smallest code build compute environments as well, not just EC2 instances 👍 maybe there were some ideas on how this could be solved already?

[lucix-aws]

Ran is no longer working on the Go SDK team.

[lucix-aws]

The only way to really fix this is for the EC2 service to break up their API model and vend smaller targeted clients. There are no active plans to do that as far as I'm aware.