AWS Lambda: panic: cannot statfs cgroup root: no such file or directory

[joshq00]

Describe the bug Running in AWS lambda, the application cannot start due to panic: cannot statfs cgroup root: no such file or directory from go/pkg/mod/github.com/zeromicro/[email protected]/core/stat/internal/cgroup_linux.go:191

See opencontainers/runc#2634 - they don't execute this code on init

To Reproduce Steps to reproduce the behavior, if applicable:

1. The code is

package main

import (
	"context"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/zeromicro/go-zero/core/stat"
)

func main() {
	stat.CpuUsage()
	lambda.Start(eventHandler)
}

type Event struct{}

func eventHandler(ctx context.Context, ev Event) error {
	return nil
}

2. The error is

panic: cannot statfs cgroup root: no such file or directory
--
goroutine 1 [running]:
github.com/zeromicro/go-zero/core/stat/internal.isCgroup2UnifiedMode.func1()
/Users/josh/go/pkg/mod/github.com/zeromicro/[email protected]/core/stat/internal/cgroup_linux.go:191 +0x13a
sync.(*Once).doSlow(0xc00008dcf0?, 0x0?)
/opt/homebrew/Cellar/go/1.18.1/libexec/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
/opt/homebrew/Cellar/go/1.18.1/libexec/src/sync/once.go:59
github.com/zeromicro/go-zero/core/stat/internal.isCgroup2UnifiedMode()
/Users/josh/go/pkg/mod/github.com/zeromicro/[email protected]/core/stat/internal/cgroup_linux.go:182 +0x31
github.com/zeromicro/go-zero/core/stat/internal.currentCgroup()
/Users/josh/go/pkg/mod/github.com/zeromicro/[email protected]/core/stat/internal/cgroup_linux.go:39 +0x17
github.com/zeromicro/go-zero/core/stat/internal.cpuSets()
/Users/josh/go/pkg/mod/github.com/zeromicro/[email protected]/core/stat/internal/cpu_linux.go:112 +0x19
github.com/zeromicro/go-zero/core/stat/internal.init.0()
/Users/josh/go/pkg/mod/github.com/zeromicro/[email protected]/core/stat/internal/cpu_linux.go:27 +0x1d
2022/06/14 21:51:22 exit status 2

Expected behavior Running application

Screenshots

Environments (please complete the following information):

• OS: Linux
• go-zero version: 1.3.4
• goctl version: 1.3.4

More description Using Go 1.18 on AWS Lambda

The code in question is

func isCgroup2UnifiedMode() bool {
	isUnifiedOnce.Do(func() {
		var st unix.Statfs_t
		err := unix.Statfs(cgroupDir, &st)
		if err != nil {
			if os.IsNotExist(err) && runningInUserNS() {
				// ignore the "not found" error if running in userns
				isUnified = false
				return
			}
			panic(fmt.Sprintf("cannot statfs cgroup root: %s", err))
		}
		isUnified = st.Type == unix.CGROUP2_SUPER_MAGIC
	})

	return isUnified
}

[kevwan]

Thanks for your PR!

Does this PR fix the problem?

What's the version of the linux kernel?

If no statfs with the kernel version, load it later won't fix the problem?

[joshq00]

It runs with this PR, but CPU reports as 0.

I'm not sure what kernel, we don't really have control over anything in the lambda environment.

[kevwan]

Can you run commands on the system? like head -n1 /proc/stat.

[joshq00]

I don't have access to my computer at the moment but I see lambda runs on "Amazon Linux" os (not second generation)

Amazon Linux:1 I think this docker image should be the equivalent

[kevwan]

I don't have access to my computer at the moment but I see lambda runs on "Amazon Linux" os (not second generation)

Amazon Linux:1 I think this docker image should be the equivalent

I tried with this docker image, it works with CPU as 0.

What problem are we going to fix? Not working or CPU as 0?

[joshq00]

This PR fixes that the initial panic from import prevents the lambda from running. CPU 0 is acceptable as it would report 0 anyway. This allows the lambda to run

[kevwan]

Amazon Linux:1

With this docker image, and not including this PR, I can run it with reporting CPU 0.

[joshq00]

You can repeat it if you deploy the code in the original post to lambda (free). It panics on import and you cannot recover.

[kevwan]

Would you please give me a lambda link? I have AWS servers, but I never used lambda services. Thanks!

[joshq00]

Here you go

Just build your binary with goos Linux and goarch amd64, put it in a zip file, and that's your artifact. You can invoke it directly from the Lambda screen.

One thing to add: try adding a recover to the original code. I think it will fail prior to this PR and succeed with this PR. My use case is using the go zero service so it has the panic handler built in

[joshq00]

Anything else I can provide here? Hopefully you were able to confirm that panic on init is not recoverable.

[kevwan]

Anything else I can provide here? Hopefully you were able to confirm that panic on init is not recoverable.

I'll try it this weekend in AWS lambda.

[joshq00]

Bump

[joshq00]

Have you tried this yet? We're forced to use my fork until this is merged

[kevwan]

Have you tried this yet? We're forced to use my fork until this is merged

Thanks for your issue & PR.