Automatically set GOMAXPROCS to match Linux container CPU quota

[alphastorm]

See https://github.com/uber-go/automaxprocs for details about the library.

See https://github.com/prysmaticlabs/prysm/pull/2770 for an example PR of this getting merged to the Prysmatic Labs ETH2 beacon chain and validator clients.

[Shadowfiend]

Compiling in an external package that has unlimited access to our memory just to set an environment variable seems odd… I'm interested in finding a better way to get this functionality that more clearly separates the environment concern from the client. Maybe we can pull this lib into a separate executable that can spit out the relevant value rather than interfere in-memory, and package that into the container as something that can be run during startup.

[alphastorm]

Compiling in an external package that has unlimited access to our memory just to set an environment variable seems odd… I'm interested in finding a better way to get this functionality that more clearly separates the environment concern from the client.

Importing the package triggers init() which shells out to maxprocs.Set(), which calculates the optimal GOMAXPROCS considering cgroups / calls runtime.GOMAXPROCS() / returns. I wouldn't call this unlimited access since the library's lifecycle is fairly limited.

Maybe we can pull this lib into a separate executable that can spit out the relevant value rather than interfere in-memory, and package that into the container as something that can be run during startup.

That seems needlessly complex, as it would require either using a wrapper script or process manager like supervisord (see https://docs.docker.com/config/containers/multi-service_container/). This library has been OSS since August 2017 and is pretty widely used. Longer term the hope is to add mainline support (see https://github.com/golang/go/issues/33803).

Whatever you choose to do, the current default behavior can cause (and is likely causing) performance degradation in affected configurations (from https://github.com/uber-go/automaxprocs/issues/12):

If there's misalignment between CFS quota, the language runtime tuning (in this case GOMAXPROCS) and the workload is mostly CPU bound (e.g. spawning a lot of active goroutines, calculations, etc.) then this could cause performance degradation.

There are numbers from synthetic and real-world benchmarks in the linked issue. The guy quoted above ran a synthetic benchmark calculating prime numbers 😄

Useful thread with a lot more references: https://threadreaderapp.com/thread/1149654812595646471.html