ruby-build not utilizing all cores by default

[varungandhi-src]

Steps to reproduce the behavior

❯ rbenv --version
rbenv 1.2.0
❯ ruby-build --version
ruby-build 20220726

❯ env - PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" RBENV_ROOT="slow" rbenv install 2.7.2
# Finishes in 4m 8s
# CPU usage seems to hover around 60%-70% utilization for 3-4 CPUs and less for others

❯ env - PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" MAKEOPTS="-j 10" RBENV_ROOT="fast" rbenv install 2.7.2
# Finishes in 2m 44s
# CPU usage is maxed out during build steps

This is on a 10 core Macbook Pro on a slow-ish network (otherwise the % speed difference would be more dramatic).

The sysctl in that code does work:

❯ sysctl -n hw.ncpu
10

Expected vs. actual behavior

It looks like a patch about 9 years ago did add support for detecting number of CPUs. https://github.com/rbenv/ruby-build/commit/6c08c56d3aa0add92c60653744324e7cdb3b24c3

Is it not getting plumbed through correctly? Or is the CPU usage being limited deliberately?

Logs

n/a

[mislav]

Hi, in cases when ruby-build needs to compile OpenSSL (which happens before compiling Ruby), it does that explicitly using -j 1 https://github.com/rbenv/ruby-build/blob/a4ac4be47f9db57eb5d39916f11a51f3ad442091/bin/ruby-build#L1195-L1197

But unless MAKEOPTS is defined, the default will always be to use sysctl -n hw.ncpu cores https://github.com/rbenv/ruby-build/blob/a4ac4be47f9db57eb5d39916f11a51f3ad442091/bin/ruby-build#L565-L569

[varungandhi-src]

Slightly confused by the wording... could you clarify if you mean 1 or 2:

1. By default, OpenSSL is compiled with -j 1 and Ruby itself should be compiled with NCPUs. ← If this is the case, it's unclear to me why compiling OpenSSL with a default -j 1 is desirable. As the numbers show, there is a big difference in build times on a modern machine with the defaults vs explicit MAKEOPTS.
2. By default, both OpenSSL and Ruby should be compiled with NCPUs. ← If this is the case, are you implying that there must be something wrong with my build setup causing a CPU usage difference?

[eregon]

It's 1., read the comment in the source above for why. Compiling OpenSSL used to fail with anything but -j 1. I've read this has been solved in recent OpenSSL but 1) we'd need to find out in which version it's fixed 2) add logic to based on the openssl version use -j 1 or not. Help welcome for those 2 things.

[mislav]

@varungandhi-src You can skip the openssl build step with ruby-build:

brew install [email protected]
RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix [email protected])" rbenv install 2.7.2

Now, if all CPU cores still aren't utilized for the duration of the build, but they get utilized when you add MAKEOPTS="-j 10", then you are right that the CPU core number is not getting plumbed through correctly and we could investigate.

[mislav]

@varungandhi-src Were you able to confirm that ruby-build doesn't utilize all available CPU cores on your machine, except in the case of building openssl? I'm wondering what we should fix here

[mislav]

Closing due to lack of information.