[NC] Restructure RPM Build process

[tangledbytes]

Explain the changes

This PR restructures how we do the RPM builds today.

What changed?

• Instead of fetching NooBaa runtime dependencies in the Dockerfile, we fetch them in the RPM build process.

Why changed?

• This approach allows generating a true architecture independent SRPM, hence a SRPM can be generated once (per RHEL version) and then rpmbuild --rebuild <rpm> can be used to generate RPMS for different architectures.

Note: Usage remains same, that is use, make rpm. This will generate a SRPM and a RPM. Use the SRPM to generate the RPMs for other architectures. The command also supports make rpm SRPM_ONLY=true which is significantly faster and jsut generates SRPMs. Make sure the environment has all the dependencies that NooBaa requires for the build process, refer to builder.Dockerfile for reference

• [ ] Doc added/updated
• [ ] Tests added

[tangledbytes]

@b-ranto ^

[tangledbytes]

@b-ranto added support for SRPM_ONLY=true in the make recipe.

[b-ranto]

I'm sharing the spec file with some cleanups I mentioned above for reference:

%define revision 20240221
%define noobaaver 5.16.0
%define nodever 20.9.0
%define releasedate Wed Feb 21 2024
%define changelogdata Initial release of NooBaa 5.16.0
%define BUILD_S3SELECT 0
%define BUILD_S3SELECT_PARQUET 0

%define noobaatar %{name}-%{version}-%{revision}.tar.gz

Name: noobaa-core
Version:  %{noobaaver}
Release:  %{revision}%{?dist}
Summary:  NooBaa RPM

License:  Apache-2.0
URL:  https://www.noobaa.io/
Source0:  %{noobaatar}

BuildRequires: python3
Recommends: jemalloc

%global __os_install_post %{nil}

%global __requires_exclude ^/usr/bin/python$

%description
NooBaa is a data service for cloud environments, providing S3 object-store interface with flexible tiering, mirroring, and spread placement policies, over any storage resource that allows GET/PUT including S3, GCS, Azure Blob, Filesystems, etc.

%prep
%setup -n noobaa -q

%build
NODEJS_VERSION="%{nodever}"
SKIP_NODE_INSTALL=1 source src/deploy/NVA_build/install_nodejs.sh $NODEJS_VERSION

mkdir -p ../node/

nodepath=$(download_node)
tar -xJf ${nodepath} -C ../node/

PATH=$PATH:%{_builddir}/node/node-v$NODEJS_VERSION-linux-$(get_arch)/bin

npm install --omit=dev && npm cache clean --force
./src/deploy/NVA_build/clone_s3select_submodules.sh
#ln -s /lib64/libboost_thread.so.1.66.0 /lib64/libboost_thread.so.1.75.0 || true
GYP_DEFINES="BUILD_S3SELECT=%{BUILD_S3SELECT} BUILD_S3SELECT_PARQUET=%{BUILD_S3SELECT_PARQUET}" npm run build

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/usr/local/

cp -R %{_builddir}/noobaa $RPM_BUILD_ROOT/usr/local/noobaa-core
cp -R %{_builddir}/node/* $RPM_BUILD_ROOT/usr/local/noobaa-core/node/

mkdir -p $RPM_BUILD_ROOT/usr/local/noobaa-core/bin
ln -s /usr/local/noobaa-core/node/bin/node $RPM_BUILD_ROOT/usr/local/noobaa-core/bin/node
ln -s /usr/local/noobaa-core/node/bin/npm $RPM_BUILD_ROOT/usr/local/noobaa-core/bin/npm
ln -s /usr/local/noobaa-core/node/bin/npx $RPM_BUILD_ROOT/usr/local/noobaa-core/bin/npx

mkdir -p $RPM_BUILD_ROOT/etc/systemd/system/
cp -R %{_builddir}/noobaa/src/deploy/noobaa_nsfs.service $RPM_BUILD_ROOT/etc/systemd/system/noobaa_nsfs.service
ln -s /usr/local/noobaa-core/src/deploy/nsfs_env.env $RPM_BUILD_ROOT/usr/local/noobaa-core/nsfs_env.env
mkdir -p $RPM_BUILD_ROOT/etc/noobaa.conf.d/

mkdir -p $RPM_BUILD_ROOT/etc/rsyslog.d/
ln -s /usr/local/noobaa-core/src/deploy/standalone/noobaa_syslog.conf $RPM_BUILD_ROOT/etc/rsyslog.d/noobaa_syslog.conf
ln -s /usr/local/noobaa-core/src/deploy/standalone/noobaa_rsyslog.conf $RPM_BUILD_ROOT/etc/rsyslog.d/noobaa_rsyslog.conf

mkdir -p $RPM_BUILD_ROOT/etc/logrotate.d/noobaa
ln -s /usr/local/noobaa-core/src/deploy/standalone/logrotate_noobaa.conf $RPM_BUILD_ROOT/etc/logrotate.d/noobaa/logrotate_noobaa.conf

%files
/usr/local/noobaa-core
/etc/systemd/system/noobaa_nsfs.service
/etc/logrotate.d/noobaa/logrotate_noobaa.conf
/etc/rsyslog.d/noobaa_rsyslog.conf
/etc/rsyslog.d/noobaa_syslog.conf
/etc/noobaa.conf.d/
%doc

%post
state=$(systemctl show -p ActiveState --value rsyslog)
if [ "${state}" == "active" ]; then
  service rsyslog restart
fi

%changelog
* %{releasedate} NooBaa Team <[email protected]>
%{changelogdata}

FYI: This spec file was tested locally and I was able to build it with rpmbuild. It still does not build with mock -- it is at least getting there (fails at a later stage).

[b-ranto]

Also, we should not be using /usr/local for RPMs, that is for local stuff only, RPMs should be installed to proper direcotyr structure (/usr) or in this case, it could be installed to /opt/.

[b-ranto]

I am sharing updated spec file that I was able to build with mock:

# Disable debuginfo -- it does not work well with nodejs apps
%global debug_package %{nil}

%define revision 20240221
%define noobaaver 5.16.0
%define nodever 20.9.0
%define releasedate Wed Feb 21 2024
%define changelogdata Initial release of NooBaa 5.16.0
%define BUILD_S3SELECT 0
%define BUILD_S3SELECT_PARQUET 0
%define _build_id_links none

%define noobaatar %{name}-%{version}-%{revision}.tar.gz

Name: noobaa-core
Version:  %{noobaaver}
Release:  %{revision}%{?dist}
Summary:  NooBaa RPM

License:  Apache-2.0
URL:  https://www.noobaa.io/
Source0:  %{noobaatar}

BuildRequires:  python3
BuildRequires:  make
BuildRequires:  gcc-c++
BuildRequires:  nasm

Recommends:     jemalloc

%description
NooBaa is a data service for cloud environments, providing S3 object-store interface with flexible tiering, mirroring, and spread placement policies, over any storage resource that allows GET/PUT including S3, GCS, Azure Blob, Filesystems, etc.

%prep
%setup -n noobaa -q

%build
SKIP_NODE_INSTALL=1 source src/deploy/NVA_build/install_nodejs.sh "${nodever}"

mkdir -p ../node/

nodepath=$(download_node)
tar -xJf ${nodepath} -C ../node/

PATH=$PATH:%{_builddir}/node/node-v${nodever}-linux-$(get_arch)/bin

npm install --omit=dev && npm cache clean --force
./src/deploy/NVA_build/clone_s3select_submodules.sh
#ln -s /lib64/libboost_thread.so.1.66.0 /lib64/libboost_thread.so.1.75.0 || true
GYP_DEFINES="BUILD_S3SELECT=%{BUILD_S3SELECT} BUILD_S3SELECT_PARQUET=%{BUILD_S3SELECT_PARQUET}" npm run build

%install
rm -rf $RPM_BUILD_ROOT

mkdir -p $RPM_BUILD_ROOT/etc/systemd/system/
mv %{_builddir}/noobaa/src/deploy/noobaa_nsfs.service $RPM_BUILD_ROOT/etc/systemd/system/noobaa_nsfs.service
mkdir -p $RPM_BUILD_ROOT/etc/noobaa.conf.d/

mkdir -p $RPM_BUILD_ROOT/etc/rsyslog.d/
mv %{_builddir}/noobaa/src/deploy/standalone/noobaa_syslog.conf $RPM_BUILD_ROOT/etc/rsyslog.d/noobaa_syslog.conf
mv %{_builddir}/noobaa/src/deploy/standalone/noobaa_rsyslog.conf $RPM_BUILD_ROOT/etc/rsyslog.d/noobaa_rsyslog.conf

mkdir -p $RPM_BUILD_ROOT/etc/logrotate.d/noobaa
mv %{_builddir}/noobaa/src/deploy/standalone/logrotate_noobaa.conf $RPM_BUILD_ROOT/etc/logrotate.d/noobaa/logrotate_noobaa.conf

mkdir -p $RPM_BUILD_ROOT/usr/local/
mv %{_builddir}/noobaa $RPM_BUILD_ROOT/usr/local/noobaa-core
mv src/deploy/nsfs_env.env $RPM_BUILD_ROOT/usr/local/noobaa-core/nsfs_env.env
mv %{_builddir}/node/* $RPM_BUILD_ROOT/usr/local/noobaa-core/node/

mkdir -p $RPM_BUILD_ROOT/usr/local/noobaa-core/bin
ln -s ../node/bin/node $RPM_BUILD_ROOT/usr/local/noobaa-core/bin/node
ln -s ../node/bin/npm $RPM_BUILD_ROOT/usr/local/noobaa-core/bin/npm
ln -s ../node/bin/npx $RPM_BUILD_ROOT/usr/local/noobaa-core/bin/npx

%files
/usr/local/noobaa-core
/etc/systemd/system/noobaa_nsfs.service
/etc/logrotate.d/noobaa/logrotate_noobaa.conf
/etc/rsyslog.d/noobaa_rsyslog.conf
/etc/rsyslog.d/noobaa_syslog.conf
/etc/noobaa.conf.d/
%doc

%post
state=$(systemctl show -p ActiveState --value rsyslog)
if [ "${state}" == "active" ]; then
  service rsyslog restart
fi

%changelog
* %{releasedate} NooBaa Team <[email protected]>
%{changelogdata}

There is still a lot of room for improvement but it is getting there.

EDIT: This still fails for alternative arches, that will require another fix probably. :-/

[b-ranto]

I have created a replacement PR with all the updates suggested above where I use this PR as a base:

https://github.com/noobaa/noobaa-core/pull/7845

I was able to finally even build it cross-platform (ppc64le) with these changes. The command I ran:

mock -r /etc/mock/rhel-9-ppc64le.cfg --enable-network noobaa-core-5.16.0-20240221.fc39.src.rpm

[tangledbytes]

Hi @b-ranto, can you please push your changes to the same branch (maybe adding yourself as the co-author)? I don't think spawning another PR is a good idea.

Or I can add your suggested changes right here.

[b-ranto]

@tangledbytes You can cherry-pick the commits here. I can't push to your branch, I don't have permissions to do that.

[tangledbytes]

@b-ranto addressed most of the comments here (by referring to PR #7845) and added you as the coauthor to this PR.

[tangledbytes]

I cannot seem to get nasm installed via their official RPM repo on arm architecture. Reverted that change.

[tangledbytes]

Will merge this PR now. There is one unaddressed concern from @b-ranto here:

• BuildRequires: nasm should be added - Not addressed in this PR because I couldn't get NASM installed from RH repo and could get it installed from NASM official repo on ARM architecture.

Test performed with the following steps:

1. Build a SRPM and RPM on the ARM platform.
2. Install the built RPM on a ARM linux OS.
3. Check if NooBaa starts up.
4. Use the SRPM to build RPM on x86_64 platform using rpm --rebuild <srpm>.
5. Install the freshly generated RPM on x86_64.
6. Check if NooBaa starts up.

Given the above validation, I think the PR is addressing the initial concern of being able to build RPMs via SRPM on other architectures.

cc: @liranmauda @b-ranto