pigsty

pigsty copied to clipboard

Add support for CentOS 9

I can help on this. Since CentOS goes Stream (upstream/development branch of RHEL), I think it is possible to evaluate the use of Rocky Linux as CentOS replacement.

We will try to do some clarity.

I think we can do some scripting to make the repo dynamic depending on OS Build & version

Since gathering os facts is disabled, this is a way to get those parameters:

in ./roles/environ/tasks/main.yml

#--------------------------------------------------------------#
# Gather RHEL build & version
#--------------------------------------------------------------#
- name: Gather RHEL build & version
  connection: local
  tags: environ_os
  block:
    - name: Gather OS Name
      shell: rpm -q --qf "%{VENDOR}" $(rpm -q --whatprovides redhat-release)
      register: osname
    - set_fact:
        os_name: "{{ osname.stdout }}"

    - name: Gather OS Major Version
      shell: rpm -q --qf "%{VERSION}" $(rpm -q --whatprovides redhat-release) | grep -o '^[^.]\+'
      register: osmajorbuild
    - set_fact:
        os_major_build: "{{ osmajorbuild.stdout }}"
    - debug:
        msg: "OS detected: {{ os_name }} {{ os_major_build }}"

This is working well on CentOS 7, Rocky Linux 8/9, AlmaLinux 8/9:

    "msg": "OS detected: CentOS 7"
    "msg": "OS detected: Rocky 8"
    "msg": "OS detected: AlmaLinux 9"

Then, we can switch to template mode on ./roles/repo/tasks/main.yml

    # add all upstream repos
    - name: Add required upstream repos
      template: src=online.repo.j2 dest=/etc/yum.repos.d/online.repo 

This is how online.repo.j2 should be done:

{% if os_name == 'CentOS' %}
[base]
name=CentOS-$releasever - Base
baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=0

[updates]
name=CentOS-$releasever - Updates
baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=0

[extras]
name=CentOS-$releasever - Extras
baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=0
{% elif os_name == 'Rocky' %}
[appstream]
name=Rocky Linux $releasever - AppStream
baseurl=https://dl.rockylinux.org/$contentdir/$releasever/AppStream/$basearch/os/
gpgcheck=0

[baseos]
name=Rocky Linux $releasever - BaseOS
baseurl=https://dl.rockylinux.org/$contentdir/$releasever/BaseOS/$basearch/os/
gpgcheck=0

[extras]
name=Rocky Linux $releasever - Extras
baseurl=https://dl.rockylinux.org/$contentdir/$releasever/extras/$basearch/os/
gpgcheck=0

{% if os_major_build|int == 8 %}
[powertools]
name=Rocky Linux $releasever - PowerTools
baseurl=https://dl.rockylinux.org/$contentdir/$releasever/PowerTools/$basearch/os/
gpgcheck=0
{% endif %}

{% elif os_name == 'AlmaLinux' %}
[appstream]
name=AlmaLinux $releasever - AppStream
baseurl=https://repo.almalinux.org/almalinux/$releasever/AppStream/$basearch/os/
gpgcheck=0

[baseos]
name=AlmaLinux $releasever - BaseOS
baseurl=https://repo.almalinux.org/almalinux/$releasever/BaseOS/$basearch/os/
gpgcheck=0

[extras]
name=AlmaLinux $releasever - Extras
baseurl=https://repo.almalinux.org/almalinux/$releasever/extras/$basearch/os/
gpgcheck=0

{% if os_major_build|int == 8 %}
[powertools]
name=AlmaLinux $releasever - PowerTools
baseurl=https://repo.almalinux.org/almalinux/$releasever/extras/$basearch/os/
gpgcheck=0
{% endif %}
{% endif %}

[epel]
name=Extra Packages for Enterprise Linux $releasever - $basearch
{% if os_major_build|int >= 8 %}
baseurl=http://download.fedoraproject.org/pub/epel/$releasever/Everything/$basearch
{% else %}
baseurl=http://download.fedoraproject.org/pub/epel/$releasever/$basearch
{% endif %}
gpgcheck=0

{% if os_major_build|int == 8 %}
[epel-modular]
name=Extra Packages for Enterprise Linux Modular 8 - $basearch
baseurl=http://download.fedoraproject.org/pub/epel/$releasever/Modular/$basearch
gpgcheck=0
{% endif %}

pgdg repo can be versioned too:

[pgdg-common]
name=PostgreSQL common RPMs for RHEL / {{ os_name }} $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-$releasever-$basearch
gpgcheck=0

[pgdg{{ pg_version }}]
name=PostgreSQL {{ pg_version }} for RHEL / {{ os_name }} $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/{{ pg_version }}/redhat/rhel-$releasever-$basearch
gpgcheck=0

[pgdg-rhel{{ os_major_build }}-extras]
name=Extra packages to support some RPMs in the PostgreSQL RPM repo RHEL / {{ os_name }} $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/common/pgdg-rhel$releasever-extras/redhat/rhel-$releasever-$basearch
gpgcheck=0

It is working ok on CentOS7. We should be able to build a working system on EL8 but not on EL9 atm:

• timescaledb and prometheus are missing el/9 repos
• harbottle is missing el/9 repo too, but pgdg-rhel[7,8,9]-extras contain some useful packages such as consul & consul-template. Also, pgdg-rhel[8,9]-extras got latest HAProxy package, build with prometheus-exporter support, so no need to create on our own (at this point i think we can live without harbottle repo)

It's a great idea to use a template repo file instead of repo_upstreams. Saves lots of config space.

HAProxy rpm used by pigsty are built with https://github.com/Vonng/haproxy-rpm.

Which enables built-in Prometheus metrics support. Maybe I can compile an El8/EL9 version and upgrade to v2.6

Another step forward.

I am testing OL8 also, which is working well.

Another issue is when we use 'yum install --downloadonly' cmd for downloading repo packages on ./roles/repo/tasks/main.yml, il will not download packages that have already been installed on control node.

I think we can rely on repotrack command, which will do both jobs and works well on newest EL builds (on EL8/9 it became an alias of 'dnf download' command).

While specifing '--arch x86_64,noarch' switch, i686 packages can be excluded. This behavior doesn't work on EL7 due to https://bugzilla.redhat.com/show_bug.cgi?id=1150115 (but code can be patched https://getridbug.com/unix-linux/download-all-dependencies-with-yumdownloader-even-if-already-installed/)

    # download packages from upstream yum repo
    - name: Download repo packages and their dependencies
      tags: repo_pkg
      environment: "{{ proxy_env }}"
      command:
        cmd: repotrack --arch x86_64,noarch {{ item }}
        chdir: "{{ nginx_home }}/{{ repo_name }}"
      with_items: "{{ repo_packages }}"

Maybe I can remove i686 packages when making pkg.tgz when using el7 ...

Added dynamic repo template https://github.com/Vonng/pigsty/commit/a2f2055e59684eadb45a77f35dbce00d833bd1ee

Feel free to add mirrors where needed :)

What about using a series of a predefined files instead of if-else templates.

Just like pg_conf?

And we may get a list of combinations such as :

• centos7

• centos8

• centos9

• centos7-china (mirror)

• centos8-china (mirror)

• centos9-china (mirror)

• rocky7

• rocky8

• rocky9

• oracle7

• oracle8

• oracle9

• alma8

• alma9

or we can use both?

• centos.repo
• oracle.repo
• rocky.repo
• alma.repo

and

if os_major_version == 7:...
if os_major_version == 8:...
if os_major_version == 9:...

I think the second solution is more sleek. Maybe we can add a var like os_region or something to be like 'europe' or 'china' and configure repo mirrors based on that?

brilliant idea. so we got one template file for each OS Name. and distinguish different repos by os_major_build & os_region.

What if we make os_name & os_major_build as config parameters. And optionally detect and set them during runtime?

• os_name: CentOS
• os_version: 7
• repo_region: default|china|europe|...

Ok!

To sum up:

• Add os_name, os_version, repo_region variables
• Move 'Gather OS specs' runtime from environ to repo but use them optionally (i.e. when os_name & os_version are not defined or os_name is set to 'auto')
• One template file for each OS distinguished by os_version & repo_region

Also, I think we can use os_version var to make dynamic repo_url_packages, for example: - https://github.com/Vonng/haproxy-rpm/releases/download/v2.6.1/haproxy-2.6.1-1.el${os_version}.x86_64.rpm

Updates:

• ntpd support is removed from EL8 onwards. Shall we consider removing it too in favour of Chrony?
• dynamic repo_url_packages urls are working well;
• some packages such as etcd and pgloader don't exists on official EL/8 repos;
• Polysh isn't working as it requires python2 packages;
• I cleaned up some python packages from repo_packages as they will be downloaded anyway as dependencies; for that sake, I will upload a new pigsty-el.yml conf example file;

There are some slight caveats in Ansible version on various OSes. For example, Oracle 8 EPEL will still install Ansible v2.9, whereas Rocky/Alma/CentOS 8 EPEL will install Ansible v2.12 that needs python 3.8 to work properly (control node only).

I will update ASAP

Everything seems fine and works on EL8/EL9 Rocky/CentOS/Alma/Oracle. It will be released on v1.6.0.

EL8 example config: https://github.com/Vonng/pigsty/blob/master/files/conf/pigsty-el8.yml EL9 excample config: https://github.com/Vonng/pigsty/blob/master/files/conf/pigsty-el9.yml

And, thanks for your great contribution 🎉 @alemacci