postgresql_cluster icon indicating copy to clipboard operation
postgresql_cluster copied to clipboard

Published 20 hours ago • verified publisher icon vitabaks

Provision of resources in a popular cloud providers for PostgreSQL cluster deployment

Open vitabaks opened this issue 2 years ago • 17 comments

Issue: https://github.com/vitabaks/postgresql_cluster/issues/458

  1. Provision of cloud resources (virtual machine + disk) to a popular cloud providers for PostgreSQL cluster deployment:

    • [x] AWS
    • [x] GCP
    • [x] Azure
    • [x] DigitalOcean
    • [x] Hetzner Cloud

  2. Create a Cloud Load Balancer - entry point for connecting to a database in a cluster (if 'cloud_load_balancer' is 'true', default: true)

    • [x] AWS Elastic Load Balancer (ELB) - external and internal
    • [x] GCP Proxy Network Load Balancer - external only (TODO: internal)
    • [ ] Azure
    • [x] DigitalOcean Load Balancer - external and internal
    • [x] Hetzner Cloud Load Balancer - external and internal

  3. Create a Bucket in the cloud and configure backups to this bucket (if 'pgbackrest_install' or 'wal_g_instal' is 'true', default: false)

    • [x] AWS S3 Bucket
    • [x] GCP Bucket
    • [x] Azure Blob Storage
    • [x] DigitalOcean Spaces
    • [ ] Hetzner Cloud (?)

Role: cloud-resources

Examples:

AWS

export AWS_ACCESS_KEY_ID=********
export AWS_SECRET_ACCESS_KEY=********

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --extra-vars \
    "cloud_provider=aws \
     cloud_load_balancer=true \
     server_count=3 \
     server_location=us-east-2 \
     server_type=m5.xlarge \
     server_image=ami-05fb0b8c1424f266b \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Note: "ami-05fb0b8c1424f266b" is Ubuntu 22.04 LTS (from 2023-12-07)

GCP

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  ********
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider='gcp' \
     server_count=3 \
     server_location='us-central1' \
     server_type='n2-standard-4' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Azure

export AZURE_SUBSCRIPTION_ID=********
export AZURE_CLIENT_ID=********
export AZURE_SECRET=********
export AZURE_TENANT=********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider='azure' \
     server_count=3 \
     server_location='eastus' \
     server_type='Standard_D2s_v3' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Note: instead of 'server_image', use variables: azure_vm_image_offer, azure_vm_image_publisher, azure_vm_image_sku, azure_vm_image_version. Default: '0001-com-ubuntu-server-jammy', 'Canonical', '22_04-lts-gen2', 'latest'.

DigitalOcean

export DO_API_TOKEN=********

ansible-playbook deploy_pgcluster.yml \
  --user=root --extra-vars \
    "cloud_provider='digitalocean' \
     cloud_load_balancer=true \
     server_count=3 \
     server_location='nyc1' \
     server_type='g-4vcpu-16gb' \
     server_image='ubuntu-22-04-x64' \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Hetzner Cloud

export HCLOUD_API_TOKEN=********

ansible-playbook deploy_pgcluster.yml \
  --user=root --extra-vars \
    "cloud_provider='hetzner' \
     cloud_load_balancer=true \
     server_count=3 \
     server_location='ash' \
     server_type='CCX23' \
     server_image='ubuntu-22.04' \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Additionally:

  • Docker image 'vitabaks/postgresql_cluster'
    • includes repository code, ansible, and all necessary dependencies.
  • New roles:
    • authorized-keys
      • These SSH public keys will be added to the server's ~/.ssh/authorized_keys file.
      • variable: ssh_public_keys (optional, default '')
    • mount
      • Configure mount points in /etc/fstab and mount the file system
      • variables:
        • mount (optional, default '')
        • pg_data_mount_path (default '/pgdata')
        • pg_data_mount_fstype (default 'ext4')
      • an empty disk will be automatically detected for cloud providers
      • ZFS pool will be created if 'zfs' specified in the mount.fstype variable.
  • Automatically generate passwords for patroni_superuser_password, patroni_replication_password, pgbouncer_auth_password (if not defined)
    • and print superuser password in “connection info” after deployment
    • if the variable 'mask_password' is 'true' (default 'false'), do not print the superuser password in connection info
  • install perf, FlameGraph and postgres dbgsym/debuginfo packages
    • variable install_perf, default 'false'
  • Extension Auto-Setup
    • This feature simplifies the integration and configuration process for PostgreSQL third-party extensions. This automated approach eliminates the need for manual configuration, allowing users to seamlessly integrate PostgreSQL extensions. To activate the "Extension Auto-Setup", specify the enable_<extension_name>=true variable.
    • List of extensions: timescaledb, citus, pg_repack, pg_cron, pgaudit, pgvector, postgis, pgrouting, pg_stat_kcache, pg_wait_sampling, pg_partman
  • Add the ability to execute pre and post-deploy command
    • Variables: pre_deploy_command, post_deploy_command
    • This can be a direct command, a bash script content, or a path to a script on the host.

Consider sponsoring the project via GitHub or Patreon

vitabaks avatar Sep 02 '23 18:09 vitabaks

Test: AWS

export AWS_ACCESS_KEY_ID=**********
export AWS_SECRET_ACCESS_KEY=**********

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --private-key=$HOME/.ssh/Vitaliy.pem --extra-vars \
    "provision=aws \
     servers_count=3 \
     server_type=m5.large \
     server_image=ami-07d445a5585fce0a1 \
     server_location=us-east-2 \
     volume_size=60 \
     ssh_key_name=Vitaliy"

Result:

PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on AWS)] ****************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Gather information about default VPC] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] ******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] **************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] ****************************************************************************************************************************************************
changed: [localhost] => (item=postgres-cluster-pgnode01)
changed: [localhost] => (item=postgres-cluster-pgnode02)
changed: [localhost] => (item=postgres-cluster-pgnode03)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] *****************************************************************************************************************************************
ok: [localhost] => (item=18.223.205.113)
ok: [localhost] => (item=3.135.186.47)
ok: [localhost] => (item=3.144.48.18)

TASK [cloud-resources : Show EC2 instance info] ****************************************************************************************************************************************************************
ok: [localhost] => (item=18.223.205.113) => {
    "msg": [
        "instance ID is i-0215e0f507b416b43",
        "instance Image is ami-07d445a5585fce0a1",
        "instance Name is postgres-cluster-pgnode01",
        "instance Type is m5.large",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 18.223.205.113",
        "Private IP is 172.31.0.73"
    ]
}
ok: [localhost] => (item=3.135.186.47) => {
    "msg": [
        "instance ID is i-0733c8a9411959a41",
        "instance Image is ami-07d445a5585fce0a1",
        "instance Name is postgres-cluster-pgnode02",
        "instance Type is m5.large",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 3.135.186.47",
        "Private IP is 172.31.3.250"
    ]
}
ok: [localhost] => (item=3.144.48.18) => {
    "msg": [
        "instance ID is i-0eee7f56136793cf6",
        "instance Image is ami-07d445a5585fce0a1",
        "instance Name is postgres-cluster-pgnode03",
        "instance Type is m5.large",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 3.144.48.18",
        "Private IP is 172.31.7.185"
    ]
}

TASK [cloud-resources : Add host to "postgres_cluster", "master" groups (in-memory inventory)] *****************************************************************************************************************
ok: [localhost] => (item=18.223.205.113)

TASK [cloud-resources : Add host to "postgres_cluster", "replica" groups (in-memory inventory)] ****************************************************************************************************************
ok: [localhost] => (item=3.135.186.47)
ok: [localhost] => (item=3.144.48.18)

TASK [cloud-resources : Add host to "etcd_cluster" group (in-memory inventory)] ********************************************************************************************************************************
ok: [localhost] => (item=18.223.205.113)
ok: [localhost] => (item=3.135.186.47)
ok: [localhost] => (item=3.144.48.18)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, count: 1, cores: 1",
        "Disk space total": "77.75 GB",
        "Kernel": "5.15.0-1030-aws",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "m5.large",
        "Virtualization type": "kvm"
    }
}
ok: [172.31.3.250] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz, count: 1, cores: 1",
        "Disk space total": "77.75 GB",
        "Kernel": "5.15.0-1030-aws",
        "Memory": "7.47 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "m5.large",
        "Virtualization type": "kvm"
    }
}
ok: [172.31.7.185] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, count: 1, cores: 1",
        "Disk space total": "77.75 GB",
        "Kernel": "5.15.0-1030-aws",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "m5.large",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [172.31.0.73] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [172.31.0.73] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [172.31.0.73] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=master)
ok: [172.31.3.250] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=postgres_cluster)
ok: [172.31.3.250] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=etcd_cluster)
ok: [172.31.3.250] => (item=replica)
ok: [172.31.7.185] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [172.31.3.250] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [172.31.7.185] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [172.31.0.73] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [172.31.7.185] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [172.31.3.250] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [172.31.3.250] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [172.31.0.73] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [172.31.3.250] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [172.31.7.185] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [172.31.0.73] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [172.31.3.250] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [172.31.7.185] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [172.31.0.73] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [172.31.3.250] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [172.31.7.185] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [172.31.0.73] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [172.31.3.250] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [172.31.7.185] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [172.31.0.73] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [172.31.3.250] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [172.31.7.185] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [172.31.0.73] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [172.31.3.250] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [172.31.7.185] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [172.31.0.73] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [172.31.3.250] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [172.31.7.185] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [172.31.0.73] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [172.31.3.250] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [172.31.0.73] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [172.31.7.185] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [172.31.0.73] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [172.31.3.250] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [172.31.7.185] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [172.31.3.250] => (item=etcd)
changed: [172.31.7.185] => (item=etcd)
changed: [172.31.0.73] => (item=etcd)
changed: [172.31.0.73] => (item=etcdctl)
changed: [172.31.3.250] => (item=etcdctl)
changed: [172.31.7.185] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "msg": "http://172.31.0.73:2379 is healthy: successfully committed proposal: took = 3.457788ms"
}
ok: [172.31.3.250] => {
    "msg": "http://172.31.3.250:2379 is healthy: successfully committed proposal: took = 3.650855ms"
}
ok: [172.31.7.185] => {
    "msg": "http://172.31.7.185:2379 is healthy: successfully committed proposal: took = 3.018372ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [172.31.0.73] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [172.31.3.250] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [172.31.7.185] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [172.31.3.250] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [172.31.7.185] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [172.31.0.73] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [172.31.3.250] => (item=['python3'])
ok: [172.31.7.185] => (item=['python3'])
ok: [172.31.0.73] => (item=['python3'])
ok: [172.31.3.250] => (item=python3)
ok: [172.31.7.185] => (item=python3)
ok: [172.31.0.73] => (item=python3)
changed: [172.31.7.185] => (item=python3-dev)
changed: [172.31.3.250] => (item=python3-dev)
changed: [172.31.0.73] => (item=python3-dev)
changed: [172.31.7.185] => (item=python3-psycopg2)
changed: [172.31.3.250] => (item=python3-psycopg2)
changed: [172.31.0.73] => (item=python3-psycopg2)
ok: [172.31.7.185] => (item=python3-setuptools)
ok: [172.31.3.250] => (item=python3-setuptools)
ok: [172.31.0.73] => (item=python3-setuptools)
changed: [172.31.3.250] => (item=python3-pip)
ok: [172.31.3.250] => (item=curl)
changed: [172.31.7.185] => (item=python3-pip)
ok: [172.31.3.250] => (item=less)
ok: [172.31.7.185] => (item=curl)
changed: [172.31.0.73] => (item=python3-pip)
ok: [172.31.3.250] => (item=sudo)
ok: [172.31.7.185] => (item=less)
ok: [172.31.0.73] => (item=curl)
ok: [172.31.3.250] => (item=vim)
ok: [172.31.7.185] => (item=sudo)
ok: [172.31.0.73] => (item=less)
ok: [172.31.3.250] => (item=gcc)
ok: [172.31.7.185] => (item=vim)
ok: [172.31.0.73] => (item=sudo)
ok: [172.31.7.185] => (item=gcc)
ok: [172.31.0.73] => (item=vim)
ok: [172.31.0.73] => (item=gcc)
changed: [172.31.3.250] => (item=jq)
ok: [172.31.3.250] => (item=iptables)
changed: [172.31.7.185] => (item=jq)
ok: [172.31.7.185] => (item=iptables)
changed: [172.31.0.73] => (item=jq)
ok: [172.31.0.73] => (item=iptables)
changed: [172.31.3.250] => (item=acl)
changed: [172.31.7.185] => (item=acl)
changed: [172.31.3.250] => (item=dnsutils)
changed: [172.31.0.73] => (item=acl)
changed: [172.31.7.185] => (item=dnsutils)
changed: [172.31.0.73] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.3.250]
changed: [172.31.0.73]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [172.31.3.250] => (item=postgresql-15)
ok: [172.31.3.250] => (item=postgresql-client-15)
changed: [172.31.0.73] => (item=postgresql-15)
ok: [172.31.3.250] => (item=postgresql-contrib-15)
ok: [172.31.0.73] => (item=postgresql-client-15)
ok: [172.31.0.73] => (item=postgresql-contrib-15)
changed: [172.31.7.185] => (item=postgresql-15)
ok: [172.31.7.185] => (item=postgresql-client-15)
ok: [172.31.7.185] => (item=postgresql-contrib-15)
changed: [172.31.0.73] => (item=postgresql-server-dev-15)
changed: [172.31.3.250] => (item=postgresql-server-dev-15)
changed: [172.31.7.185] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [172.31.3.250] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [172.31.0.73] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [172.31.7.185] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.0.73]
changed: [172.31.3.250]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.0.73]
changed: [172.31.3.250]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [172.31.0.73] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=master)
ok: [172.31.3.250] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=postgres_cluster)
ok: [172.31.3.250] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=etcd_cluster)
ok: [172.31.3.250] => (item=replica)
ok: [172.31.7.185] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [172.31.0.73] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [172.31.3.250] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [172.31.7.185] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [172.31.3.250] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [172.31.3.250] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [172.31.0.73] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [172.31.7.185] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [172.31.3.250] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [172.31.0.73] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [172.31.7.185] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [172.31.3.250] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [172.31.0.73] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [172.31.7.185] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [172.31.3.250] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [172.31.7.185] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [172.31.0.73] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [172.31.3.250] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [172.31.0.73] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [172.31.7.185] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [172.31.3.250] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [172.31.7.185] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [172.31.0.73] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [172.31.3.250] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [172.31.7.185] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [172.31.0.73] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [172.31.3.250] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [172.31.7.185] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [172.31.0.73] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [172.31.3.250] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [172.31.0.73] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [172.31.0.73] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [172.31.3.250] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [172.31.7.185] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [172.31.3.250] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [172.31.0.73] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [172.31.7.185] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [172.31.3.250] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [172.31.0.73] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [172.31.7.185] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [172.31.3.250] => (item=LANG=en_US.utf-8)
changed: [172.31.0.73] => (item=LANG=en_US.utf-8)
changed: [172.31.7.185] => (item=LANG=en_US.utf-8)
changed: [172.31.3.250] => (item=LANGUAGE=en_US.utf-8)
changed: [172.31.0.73] => (item=LANGUAGE=en_US.utf-8)
changed: [172.31.7.185] => (item=LANGUAGE=en_US.utf-8)
changed: [172.31.3.250] => (item=LC_ALL=en_US.utf-8)
changed: [172.31.0.73] => (item=LC_ALL=en_US.utf-8)
changed: [172.31.7.185] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [172.31.7.185]
ok: [172.31.3.250]
ok: [172.31.0.73]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.7.185]
ok: [172.31.3.250]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.0.73]
changed: [172.31.3.250]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [172.31.0.73] => (item=absent)
changed: [172.31.3.250] => (item=absent)
changed: [172.31.7.185] => (item=absent)
changed: [172.31.3.250] => (item=directory)
changed: [172.31.0.73] => (item=directory)
changed: [172.31.7.185] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [172.31.0.73]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [172.31.0.73]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [172.31.0.73]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
ok: [172.31.3.250]
changed: [172.31.0.73]
ok: [172.31.7.185]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [172.31.7.185]
ok: [172.31.3.250]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [172.31.0.73] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [172.31.0.73]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: postgres-cluster -----+---------+-----------+----+-----------+",
        "| Member          | Host         | Role    | State     | TL | Lag in MB |",
        "+-----------------+--------------+---------+-----------+----+-----------+",
        "| ip-172-31-0-73  | 172.31.0.73  | Leader  | running   |  1 |           |",
        "| ip-172-31-3-250 | 172.31.3.250 | Replica | streaming |  1 |         0 |",
        "| ip-172-31-7-185 | 172.31.7.185 | Replica | streaming |  1 |         0 |",
        "+-----------------+--------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "msg": [
        "+------------------------------------------------+",
        "address 172.31.0.73,172.31.3.250,172.31.7.185",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
172.31.0.73                : ok=104  changed=58   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
172.31.3.250               : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
172.31.7.185               : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=13   changed=2    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0

image

passed

vitabaks avatar Sep 02 '23 21:09 vitabaks

Test: GCP

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  *****
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='gcp' \
     servers_count=5 \
     server_type='n2-standard-4' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     server_location='us-central1' \
     volume_size=60 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:

PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on GCP)] ****************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] *******************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : GCP: Gather information about project] *************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] ********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify private firewall rule] *******************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify instance] ********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode04)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode05)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37)
ok: [localhost] => (item=34.134.213.137)
ok: [localhost] => (item=34.66.129.241)
ok: [localhost] => (item=35.192.39.18)
ok: [localhost] => (item=35.223.176.129)

TASK [cloud-resources : Show GCP instance info] ****************************************************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37) => {
    "msg": [
        "instance ID is 1335351888603344788",
        "instance Name is vitaliy-test-pgcluster-pgnode01",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 34.70.223.37",
        "Private IP is 10.128.15.213"
    ]
}
ok: [localhost] => (item=34.134.213.137) => {
    "msg": [
        "instance ID is 8329019901739626345",
        "instance Name is vitaliy-test-pgcluster-pgnode02",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 34.134.213.137",
        "Private IP is 10.128.15.216"
    ]
}
ok: [localhost] => (item=34.66.129.241) => {
    "msg": [
        "instance ID is 1438317652146220924",
        "instance Name is vitaliy-test-pgcluster-pgnode03",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 34.66.129.241",
        "Private IP is 10.128.15.217"
    ]
}
ok: [localhost] => (item=35.192.39.18) => {
    "msg": [
        "instance ID is 7150671383286607695",
        "instance Name is vitaliy-test-pgcluster-pgnode04",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 35.192.39.18",
        "Private IP is 10.128.15.218"
    ]
}
ok: [localhost] => (item=35.223.176.129) => {
    "msg": [
        "instance ID is 6973353061470793537",
        "instance Name is vitaliy-test-pgcluster-pgnode05",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 35.223.176.129",
        "Private IP is 10.128.15.219"
    ]
}

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 34.70.223.37, private_ip: 10.128.15.213)
ok: [localhost] => (item=public_ip: 34.134.213.137, private_ip: 10.128.15.216)
ok: [localhost] => (item=public_ip: 34.66.129.241, private_ip: 10.128.15.217)
ok: [localhost] => (item=public_ip: 35.192.39.18, private_ip: 10.128.15.218)
ok: [localhost] => (item=public_ip: 35.223.176.129, private_ip: 10.128.15.219)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=34.134.213.137)
ok: [localhost] => (item=34.66.129.241)
ok: [localhost] => (item=35.192.39.18)
ok: [localhost] => (item=35.223.176.129)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37)
ok: [localhost] => (item=34.134.213.137)
ok: [localhost] => (item=34.66.129.241)
ok: [localhost] => (item=35.192.39.18)
ok: [localhost] => (item=35.223.176.129)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]
ok: [10.128.15.216]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.216] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.217] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.218] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.219] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.128.15.213] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.128.15.213] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.219]
ok: [10.128.15.218]
ok: [10.128.15.216]
ok: [10.128.15.217]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.216]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.216]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.219]
ok: [10.128.15.217]
ok: [10.128.15.218]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]
ok: [10.128.15.216]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.219]
ok: [10.128.15.213]
ok: [10.128.15.218]
ok: [10.128.15.217]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.128.15.218]
ok: [10.128.15.217]
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.219]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.128.15.213] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=master)
ok: [10.128.15.216] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=postgres_cluster)
ok: [10.128.15.216] => (item=postgres_cluster)
ok: [10.128.15.217] => (item=etcd_cluster)
ok: [10.128.15.216] => (item=replica)
ok: [10.128.15.217] => (item=postgres_cluster)
ok: [10.128.15.218] => (item=etcd_cluster)
ok: [10.128.15.217] => (item=replica)
ok: [10.128.15.218] => (item=postgres_cluster)
ok: [10.128.15.219] => (item=etcd_cluster)
ok: [10.128.15.218] => (item=replica)
ok: [10.128.15.219] => (item=postgres_cluster)
ok: [10.128.15.219] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.219] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.216] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.217] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.218] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.213] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.213] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.219] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.217] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.213] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.218] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.217] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.213] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.219] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.218] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.217] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.216] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.219] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.213] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.218] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.219] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.213] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.218] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.216] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.213] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.219] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.218] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.213] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.219] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.216] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.217] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.218] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.218] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.218] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.216] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.219] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.219] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.216] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.218] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.218] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.216] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.218]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.128.15.219] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.128.15.217] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.128.15.213] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.128.15.216] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.128.15.218] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.219]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.128.15.213] => (item=etcd)
changed: [10.128.15.217] => (item=etcd)
changed: [10.128.15.216] => (item=etcd)
changed: [10.128.15.218] => (item=etcd)
changed: [10.128.15.219] => (item=etcd)
changed: [10.128.15.213] => (item=etcdctl)
changed: [10.128.15.217] => (item=etcdctl)
changed: [10.128.15.216] => (item=etcdctl)
changed: [10.128.15.218] => (item=etcdctl)
changed: [10.128.15.219] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.218]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.216]
changed: [10.128.15.218]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]
changed: [10.128.15.219]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.217]
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.218]
ok: [10.128.15.217]
ok: [10.128.15.216]
ok: [10.128.15.219]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "msg": "http://10.128.15.213:2379 is healthy: successfully committed proposal: took = 4.548685ms"
}
ok: [10.128.15.216] => {
    "msg": "http://10.128.15.216:2379 is healthy: successfully committed proposal: took = 3.359365ms"
}
ok: [10.128.15.217] => {
    "msg": "http://10.128.15.217:2379 is healthy: successfully committed proposal: took = 4.280164ms"
}
ok: [10.128.15.218] => {
    "msg": "http://10.128.15.218:2379 is healthy: successfully committed proposal: took = 3.928711ms"
}
ok: [10.128.15.219] => {
    "msg": "http://10.128.15.219:2379 is healthy: successfully committed proposal: took = 3.204371ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.217]
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.217] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.219] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.218] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.216] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.217] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.218] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.216] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.219] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.128.15.213] => (item=['python3'])
ok: [10.128.15.217] => (item=['python3'])
ok: [10.128.15.216] => (item=['python3'])
ok: [10.128.15.219] => (item=['python3'])
ok: [10.128.15.218] => (item=['python3'])
ok: [10.128.15.213] => (item=python3)
ok: [10.128.15.217] => (item=python3)
ok: [10.128.15.216] => (item=python3)
ok: [10.128.15.219] => (item=python3)
ok: [10.128.15.218] => (item=python3)
changed: [10.128.15.217] => (item=python3-dev)
changed: [10.128.15.219] => (item=python3-dev)
changed: [10.128.15.213] => (item=python3-dev)
changed: [10.128.15.218] => (item=python3-dev)
changed: [10.128.15.216] => (item=python3-dev)
changed: [10.128.15.217] => (item=python3-psycopg2)
changed: [10.128.15.219] => (item=python3-psycopg2)
ok: [10.128.15.217] => (item=python3-setuptools)
changed: [10.128.15.218] => (item=python3-psycopg2)
changed: [10.128.15.216] => (item=python3-psycopg2)
changed: [10.128.15.213] => (item=python3-psycopg2)
ok: [10.128.15.219] => (item=python3-setuptools)
ok: [10.128.15.218] => (item=python3-setuptools)
ok: [10.128.15.213] => (item=python3-setuptools)
ok: [10.128.15.216] => (item=python3-setuptools)
changed: [10.128.15.217] => (item=python3-pip)
changed: [10.128.15.219] => (item=python3-pip)
changed: [10.128.15.218] => (item=python3-pip)
changed: [10.128.15.213] => (item=python3-pip)
ok: [10.128.15.217] => (item=curl)
ok: [10.128.15.219] => (item=curl)
ok: [10.128.15.218] => (item=curl)
changed: [10.128.15.216] => (item=python3-pip)
ok: [10.128.15.213] => (item=curl)
ok: [10.128.15.217] => (item=less)
ok: [10.128.15.219] => (item=less)
ok: [10.128.15.218] => (item=less)
ok: [10.128.15.213] => (item=less)
ok: [10.128.15.216] => (item=curl)
ok: [10.128.15.217] => (item=sudo)
ok: [10.128.15.219] => (item=sudo)
ok: [10.128.15.218] => (item=sudo)
ok: [10.128.15.213] => (item=sudo)
ok: [10.128.15.216] => (item=less)
ok: [10.128.15.217] => (item=vim)
ok: [10.128.15.219] => (item=vim)
ok: [10.128.15.218] => (item=vim)
ok: [10.128.15.213] => (item=vim)
ok: [10.128.15.216] => (item=sudo)
ok: [10.128.15.217] => (item=gcc)
ok: [10.128.15.219] => (item=gcc)
ok: [10.128.15.218] => (item=gcc)
ok: [10.128.15.213] => (item=gcc)
ok: [10.128.15.216] => (item=vim)
ok: [10.128.15.216] => (item=gcc)
changed: [10.128.15.219] => (item=jq)
changed: [10.128.15.217] => (item=jq)
changed: [10.128.15.218] => (item=jq)
changed: [10.128.15.213] => (item=jq)
ok: [10.128.15.219] => (item=iptables)
ok: [10.128.15.217] => (item=iptables)
ok: [10.128.15.218] => (item=iptables)
ok: [10.128.15.213] => (item=iptables)
changed: [10.128.15.216] => (item=jq)
ok: [10.128.15.216] => (item=iptables)
changed: [10.128.15.219] => (item=acl)
changed: [10.128.15.217] => (item=acl)
changed: [10.128.15.213] => (item=acl)
changed: [10.128.15.218] => (item=acl)
changed: [10.128.15.216] => (item=acl)
changed: [10.128.15.219] => (item=dnsutils)
changed: [10.128.15.217] => (item=dnsutils)
changed: [10.128.15.213] => (item=dnsutils)
changed: [10.128.15.218] => (item=dnsutils)
changed: [10.128.15.216] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]
changed: [10.128.15.217]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.128.15.213] => (item=postgresql-15)
changed: [10.128.15.216] => (item=postgresql-15)
changed: [10.128.15.217] => (item=postgresql-15)
changed: [10.128.15.218] => (item=postgresql-15)
ok: [10.128.15.213] => (item=postgresql-client-15)
ok: [10.128.15.216] => (item=postgresql-client-15)
changed: [10.128.15.219] => (item=postgresql-15)
ok: [10.128.15.217] => (item=postgresql-client-15)
ok: [10.128.15.218] => (item=postgresql-client-15)
ok: [10.128.15.213] => (item=postgresql-contrib-15)
ok: [10.128.15.216] => (item=postgresql-contrib-15)
ok: [10.128.15.219] => (item=postgresql-client-15)
ok: [10.128.15.217] => (item=postgresql-contrib-15)
ok: [10.128.15.218] => (item=postgresql-contrib-15)
ok: [10.128.15.219] => (item=postgresql-contrib-15)
changed: [10.128.15.216] => (item=postgresql-server-dev-15)
changed: [10.128.15.213] => (item=postgresql-server-dev-15)
changed: [10.128.15.218] => (item=postgresql-server-dev-15)
changed: [10.128.15.219] => (item=postgresql-server-dev-15)
changed: [10.128.15.217] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.216] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.219] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.218] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.217] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.128.15.213] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=master)
ok: [10.128.15.216] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=postgres_cluster)
ok: [10.128.15.216] => (item=postgres_cluster)
ok: [10.128.15.217] => (item=etcd_cluster)
ok: [10.128.15.216] => (item=replica)
ok: [10.128.15.217] => (item=postgres_cluster)
ok: [10.128.15.218] => (item=etcd_cluster)
ok: [10.128.15.217] => (item=replica)
ok: [10.128.15.218] => (item=postgres_cluster)
ok: [10.128.15.218] => (item=replica)
ok: [10.128.15.219] => (item=etcd_cluster)
ok: [10.128.15.219] => (item=postgres_cluster)
ok: [10.128.15.219] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.128.15.213] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.216] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.217] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.218] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.219] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.213] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.217] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.213] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.216] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.217] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.218] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.219] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.213] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.216] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.217] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.219] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.218] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.213] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.216] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.217] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.219] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.213] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.216] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.217] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.213] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.213] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.217] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.219] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.213] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.218] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.217] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.217] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.218] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.213] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.219] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.218] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.217] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.213] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.217] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.219] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.218] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.218] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.217]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.128.15.213] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.218] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.217] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.216] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.213] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.218] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.217] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.216] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.219] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.219] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.128.15.213] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.218] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.216] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.217] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.219] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.128.15.213] => (item=LANG=en_US.utf-8)
changed: [10.128.15.218] => (item=LANG=en_US.utf-8)
changed: [10.128.15.216] => (item=LANG=en_US.utf-8)
changed: [10.128.15.217] => (item=LANG=en_US.utf-8)
changed: [10.128.15.213] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.219] => (item=LANG=en_US.utf-8)
changed: [10.128.15.218] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.216] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.217] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.213] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.218] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.216] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.217] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.219] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.219] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.213]
changed: [10.128.15.218]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]
changed: [10.128.15.219]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.217]
changed: [10.128.15.213]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.213]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.217]
ok: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.217]
changed: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [10.128.15.216] => (item=absent)
changed: [10.128.15.218] => (item=absent)
changed: [10.128.15.219] => (item=absent)
changed: [10.128.15.216] => (item=directory)
changed: [10.128.15.218] => (item=directory)
changed: [10.128.15.219] => (item=directory)
changed: [10.128.15.213] => (item=absent)
changed: [10.128.15.217] => (item=absent)
changed: [10.128.15.213] => (item=directory)
changed: [10.128.15.217] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.128.15.213]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.128.15.213]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
changed: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.217]
ok: [10.128.15.219]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.217]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.128.15.219]
ok: [10.128.15.218]
ok: [10.128.15.216]
ok: [10.128.15.217]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.217]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.219]
ok: [10.128.15.218]
ok: [10.128.15.213]
ok: [10.128.15.217]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.128.15.213] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.213]
ok: [10.128.15.217]
ok: [10.128.15.219]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.217]
ok: [10.128.15.213]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.128.15.213]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +---------------+---------+-----------+----+-----------+",
        "| Member                          | Host          | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+---------------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.128.15.213 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.128.15.216 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.128.15.217 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode04 | 10.128.15.218 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode05 | 10.128.15.219 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+---------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "msg": [
        "+------------------------------------------------+",
        "address 10.128.15.213,10.128.15.216,10.128.15.217,10.128.15.218,10.128.15.219",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.128.15.213              : ok=104  changed=58   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
10.128.15.216              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.128.15.217              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.128.15.218              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.128.15.219              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=13   changed=2    unreachable=0    failed=0    skipped=41   rescued=0    ignored=0

image

passed

vitabaks avatar Sep 03 '23 18:09 vitabaks

Test: DigitalOcean

export DO_API_TOKEN=********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='digitalocean' \
     servers_count=3 \
     server_type='g-4vcpu-16gb' \
     server_image='ubuntu-22-04-x64' \
     server_location='nyc1' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:

PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on DIGITALOCEAN)] *******************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get Fingerprint for SSH key 'vitaliy'] ***********************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] *****************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitaliy-test-pgcluster'] ***********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify SSH firewall rule] **************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify private firewall rule] **********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] ************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Set variable: droplet_result] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] ******************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] *****************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240)
ok: [localhost] => (item=167.99.228.9)
ok: [localhost] => (item=159.89.84.26)

TASK [cloud-resources : Show Droplet info] *********************************************************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240) => {
    "msg": [
        "ID: 374097982",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-4vcpu-16gb",
        "Volume Size: 100 GB",
        "Public IP: 198.199.90.240",
        "Private IP: 10.116.0.2"
    ]
}
ok: [localhost] => (item=167.99.228.9) => {
    "msg": [
        "ID: 374098039",
        "Name: vitaliy-test-pgcluster-pgnode02",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-4vcpu-16gb",
        "Volume Size: 100 GB",
        "Public IP: 167.99.228.9",
        "Private IP: 10.116.0.3"
    ]
}
ok: [localhost] => (item=159.89.84.26) => {
    "msg": [
        "ID: 374098074",
        "Name: vitaliy-test-pgcluster-pgnode03",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-4vcpu-16gb",
        "Volume Size: 100 GB",
        "Public IP: 159.89.84.26",
        "Private IP: 10.116.0.4"
    ]
}

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 198.199.90.240, private_ip: 10.116.0.2)
ok: [localhost] => (item=public_ip: 167.99.228.9, private_ip: 10.116.0.3)
ok: [localhost] => (item=public_ip: 159.89.84.26, private_ip: 10.116.0.4)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=167.99.228.9)
ok: [localhost] => (item=159.89.84.26)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240)
ok: [localhost] => (item=167.99.228.9)
ok: [localhost] => (item=159.89.84.26)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz, count: 1, cores: 4",
        "Disk space total": "48.6 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "15.63 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}
ok: [10.116.0.3] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz, count: 1, cores: 4",
        "Disk space total": "48.6 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "15.63 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}
ok: [10.116.0.4] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz, count: 1, cores: 4",
        "Disk space total": "48.6 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "15.63 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.116.0.2] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.116.0.2] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.116.0.2] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=master)
ok: [10.116.0.3] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=postgres_cluster)
ok: [10.116.0.3] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=etcd_cluster)
ok: [10.116.0.3] => (item=replica)
ok: [10.116.0.4] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.116.0.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.116.0.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.116.0.4] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.116.0.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.116.0.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.116.0.2] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.116.0.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.116.0.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.116.0.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.116.0.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.116.0.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.116.0.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.116.0.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.116.0.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.116.0.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.116.0.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.116.0.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.116.0.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.116.0.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.116.0.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.116.0.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.116.0.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.116.0.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.116.0.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.116.0.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.116.0.4] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.116.0.3] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.116.0.2] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.116.0.4] => (item=etcd)
changed: [10.116.0.3] => (item=etcd)
changed: [10.116.0.2] => (item=etcd)
changed: [10.116.0.4] => (item=etcdctl)
changed: [10.116.0.2] => (item=etcdctl)
changed: [10.116.0.3] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "msg": "http://10.116.0.2:2379 is healthy: successfully committed proposal: took = 5.598689ms"
}
ok: [10.116.0.3] => {
    "msg": "http://10.116.0.3:2379 is healthy: successfully committed proposal: took = 4.467634ms"
}
ok: [10.116.0.4] => {
    "msg": "http://10.116.0.4:2379 is healthy: successfully committed proposal: took = 6.142228ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.116.0.3] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.116.0.4] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.116.0.2] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.116.0.4] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.116.0.3] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.116.0.2] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.116.0.2] => (item=['python3'])
ok: [10.116.0.4] => (item=['python3'])
ok: [10.116.0.3] => (item=['python3'])
ok: [10.116.0.4] => (item=python3)
ok: [10.116.0.3] => (item=python3)
ok: [10.116.0.2] => (item=python3)
changed: [10.116.0.4] => (item=python3-dev)
changed: [10.116.0.3] => (item=python3-dev)
changed: [10.116.0.4] => (item=python3-psycopg2)
changed: [10.116.0.2] => (item=python3-dev)
ok: [10.116.0.4] => (item=python3-setuptools)
changed: [10.116.0.3] => (item=python3-psycopg2)
ok: [10.116.0.3] => (item=python3-setuptools)
changed: [10.116.0.2] => (item=python3-psycopg2)
changed: [10.116.0.4] => (item=python3-pip)
ok: [10.116.0.2] => (item=python3-setuptools)
ok: [10.116.0.4] => (item=curl)
ok: [10.116.0.4] => (item=less)
changed: [10.116.0.3] => (item=python3-pip)
ok: [10.116.0.4] => (item=sudo)
ok: [10.116.0.3] => (item=curl)
ok: [10.116.0.4] => (item=vim)
ok: [10.116.0.3] => (item=less)
ok: [10.116.0.4] => (item=gcc)
ok: [10.116.0.3] => (item=sudo)
ok: [10.116.0.3] => (item=vim)
ok: [10.116.0.3] => (item=gcc)
changed: [10.116.0.4] => (item=jq)
ok: [10.116.0.4] => (item=iptables)
changed: [10.116.0.2] => (item=python3-pip)
ok: [10.116.0.2] => (item=curl)
changed: [10.116.0.3] => (item=jq)
ok: [10.116.0.2] => (item=less)
ok: [10.116.0.3] => (item=iptables)
changed: [10.116.0.4] => (item=acl)
ok: [10.116.0.2] => (item=sudo)
ok: [10.116.0.2] => (item=vim)
changed: [10.116.0.4] => (item=dnsutils)
changed: [10.116.0.3] => (item=acl)
ok: [10.116.0.2] => (item=gcc)
changed: [10.116.0.3] => (item=dnsutils)
changed: [10.116.0.2] => (item=jq)
ok: [10.116.0.2] => (item=iptables)
changed: [10.116.0.2] => (item=acl)
changed: [10.116.0.2] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.116.0.4] => (item=postgresql-15)
changed: [10.116.0.3] => (item=postgresql-15)
ok: [10.116.0.4] => (item=postgresql-client-15)
changed: [10.116.0.2] => (item=postgresql-15)
ok: [10.116.0.3] => (item=postgresql-client-15)
ok: [10.116.0.4] => (item=postgresql-contrib-15)
ok: [10.116.0.2] => (item=postgresql-client-15)
ok: [10.116.0.3] => (item=postgresql-contrib-15)
ok: [10.116.0.2] => (item=postgresql-contrib-15)
changed: [10.116.0.4] => (item=postgresql-server-dev-15)
changed: [10.116.0.3] => (item=postgresql-server-dev-15)
changed: [10.116.0.2] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.116.0.4] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.116.0.2] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.116.0.3] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]
changed: [10.116.0.2]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.116.0.2] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=master)
ok: [10.116.0.3] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=postgres_cluster)
ok: [10.116.0.3] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=etcd_cluster)
ok: [10.116.0.3] => (item=replica)
ok: [10.116.0.4] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.116.0.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.116.0.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.116.0.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.116.0.4] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.116.0.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.116.0.3] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.116.0.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.116.0.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.116.0.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.116.0.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.116.0.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.116.0.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.116.0.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.116.0.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.116.0.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.116.0.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.116.0.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.116.0.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.116.0.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.116.0.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.116.0.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.116.0.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.116.0.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.116.0.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.2]
changed: [10.116.0.4]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.116.0.2] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.116.0.4] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.116.0.3] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.116.0.2] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.116.0.4] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.116.0.3] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.116.0.4] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.116.0.3] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.116.0.2] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.116.0.4] => (item=LANG=en_US.utf-8)
changed: [10.116.0.2] => (item=LANG=en_US.utf-8)
changed: [10.116.0.3] => (item=LANG=en_US.utf-8)
changed: [10.116.0.4] => (item=LANGUAGE=en_US.utf-8)
changed: [10.116.0.2] => (item=LANGUAGE=en_US.utf-8)
changed: [10.116.0.3] => (item=LANGUAGE=en_US.utf-8)
changed: [10.116.0.4] => (item=LC_ALL=en_US.utf-8)
changed: [10.116.0.2] => (item=LC_ALL=en_US.utf-8)
changed: [10.116.0.3] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.2]
changed: [10.116.0.4]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [10.116.0.4] => (item=absent)
changed: [10.116.0.3] => (item=absent)
changed: [10.116.0.4] => (item=directory)
changed: [10.116.0.3] => (item=directory)
changed: [10.116.0.2] => (item=absent)
changed: [10.116.0.2] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.116.0.2]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.116.0.2]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
changed: [10.116.0.2]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]
changed: [10.116.0.2]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]
changed: [10.116.0.2]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.116.0.2] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.116.0.2]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +------------+---------+-----------+----+-----------+",
        "| Member                          | Host       | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+------------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "msg": [
        "+------------------------------------------------+",
        "address 10.116.0.2,10.116.0.3,10.116.0.4",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.116.0.2                 : ok=104  changed=56   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
10.116.0.3                 : ok=89   changed=53   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.116.0.4                 : ok=89   changed=53   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=20   changed=5    unreachable=0    failed=0    skipped=70   rescued=0    ignored=0

image

passed

vitabaks avatar Sep 10 '23 21:09 vitabaks

Test: Hetzner

export HCLOUD_API_TOKEN=******

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='hetzner' \
     servers_count=3 \
     server_type='CCX13' \
     server_image='ubuntu-22.04' \
     server_location='ash' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:


PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on HETZNER)] ************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'hcloud' dependency is present on controlling host] ************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Hetzner Cloud: Gather information about SSH key 'vitaliy'] *****************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: ssh_key_names] ***********************************************************************************************************************************************************
ok: [localhost] => (item=None)
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Gather information about network zones] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Extract network zone for server_location] *******************************************************************************************************************************
ok: [localhost] => (item=network_zone: us-east)

TASK [cloud-resources : Hetzner Cloud: Gather information about networks] **************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a network 'postgres-cluster-network'] ****************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a subnetwork in the network 'postgres-cluster-network'] **********************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: server_network] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify SSH firewall] ******************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify private firewall] **************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Gather information about firewalls] *************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Extract firewall names for 'vitaliy-test-pgcluster'] ********************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify server] ************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Add server to network 'postgres-cluster-network'] ***********************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Create or modify volume] ************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76)
ok: [localhost] => (item=5.161.63.15)
ok: [localhost] => (item=5.161.229.14)

TASK [cloud-resources : Show Server info] **********************************************************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76) => {
    "msg": [
        "ID: 37095542",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.228.76",
        "Private IP: 10.0.1.1"
    ]
}
ok: [localhost] => (item=5.161.63.15) => {
    "msg": [
        "ID: 37095550",
        "Name: vitaliy-test-pgcluster-pgnode02",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.63.15",
        "Private IP: 10.0.1.2"
    ]
}
ok: [localhost] => (item=5.161.229.14) => {
    "msg": [
        "ID: 37095551",
        "Name: vitaliy-test-pgcluster-pgnode03",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.229.14",
        "Private IP: 10.0.1.3"
    ]
}

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 5.161.228.76, private_ip: 10.0.1.1)
ok: [localhost] => (item=public_ip: 5.161.63.15, private_ip: 10.0.1.2)
ok: [localhost] => (item=public_ip: 5.161.229.14, private_ip: 10.0.1.3)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=5.161.63.15)
ok: [localhost] => (item=5.161.229.14)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76)
ok: [localhost] => (item=5.161.63.15)
ok: [localhost] => (item=5.161.229.14)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC Processor, count: 1, cores: 1",
        "Disk space total": "75.04 GB",
        "Kernel": "5.15.0-79-generic",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "vServer",
        "Virtualization type": "kvm"
    }
}
ok: [10.0.1.2] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC Processor, count: 1, cores: 1",
        "Disk space total": "75.04 GB",
        "Kernel": "5.15.0-79-generic",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "vServer",
        "Virtualization type": "kvm"
    }
}
ok: [10.0.1.3] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC Processor, count: 1, cores: 1",
        "Disk space total": "75.04 GB",
        "Kernel": "5.15.0-79-generic",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "vServer",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.0.1.1] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.0.1.1] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [10.0.1.3]
changed: [10.0.1.1]
changed: [10.0.1.2]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.0.1.3]
ok: [10.0.1.2]
ok: [10.0.1.1]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.1] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=master)
ok: [10.0.1.2] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=postgres_cluster)
ok: [10.0.1.2] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=etcd_cluster)
ok: [10.0.1.2] => (item=replica)
ok: [10.0.1.3] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.0.1.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.1] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.1] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.1] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.1] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.1] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.1] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.1] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.1] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.1] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.1] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.1] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.0.1.1] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.0.1.2] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.0.1.3] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.0.1.1] => (item=etcd)
changed: [10.0.1.3] => (item=etcd)
changed: [10.0.1.2] => (item=etcd)
changed: [10.0.1.1] => (item=etcdctl)
changed: [10.0.1.3] => (item=etcdctl)
changed: [10.0.1.2] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "msg": "http://10.0.1.1:2379 is healthy: successfully committed proposal: took = 2.963209ms"
}
ok: [10.0.1.2] => {
    "msg": "http://10.0.1.2:2379 is healthy: successfully committed proposal: took = 3.411725ms"
}
ok: [10.0.1.3] => {
    "msg": "http://10.0.1.3:2379 is healthy: successfully committed proposal: took = 2.991831ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.0.1.3] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.1] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.2] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.0.1.2] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.3] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.1] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.0.1.1] => (item=['python3'])
ok: [10.0.1.3] => (item=['python3'])
ok: [10.0.1.2] => (item=['python3'])
ok: [10.0.1.1] => (item=python3)
ok: [10.0.1.3] => (item=python3)
ok: [10.0.1.2] => (item=python3)
changed: [10.0.1.3] => (item=python3-dev)
changed: [10.0.1.2] => (item=python3-dev)
changed: [10.0.1.1] => (item=python3-dev)
changed: [10.0.1.3] => (item=python3-psycopg2)
changed: [10.0.1.2] => (item=python3-psycopg2)
changed: [10.0.1.1] => (item=python3-psycopg2)
ok: [10.0.1.3] => (item=python3-setuptools)
ok: [10.0.1.1] => (item=python3-setuptools)
ok: [10.0.1.2] => (item=python3-setuptools)
changed: [10.0.1.3] => (item=python3-pip)
ok: [10.0.1.3] => (item=curl)
changed: [10.0.1.1] => (item=python3-pip)
ok: [10.0.1.3] => (item=less)
ok: [10.0.1.1] => (item=curl)
changed: [10.0.1.2] => (item=python3-pip)
ok: [10.0.1.3] => (item=sudo)
ok: [10.0.1.1] => (item=less)
ok: [10.0.1.2] => (item=curl)
ok: [10.0.1.3] => (item=vim)
ok: [10.0.1.1] => (item=sudo)
ok: [10.0.1.2] => (item=less)
ok: [10.0.1.3] => (item=gcc)
ok: [10.0.1.1] => (item=vim)
ok: [10.0.1.2] => (item=sudo)
ok: [10.0.1.1] => (item=gcc)
ok: [10.0.1.2] => (item=vim)
changed: [10.0.1.3] => (item=jq)
ok: [10.0.1.2] => (item=gcc)
ok: [10.0.1.3] => (item=iptables)
changed: [10.0.1.1] => (item=jq)
ok: [10.0.1.3] => (item=acl)
changed: [10.0.1.2] => (item=jq)
ok: [10.0.1.1] => (item=iptables)
ok: [10.0.1.2] => (item=iptables)
ok: [10.0.1.1] => (item=acl)
changed: [10.0.1.3] => (item=dnsutils)
ok: [10.0.1.2] => (item=acl)
changed: [10.0.1.1] => (item=dnsutils)
changed: [10.0.1.2] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.0.1.1] => (item=postgresql-15)
changed: [10.0.1.2] => (item=postgresql-15)
changed: [10.0.1.3] => (item=postgresql-15)
ok: [10.0.1.1] => (item=postgresql-client-15)
ok: [10.0.1.2] => (item=postgresql-client-15)
ok: [10.0.1.3] => (item=postgresql-client-15)
ok: [10.0.1.1] => (item=postgresql-contrib-15)
ok: [10.0.1.2] => (item=postgresql-contrib-15)
ok: [10.0.1.3] => (item=postgresql-contrib-15)
changed: [10.0.1.2] => (item=postgresql-server-dev-15)
changed: [10.0.1.3] => (item=postgresql-server-dev-15)
changed: [10.0.1.1] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.0.1.1] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.3] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.2] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.1] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=master)
ok: [10.0.1.2] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=postgres_cluster)
ok: [10.0.1.2] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=etcd_cluster)
ok: [10.0.1.2] => (item=replica)
ok: [10.0.1.3] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.0.1.1] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.1] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.1] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.1] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.1] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.1] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.1] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.1] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.1] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.1] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.1] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.1]
changed: [10.0.1.3]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.0.1.1] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.3] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.2] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.1] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.3] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.2] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.0.1.1] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.3] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.2] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.0.1.1] => (item=LANG=en_US.utf-8)
changed: [10.0.1.2] => (item=LANG=en_US.utf-8)
changed: [10.0.1.3] => (item=LANG=en_US.utf-8)
changed: [10.0.1.1] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.2] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.3] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.1] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.3] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.2] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.2]
ok: [10.0.1.1]
ok: [10.0.1.3]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.1]
changed: [10.0.1.3]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [10.0.1.1] => (item=absent)
changed: [10.0.1.2] => (item=absent)
changed: [10.0.1.3] => (item=absent)
changed: [10.0.1.1] => (item=directory)
changed: [10.0.1.2] => (item=directory)
changed: [10.0.1.3] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.0.1.1]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.0.1.1]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.0.1.1]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
changed: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [10.0.1.3]
ok: [10.0.1.1]
ok: [10.0.1.2]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.0.1.1] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.0.1.1]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +----------+---------+-----------+----+-----------+",
        "| Member                          | Host     | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+----------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "msg": [
        "+------------------------------------------------+",
        "address 10.0.1.1,10.0.1.2,10.0.1.3",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.0.1.1                   : ok=104  changed=58   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
10.0.1.2                   : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.0.1.3                   : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=24   changed=6    unreachable=0    failed=0    skipped=107  rescued=0    ignored=0

image

passed

vitabaks avatar Sep 14 '23 22:09 vitabaks

Test: Azure

export AZURE_SUBSCRIPTION_ID=********
export AZURE_CLIENT_ID=********
export AZURE_SECRET=********
export AZURE_TENANT=********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='azure' \
     servers_count=3 \
     server_type='Standard_DS1_v2' \
     server_location='eastus' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-public-ip)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-public-ip)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-network-interface)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-network-interface)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-network-interface)

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Show Azure VM info] ********************************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode01",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202309190', 'exactVersion': '22.04.202309190'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 52.188.152.174",
        "Private IP: 10.0.1.4"
    ]
}
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode02) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode02",
        "Name: vitaliy-test-pgcluster-pgnode02",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202309190', 'exactVersion': '22.04.202309190'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 40.114.91.246",
        "Private IP: 10.0.1.5"
    ]
}
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode03) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode03",
        "Name: vitaliy-test-pgcluster-pgnode03",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202309190', 'exactVersion': '22.04.202309190'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 13.92.27.118",
        "Private IP: 10.0.1.6"
    ]
}

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 52.188.152.174, private_ip: 10.0.1.4)
ok: [localhost] => (item=public_ip: 40.114.91.246, private_ip: 10.0.1.5)
ok: [localhost] => (item=public_ip: 13.92.27.118, private_ip: 10.0.1.6)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=52.188.152.174)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=40.114.91.246)
ok: [localhost] => (item=13.92.27.118)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=52.188.152.174)
ok: [localhost] => (item=40.114.91.246)
ok: [localhost] => (item=13.92.27.118)

PLAY [deploy_pgcluster.yml | Perform pre-checks] ***************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.6]
ok: [10.0.1.4]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz, count: 1, cores: 1",
        "Disk space total": "84.46 GB",
        "Kernel": "6.2.0-1012-azure",
        "Memory": "3.32 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Virtual Machine",
        "Virtualization type": "VirtualPC"
    }
}
ok: [10.0.1.5] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz, count: 1, cores: 1",
        "Disk space total": "84.46 GB",
        "Kernel": "6.2.0-1012-azure",
        "Memory": "3.32 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Virtual Machine",
        "Virtualization type": "VirtualPC"
    }
}
ok: [10.0.1.6] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, count: 1, cores: 1",
        "Disk space total": "84.46 GB",
        "Kernel": "6.2.0-1012-azure",
        "Memory": "3.32 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Virtual Machine",
        "Virtualization type": "VirtualPC"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.0.1.4] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.0.1.4] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/pgdata/16/main" is not initialized] ****************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pre-checks : Generate a password for patroni superuser] **************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pre-checks : Generate a password for patroni replication user] *******************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pre-checks : Generate a password for pgbouncer auth user] ************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.4] => (item=etcd_cluster)
ok: [10.0.1.4] => (item=master)
ok: [10.0.1.4] => (item=postgres_cluster)
ok: [10.0.1.5] => (item=etcd_cluster)
ok: [10.0.1.5] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=etcd_cluster)
ok: [10.0.1.5] => (item=replica)
ok: [10.0.1.6] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.5] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.6] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.4] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.5] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.6] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.5] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.6] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.5] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.6] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.5] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.5] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.6] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.6] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.5] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.6] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.5] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.6] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.5] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.6] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.5] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.6] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.5] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.0.1.4] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.0.1.6] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.0.1.5] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.0.1.6] => (item=etcd)
changed: [10.0.1.4] => (item=etcd)
changed: [10.0.1.5] => (item=etcd)
changed: [10.0.1.4] => (item=etcdctl)
changed: [10.0.1.6] => (item=etcdctl)
changed: [10.0.1.5] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": "http://10.0.1.4:2379 is healthy: successfully committed proposal: took = 3.555672ms\n"
}
ok: [10.0.1.5] => {
    "msg": "http://10.0.1.5:2379 is healthy: successfully committed proposal: took = 6.794688ms\n"
}
ok: [10.0.1.6] => {
    "msg": "http://10.0.1.6:2379 is healthy: successfully committed proposal: took = 5.695698ms\n"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.6] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.5] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.6] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.5] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.0.1.4] => (item=['python3'])
ok: [10.0.1.5] => (item=['python3'])
ok: [10.0.1.6] => (item=['python3'])
ok: [10.0.1.4] => (item=python3)
ok: [10.0.1.6] => (item=python3)
ok: [10.0.1.5] => (item=python3)
changed: [10.0.1.4] => (item=python3-dev)
changed: [10.0.1.6] => (item=python3-dev)
changed: [10.0.1.4] => (item=python3-psycopg2)
changed: [10.0.1.5] => (item=python3-dev)
ok: [10.0.1.4] => (item=python3-setuptools)
changed: [10.0.1.6] => (item=python3-psycopg2)
ok: [10.0.1.6] => (item=python3-setuptools)
changed: [10.0.1.5] => (item=python3-psycopg2)
ok: [10.0.1.5] => (item=python3-setuptools)
changed: [10.0.1.4] => (item=python3-pip)
ok: [10.0.1.4] => (item=curl)
ok: [10.0.1.4] => (item=less)
ok: [10.0.1.4] => (item=sudo)
ok: [10.0.1.4] => (item=vim)
ok: [10.0.1.4] => (item=gcc)
changed: [10.0.1.6] => (item=python3-pip)
ok: [10.0.1.6] => (item=curl)
changed: [10.0.1.4] => (item=jq)
changed: [10.0.1.5] => (item=python3-pip)
ok: [10.0.1.4] => (item=iptables)
ok: [10.0.1.6] => (item=less)
ok: [10.0.1.5] => (item=curl)
ok: [10.0.1.6] => (item=sudo)
ok: [10.0.1.5] => (item=less)
ok: [10.0.1.6] => (item=vim)
changed: [10.0.1.4] => (item=acl)
ok: [10.0.1.6] => (item=gcc)
ok: [10.0.1.5] => (item=sudo)
ok: [10.0.1.5] => (item=vim)
changed: [10.0.1.4] => (item=dnsutils)
ok: [10.0.1.5] => (item=gcc)
changed: [10.0.1.6] => (item=jq)
ok: [10.0.1.6] => (item=iptables)
changed: [10.0.1.5] => (item=jq)
ok: [10.0.1.5] => (item=iptables)
changed: [10.0.1.6] => (item=acl)
changed: [10.0.1.6] => (item=dnsutils)
changed: [10.0.1.5] => (item=acl)
changed: [10.0.1.5] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.0.1.4] => (item=postgresql-16)
ok: [10.0.1.4] => (item=postgresql-client-16)
ok: [10.0.1.4] => (item=postgresql-contrib-16)
changed: [10.0.1.6] => (item=postgresql-16)
ok: [10.0.1.6] => (item=postgresql-client-16)
changed: [10.0.1.5] => (item=postgresql-16)
ok: [10.0.1.6] => (item=postgresql-contrib-16)
ok: [10.0.1.5] => (item=postgresql-client-16)
ok: [10.0.1.5] => (item=postgresql-contrib-16)
changed: [10.0.1.4] => (item=postgresql-server-dev-16)
changed: [10.0.1.6] => (item=postgresql-server-dev-16)
changed: [10.0.1.5] => (item=postgresql-server-dev-16)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.5] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.6] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.4] => (item=etcd_cluster)
ok: [10.0.1.4] => (item=master)
ok: [10.0.1.5] => (item=etcd_cluster)
ok: [10.0.1.4] => (item=postgres_cluster)
ok: [10.0.1.5] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=etcd_cluster)
ok: [10.0.1.5] => (item=replica)
ok: [10.0.1.6] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.0.1.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.5] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.6] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.4] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.6] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.5] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.6] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.5] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.6] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.6] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.5] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.6] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.5] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.6] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.5] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.6] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.5] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.6] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.5] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.6] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.5] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.6] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.0.1.4] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.5] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.6] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.4] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.5] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.6] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.0.1.4] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.6] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.5] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.0.1.4] => (item=LANG=en_US.utf-8)
changed: [10.0.1.6] => (item=LANG=en_US.utf-8)
changed: [10.0.1.5] => (item=LANG=en_US.utf-8)
changed: [10.0.1.4] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.6] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.5] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.4] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.6] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.5] => (item=LC_ALL=en_US.utf-8)

TASK [mount : Detect empty volume] *****************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [mount : Create "ext4" filesystem on the disk "/dev/sdc"] *************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.5]
changed: [10.0.1.4]

TASK [mount : Get UUID of the disk "/dev/sdc"] *****************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [mount : Set variable: mount.src] *************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [mount : Mount the filesystem] ****************************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'src': '56a254c9-efd8-4f2d-ae12-dc55036a747f', 'path': '/pgdata'})
changed: [10.0.1.6] => (item={'src': '2c71f0fa-29d3-4be4-ab4e-342787754147', 'path': '/pgdata'})
changed: [10.0.1.5] => (item={'src': '96cdeacf-9935-4ec8-8674-df82e0477a90', 'path': '/pgdata'})

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.5]
changed: [10.0.1.4]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.4]
changed: [10.0.1.5]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.0.1.6]
ok: [10.0.1.5]
ok: [10.0.1.4]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Install patroni 3.1.0] *************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/pgdata/16/main" exists] *************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/pgdata/16/main" is empty] ******************************************************************************************************************
changed: [10.0.1.4] => (item=absent)
changed: [10.0.1.6] => (item=absent)
changed: [10.0.1.5] => (item=absent)
changed: [10.0.1.4] => (item=directory)
changed: [10.0.1.6] => (item=directory)
changed: [10.0.1.5] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.0.1.4]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.0.1.4]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
changed: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Disable "postgresql@16-main" service] **********************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.0.1.4] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.0.1.4]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7492 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +----------+---------+-----------+----+-----------+",
        "| Member                          | Host     | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+----------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.0.1.4 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.0.1.5 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.0.1.6 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": [
        "+------------------------------------------------+",
        "address: 10.0.1.4,10.0.1.5,10.0.1.6",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: Yehap75g3TIZbjc7vjVgJBvaLiWckRFz",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.0.1.4                   : ok=112  changed=60   unreachable=0    failed=0    skipped=355  rescued=0    ignored=0   
10.0.1.5                   : ok=94   changed=57   unreachable=0    failed=0    skipped=333  rescued=0    ignored=0   
10.0.1.6                   : ok=94   changed=57   unreachable=0    failed=0    skipped=333  rescued=0    ignored=0   
localhost                  : ok=18   changed=4    unreachable=0    failed=0    skipped=153  rescued=0    ignored=0

image

passed

vitabaks avatar Sep 27 '23 14:09 vitabaks

Test: Deploying to AWS with backups (PgBackRest)

export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=************

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision=aws \
     servers_count=1 \
     server_type=t3.large \
     server_image=ami-07d445a5585fce0a1 \
     server_location=us-east-2 \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true \
     PGBACKREST_S3_KEY=${AWS_ACCESS_KEY_ID} \
     PGBACKREST_S3_KEY_SECRET=${AWS_SECRET_ACCESS_KEY}"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Remove temporary SSH key 'ssh_key_tmp_rqsgybf' from cloud (if any)] ****************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Add SSH key 'ssh_key_tmp_rqsgybf' to cloud] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Gather information about default VPC] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] ***************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create S3 bucket 'backups-vitaliy-test-pgcluster'] *********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] ******************************************************************************************************************************************
ok: [localhost] => (item=18.226.49.185)

TASK [cloud-resources : Show EC2 instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=18.226.49.185) => {
    "msg": [
        "ID: i-0b233bcdfe89d78fc",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: ami-07d445a5585fce0a1",
        "Type: t3.large",
        "Volume Size: 100 GB",
        "Public IP: 18.226.49.185",
        "Private IP: 172.31.5.233"
    ]
}
...

TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in the AWS S3 bucket] ******************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Make sure pgdg apt key is installed] *********************************************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Make sure pgdg repository is installed] ******************************************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Update apt cache] ****************************************************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Install pgbackrest] **************************************************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Ensure spool directory exist] ****************************************************************************************************************************************************************
ok: [172.31.5.233] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] ***************************************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] ******************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Make sure that the cron package is installed] ************************************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] *************************************************************************************************************************************************
changed: [172.31.5.233] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [172.31.5.233] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ****************************************************************************************************************************************
changed: [172.31.5.233]
...

ubuntu@ip-172-31-5-233:~$ cat /etc/pgbackrest/pgbackrest.conf
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=s3
repo1-path=/pgbackrest
repo1-s3-key=******
repo1-s3-key-secret=************
repo1-s3-bucket=backups-vitaliy-test-pgcluster
repo1-s3-endpoint=s3.us-east-2.amazonaws.com
repo1-s3-region=us-east-2
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

ubuntu@ip-172-31-5-233:~$ sudo psql -U postgres
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select * from pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786DB8
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-26 19:25:21.590373+00 |            0 |                 |                  | 2023-11-26 19:22:52.861852+00
(1 row)

postgres=# \q

ubuntu@ip-172-31-5-233:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup

ubuntu@ip-172-31-5-233:~$ sudo su - postgres
postgres@ip-172-31-5-233:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-11-26 19:26:04.158 P00   INFO: backup command begin 2.48: --exec-id=22857-942e136d --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-block --repo1-bundle --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=s3.us-east-2.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-2 --repo1-type=s3 --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-11-26 19:26:04.924 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-11-26 19:26:05.625 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000028
2023-11-26 19:26:05.625 P00   INFO: check archive for prior segment 000000010000000000000002
2023-11-26 19:26:07.280 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-11-26 19:26:07.480 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000138
2023-11-26 19:26:07.522 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-11-26 19:26:07.860 P00   INFO: new backup label = 20231126-192604F
2023-11-26 19:26:08.275 P00   INFO: full backup size = 22.3MB, file total = 972
2023-11-26 19:26:08.275 P00   INFO: backup command end: completed successfully (4121ms)
2023-11-26 19:26:08.276 P00   INFO: expire command begin 2.48: --exec-id=22857-942e136d --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=s3.us-east-2.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-2 --repo1-type=s3 --stanza=vitaliy-test-pgcluster
2023-11-26 19:26:08.362 P00   INFO: expire command end: completed successfully (87ms)
postgres@ip-172-31-5-233:~$ 
postgres@ip-172-31-5-233:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231126-192604F
            timestamp start/stop: 2023-11-26 19:26:04+00 / 2023-11-26 19:26:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB
postgres@ip-172-31-5-233:~$

passed

vitabaks avatar Nov 26 '23 19:11 vitabaks

Test: Deploying to AWS with backups (WAL-G)

export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=************

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision=aws \
     servers_count=1 \
     server_type=t3.large \
     server_image=ami-07d445a5585fce0a1 \
     server_location=us-east-2 \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-walg-pgcluster \
     wal_g_install=true \
     WALG_AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
     WALG_AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Remove temporary SSH key 'ssh_key_tmp_xllykek' from cloud (if any)] ****************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Add SSH key 'ssh_key_tmp_xllykek' to cloud] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Gather information about default VPC] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] ***************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create S3 bucket 'backups-vitaliy-test-walg-pgcluster'] ****************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-walg-pgcluster-pgnode01)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] ******************************************************************************************************************************************
ok: [localhost] => (item=3.15.139.143)

TASK [cloud-resources : Show EC2 instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=3.15.139.143) => {
    "msg": [
        "ID: i-0275355ef89391ed9",
        "Name: vitaliy-test-walg-pgcluster-pgnode01",
        "Image: ami-07d445a5585fce0a1",
        "Type: t3.large",
        "Volume Size: 100 GB",
        "Public IP: 3.15.139.143",
        "Private IP: 172.31.1.233"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in the AWS S3 bucket] ****************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Check if WAL-G is already installed] **************************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Install lib dependencies to build WAL-G] **********************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Check the installed Go version] *******************************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Check the latest available Go version] ************************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Download Go v1.21.4] ******************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Install Go] ***************************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Download WAL-G v2.0.1 source code] ****************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Build WAL-G deps] *********************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Build and install WAL-G] **************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Add WAL-G cron jobs] ******************************************************************************************************************************************************************************
changed: [172.31.1.233] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test-walg-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [172.31.1.233] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

ubuntu@ip-172-31-1-233:~$ cat /var/lib/postgresql/.walg.json
{
  "AWS_ACCESS_KEY_ID": "******",
  "AWS_SECRET_ACCESS_KEY": "*********",
  "WALG_S3_PREFIX": "s3://backups-vitaliy-test-walg-pgcluster",
  "AWS_REGION": "us-east-2",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test-walg-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}

ubuntu@ip-172-31-1-233:~$ sudo psql -U postgres
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D40
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-26 19:44:50.612286+00 |            0 |                 |                  | 2023-11-26 19:34:09.001261+00
(1 row)

postgres=# \q
ubuntu@ip-172-31-1-233:~$ sudo su - postgres
postgres@ip-172-31-1-233:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test-walg-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@ip-172-31-1-233:~$ 
postgres@ip-172-31-1-233:~$ wal-g backup-push /pgdata/16/vitaliy-test-walg-pgcluster
INFO: 2023/11/26 19:45:22.202237 Couldn't find previous backup. Doing full backup.
INFO: 2023/11/26 19:45:22.212877 Calling pg_start_backup()
INFO: 2023/11/26 19:45:22.383367 Starting a new tar bundle
INFO: 2023/11/26 19:45:22.383416 Walking ...
INFO: 2023/11/26 19:45:22.384018 Starting part 1 ...
INFO: 2023/11/26 19:45:22.713382 Packing ...
INFO: 2023/11/26 19:45:22.714229 Finished writing part 1.
INFO: 2023/11/26 19:45:22.850398 Starting part 2 ...
INFO: 2023/11/26 19:45:22.851227 /global/pg_control
INFO: 2023/11/26 19:45:22.854428 Finished writing part 2.
INFO: 2023/11/26 19:45:22.854458 Calling pg_stop_backup()
INFO: 2023/11/26 19:45:22.908507 Starting part 3 ...
INFO: 2023/11/26 19:45:22.908578 backup_label
INFO: 2023/11/26 19:45:22.908591 tablespace_map
INFO: 2023/11/26 19:45:22.929796 Finished writing part 3.
INFO: 2023/11/26 19:45:23.121834 Wrote backup with name base_000000010000000000000003
postgres@ip-172-31-1-233:~$ 
postgres@ip-172-31-1-233:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-11-26T19:45:24Z 000000010000000000000003
postgres@ip-172-31-1-233:~$

image

passed

vitabaks avatar Nov 26 '23 20:11 vitabaks

Test: Deploying to GCP with backups (PgBackRest)

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  *********
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='gcp' \
     servers_count=1 \
     server_type='n2-standard-2' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     server_location='us-central1' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] *******************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : GCP: Gather information about project] *************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] ********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Gather information about network] *************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Extract ip_range for network 'default'] *******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify SSH public firewall] *********************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify Postgres cluster firewall] ***************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create bucket 'backups-vitaliy-test-pgcluster'] ***********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify VM instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=35.239.57.82)

TASK [cloud-resources : Show GCP instance info] ****************************************************************************************************************************************************************
ok: [localhost] => (item=35.239.57.82) => {
    "msg": [
        "ID: 1115300626491322267",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: ubuntu-2204-lts",
        "Type: n2-standard-2",
        "Volume Size: 100 GB",
        "Public IP: 35.239.57.82",
        "Private IP: 10.128.0.2"
    ]
}
...
TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in the GCS Bucket] ********************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Get GCP service account contents from localhost] ********************************************************************************************************************************************
ok: [10.128.0.2 -> localhost]

TASK [pgbackrest : Copy GCP service account contents to /var/lib/postgresql/gcs-key.json] **********************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ***************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Make sure pgdg apt key is installed] ********************************************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Make sure pgdg repository is installed] *****************************************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Update apt cache] ***************************************************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Install pgbackrest] *************************************************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Ensure spool directory exist] ***************************************************************************************************************************************************************
ok: [10.128.0.2] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] **************************************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] *****************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Make sure that the cron package is installed] ***********************************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] ************************************************************************************************************************************************
changed: [10.128.0.2] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [10.128.0.2] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ***************************************************************************************************************************************
changed: [10.128.0.2]
...

postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/pgbackrest/pgbackrest.conf 
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=gcs
repo1-path=/pgbackrest
repo1-gcs-key=/var/lib/postgresql/gcs-key.json
repo1-gcs-bucket=backups-vitaliy-test-pgcluster
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

postgres@vitaliy-test-pgcluster-pgnode01:~$ ls -l /var/lib/postgresql/gcs-key.json
-rw------- 1 postgres postgres 2309 Nov 27 13:01 /var/lib/postgresql/gcs-key.json
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: error (no valid backups)
    cipher: none

    db (current)
        wal archive min/max (16): none present
postgres@vitaliy-test-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D40
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-27 13:05:38.828697+00 |            0 |                 |                  | 2023-11-27 13:03:29.574019+00
(1 row)

postgres=# \q
postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-11-27 13:06:03.308 P00   INFO: backup command begin 2.48: --exec-id=16256-0a838f5f --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-block --repo1-bundle --repo1-gcs-bucket=backups-vitaliy-test-pgcluster --repo1-gcs-key=<redacted> --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=gcs --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-11-27 13:06:04.206 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-11-27 13:06:04.907 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000028
2023-11-27 13:06:04.907 P00   INFO: check archive for prior segment 000000010000000000000002
2023-11-27 13:06:07.633 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-11-27 13:06:07.833 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000100
2023-11-27 13:06:07.991 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-11-27 13:06:08.375 P00   INFO: new backup label = 20231127-130604F
2023-11-27 13:06:09.590 P00   INFO: full backup size = 22.3MB, file total = 972
2023-11-27 13:06:09.591 P00   INFO: backup command end: completed successfully (6285ms)
2023-11-27 13:06:09.591 P00   INFO: expire command begin 2.48: --exec-id=16256-0a838f5f --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-gcs-bucket=backups-vitaliy-test-pgcluster --repo1-gcs-key=<redacted> --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=gcs --stanza=vitaliy-test-pgcluster
2023-11-27 13:06:10.093 P00   INFO: expire command end: completed successfully (502ms)
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231127-130604F
            timestamp start/stop: 2023-11-27 13:06:04+00 / 2023-11-27 13:06:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB
postgres@vitaliy-test-pgcluster-pgnode01:~$

image

passed

vitabaks avatar Nov 27 '23 13:11 vitabaks

Test: Deploying to GCP with backups (WAL-G)

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  *********
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='gcp' \
     servers_count=1 \
     server_type='n2-standard-2' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     server_location='us-central1' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test2-pgcluster \
     wal_g_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] ********************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : GCP: Gather information about project] **************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Gather information about network] **************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Extract ip_range for network 'default'] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify SSH public firewall] **********************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify Postgres cluster firewall] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create bucket 'backups-vitaliy-test2-pgcluster'] ***********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify VM instance] ******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=35.226.211.79)

TASK [cloud-resources : Show GCP instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=35.226.211.79) => {
    "msg": [
        "ID: 2881656607585414311",
        "Name: vitaliy-test2-pgcluster-pgnode01",
        "Image: ubuntu-2204-lts",
        "Type: n2-standard-2",
        "Volume Size: 100 GB",
        "Public IP: 35.226.211.79",
        "Private IP: 10.128.0.3"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in the AWS S3 bucket] ****************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Get GCP service account contents from localhost] **************************************************************************************************************************************************
ok: [10.128.0.3 -> localhost]

TASK [wal-g : Copy GCP service account contents to /var/lib/postgresql/gcs-key.json] ****************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Check if WAL-G is already installed] **************************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Install lib dependencies to build WAL-G] **********************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Check the installed Go version] *******************************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Check the latest available Go version] ************************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Download Go v1.21.4] ******************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Install Go] ***************************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Download WAL-G v2.0.1 source code] ****************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Build WAL-G deps] *********************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Build and install WAL-G] **************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Add WAL-G cron jobs] ******************************************************************************************************************************************************************************
changed: [10.128.0.3] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [10.128.0.3] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

postgres@vitaliy-test2-pgcluster-pgnode01:~$ ls -la
total 20
drwxr-xr-x  2 postgres postgres 4096 Nov 27 13:11 .
drwxr-xr-x 41 root     root     4096 Nov 27 13:05 ..
-rw-------  1 postgres postgres  170 Nov 27 13:10 .pgpass
-rw-r--r--  1 postgres postgres  443 Nov 27 13:10 .walg.json
-rw-------  1 postgres postgres 2309 Nov 27 13:05 gcs-key.json
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat .walg.json 
{
  "GOOGLE_APPLICATION_CREDENTIALS": "/var/lib/postgresql/gcs-key.json",
  "WALG_GS_PREFIX": "gs://backups-vitaliy-test2-pgcluster",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test2-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}
postgres@vitaliy-test2-pgcluster-pgnode01:~$ ls -l /var/lib/postgresql/gcs-key.json
-rw------- 1 postgres postgres 2309 Nov 27 13:05 /var/lib/postgresql/gcs-key.json
postgres@vitaliy-test2-pgcluster-pgnode01:~$ less /var/lib/postgresql/gcs-key.json
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-27 13:22:43.167254+00 |            0 |                 |                  | 2023-11-27 13:11:39.862766+00
(1 row)

postgres=# \q
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster
INFO: 2023/11/27 13:23:10.614574 Couldn't find previous backup. Doing full backup.
INFO: 2023/11/27 13:23:10.624595 Calling pg_start_backup()
INFO: 2023/11/27 13:23:10.820279 Starting a new tar bundle
INFO: 2023/11/27 13:23:10.820321 Walking ...
INFO: 2023/11/27 13:23:10.820533 Starting part 1 ...
INFO: 2023/11/27 13:23:11.073171 Packing ...
INFO: 2023/11/27 13:23:11.073925 Finished writing part 1.
INFO: 2023/11/27 13:23:11.475888 Starting part 2 ...
INFO: 2023/11/27 13:23:11.476047 /global/pg_control
INFO: 2023/11/27 13:23:11.477486 Finished writing part 2.
INFO: 2023/11/27 13:23:11.477513 Calling pg_stop_backup()
INFO: 2023/11/27 13:23:11.527518 Starting part 3 ...
INFO: 2023/11/27 13:23:11.527613 backup_label
INFO: 2023/11/27 13:23:11.527624 tablespace_map
INFO: 2023/11/27 13:23:11.569783 Finished writing part 3.
INFO: 2023/11/27 13:23:13.046710 Wrote backup with name base_000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-11-27T13:23:12Z 000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$

image image

passed

vitabaks avatar Nov 27 '23 13:11 vitabaks

Test: Deploying to Azure with backups (PgBackRest)

export AZURE_SUBSCRIPTION_ID=*********
export AZURE_CLIENT_ID=*********
export AZURE_SECRET=*********
export AZURE_TENANT=*********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='azure' \
     servers_count=1 \
     server_type='Standard_DS1_v2' \
     server_location='eastus' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-network-interface)

TASK [cloud-resources : Azure: Create Storage Account 'vitaliytestpgcluster'] **********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Get Storage Account info] *******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: azure_storage_account_key] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create Blob Storage 'backups-vitaliy-test-pgcluster'] ***************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Show Azure VM info] ********************************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode01",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202311010', 'exactVersion': '22.04.202311010'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 172.190.155.240",
        "Private IP: 10.0.1.4"
    ]
}
...
TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in Azure Blob Storage] ****************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ***************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Make sure pgdg apt key is installed] ********************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Make sure pgdg repository is installed] *****************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Update apt cache] ***************************************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Install pgbackrest] *************************************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Ensure spool directory exist] ***************************************************************************************************************************************************************
ok: [10.0.1.4] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] **************************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] *****************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Make sure that the cron package is installed] ***********************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] ************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [10.0.1.4] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Get repo1-path value] *********************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest/stanza-create : Make sure the /pgbackrest directory exists] ***********************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ***************************************************************************************************************************************
changed: [10.0.1.4]
...

postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/pgbackrest/pgbackrest.conf 
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=azure
repo1-path=/pgbackrest
repo1-azure-key=************
repo1-azure-key-type=shared
repo1-azure-account=vitaliytestpgcluster
repo1-azure-container=backups-vitaliy-test-pgcluster
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

postgres@vitaliy-test-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver ;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-28 14:46:04.139787+00 |            0 |                 |                  | 2023-11-28 14:43:30.124269+00
(1 row)

postgres=# \q
postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-11-28 14:46:26.318 P00   INFO: backup command begin 2.49: --exec-id=11975-627d9fba --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-azure-account=<redacted> --repo1-azure-container=backups-vitaliy-test-pgcluster --repo1-azure-key=<redacted> --repo1-azure-key-type=shared --repo1-block --repo1-bundle --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=azure --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-11-28 14:46:27.058 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-11-28 14:46:27.759 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000060
2023-11-28 14:46:27.763 P00   INFO: check archive for prior segment 000000010000000000000002
2023-11-28 14:46:29.296 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-11-28 14:46:29.497 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000138
2023-11-28 14:46:29.526 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-11-28 14:46:29.840 P00   INFO: new backup label = 20231128-144626F
2023-11-28 14:46:29.990 P00   INFO: full backup size = 22.3MB, file total = 972
2023-11-28 14:46:29.990 P00   INFO: backup command end: completed successfully (3675ms)
2023-11-28 14:46:29.990 P00   INFO: expire command begin 2.49: --exec-id=11975-627d9fba --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-azure-account=<redacted> --repo1-azure-container=backups-vitaliy-test-pgcluster --repo1-azure-key=<redacted> --repo1-azure-key-type=shared --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=azure --stanza=vitaliy-test-pgcluster
2023-11-28 14:46:30.032 P00   INFO: expire command end: completed successfully (42ms)
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231128-144626F
            timestamp start/stop: 2023-11-28 14:46:26+00 / 2023-11-28 14:46:29+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB

image image

passed

vitabaks avatar Nov 28 '23 14:11 vitabaks

Test: Deploying to Azure with backups (WAL-G)

export AZURE_SUBSCRIPTION_ID=*********
export AZURE_CLIENT_ID=*********
export AZURE_SECRET=*********
export AZURE_TENANT=*********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='azure' \
     servers_count=1 \
     server_type='Standard_DS1_v2' \
     server_location='eastus' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test2-pgcluster \
     wal_g_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-network-interface)

TASK [cloud-resources : Azure: Create Storage Account 'vitaliytest2pgcluster'] *********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Get Storage Account info] *******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: azure_storage_account_key] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create Blob Storage 'backups-vitaliy-test2-pgcluster'] **************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01)

TASK [cloud-resources : Show Azure VM info] ********************************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test2-pgcluster-pgnode01",
        "Name: vitaliy-test2-pgcluster-pgnode01",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202311010', 'exactVersion': '22.04.202311010'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 20.120.18.203",
        "Private IP: 10.0.1.5"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in Azure Blob Storage] **************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Check if WAL-G is already installed] *************************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Install lib dependencies to build WAL-G] *********************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Check the installed Go version] ******************************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Check the latest available Go version] ***********************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Download Go v1.21.4] *****************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Install Go] **************************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Download WAL-G v2.0.1 source code] ***************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Build WAL-G deps] ********************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Build and install WAL-G] *************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ***********************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Make sure that the cron package is installed] ****************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Add WAL-G cron jobs] *****************************************************************************************************************************************************************************
changed: [10.0.1.5] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [10.0.1.5] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat .walg.json 
{
  "AZURE_STORAGE_ACCOUNT": "vitaliytest2pgcluster",
  "AZURE_STORAGE_ACCESS_KEY": "************",
  "WALG_AZ_PREFIX": "azure://backups-vitaliy-test2-pgcluster",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test2-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}
postgres@vitaliy-test2-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver();
ERROR:  function pg_stat_archiver() does not exist
LINE 1: select * from pg_stat_archiver();
                      ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
postgres=# select * from pg_stat_archiver ;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-28 14:24:42.322069+00 |            0 |                 |                  | 2023-11-28 14:19:28.610455+00
(1 row)

postgres=# \q
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster
INFO: 2023/11/28 14:25:14.086633 Couldn't find previous backup. Doing full backup.
INFO: 2023/11/28 14:25:14.098848 Calling pg_start_backup()
INFO: 2023/11/28 14:25:14.427354 Starting a new tar bundle
INFO: 2023/11/28 14:25:14.427384 Walking ...
INFO: 2023/11/28 14:25:14.427561 Starting part 1 ...
INFO: 2023/11/28 14:25:14.682199 Packing ...
INFO: 2023/11/28 14:25:14.683213 Finished writing part 1.
INFO: 2023/11/28 14:25:14.851083 Starting part 2 ...
INFO: 2023/11/28 14:25:14.851536 /global/pg_control
INFO: 2023/11/28 14:25:14.867399 Finished writing part 2.
INFO: 2023/11/28 14:25:14.867600 Calling pg_stop_backup()
INFO: 2023/11/28 14:25:14.967937 Starting part 3 ...
INFO: 2023/11/28 14:25:14.968095 backup_label
INFO: 2023/11/28 14:25:14.968112 tablespace_map
INFO: 2023/11/28 14:25:15.008414 Finished writing part 3.
INFO: 2023/11/28 14:25:15.167885 Wrote backup with name base_000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-11-28T14:25:15Z 000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$

image image

passed

vitabaks avatar Nov 28 '23 14:11 vitabaks

Test: Deploying to DigitalOcean with backups (PgBackRest)

export DO_API_TOKEN=*********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='digitalocean' \
     servers_count=1 \
     server_type='g-2vcpu-8gb' \
     server_image='ubuntu-22-04-x64' \
     server_location='nyc1' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true \
     digital_ocean_spaces_region=nyc3 \
     AWS_ACCESS_KEY_ID='******' \
     AWS_SECRET_ACCESS_KEY='*********'"

Note: Generate an access key and a secret key to access the Spaces API. If you don’t already have keys, you can generate by visiting the control panel’s API page - https://cloud.digitalocean.com/account/api/spaces

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] ***************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get fingerprint for SSH key 'vitaliy'] ************************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] ******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitaliy-test-pgcluster'] ************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify public firewall] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Postgres cluster firewall] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] *************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Set variable: droplet_result] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create Spaces Bucket 'backups-vitaliy-test-pgcluster'] ********************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] *******************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] ******************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=157.230.8.137)

TASK [cloud-resources : Show Droplet info] **********************************************************************************************************************************************************************
ok: [localhost] => (item=157.230.8.137) => {
    "msg": [
        "ID: 387849512",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 157.230.8.137",
        "Private IP: 10.116.0.2"
    ]
}
...
TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in DigitalOcean Spaces Object Storage] *************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Make sure pgdg apt key is installed] *********************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Make sure pgdg repository is installed] ******************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Update apt cache] ****************************************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Install pgbackrest] **************************************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Ensure spool directory exist] ****************************************************************************************************************************************************************
ok: [10.116.0.2] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] ***************************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] ******************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Make sure that the cron package is installed] ************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] *************************************************************************************************************************************************
changed: [10.116.0.2] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [10.116.0.2] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Get repo1-path value] **********************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest/stanza-create : Make sure the /pgbackrest directory exists] ************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ****************************************************************************************************************************************
changed: [10.116.0.2]
...

root@vitaliy-test-pgcluster-pgnode01:~# cat /etc/pgbackrest/pgbackrest.conf
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=s3
repo1-path=/pgbackrest
repo1-s3-key=******
repo1-s3-key-secret=*********
repo1-s3-bucket=backups-vitaliy-test-pgcluster
repo1-s3-endpoint=https://nyc3.digitaloceanspaces.com
repo1-s3-region=nyc3
repo1-s3-uri-style=path
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

root@vitaliy-test-pgcluster-pgnode01:~# su - postgres
postgres@vitaliy-test-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver ;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |         stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+------------------------------
              1 | 000000010000000000000001 | 2023-12-01 11:41:01.950786+00 |            0 |                 |                  | 2023-12-01 11:39:28.53462+00
(1 row)

postgres=# \q
postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-12-01 11:41:38.993 P00   INFO: backup command begin 2.49: --exec-id=24499-ace39a90 --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-block --repo1-bundle --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=https://nyc3.digitaloceanspaces.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=nyc3 --repo1-s3-uri-style=path --repo1-type=s3 --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-12-01 11:41:39.538 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-12-01 11:41:40.239 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000028
2023-12-01 11:41:40.239 P00   INFO: check archive for prior segment 000000010000000000000002
2023-12-01 11:41:42.018 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-12-01 11:41:42.220 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000100
2023-12-01 11:41:42.244 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-12-01 11:41:42.367 P00   INFO: new backup label = 20231201-114139F
2023-12-01 11:41:42.562 P00   INFO: full backup size = 22.3MB, file total = 972
2023-12-01 11:41:42.562 P00   INFO: backup command end: completed successfully (3571ms)
2023-12-01 11:41:42.562 P00   INFO: expire command begin 2.49: --exec-id=24499-ace39a90 --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=https://nyc3.digitaloceanspaces.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=nyc3 --repo1-s3-uri-style=path --repo1-type=s3 --stanza=vitaliy-test-pgcluster
2023-12-01 11:41:42.636 P00   INFO: expire command end: completed successfully (74ms)
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231201-114139F
            timestamp start/stop: 2023-12-01 11:41:39+00 / 2023-12-01 11:41:42+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB
postgres@vitaliy-test-pgcluster-pgnode01:~$

image image

passed

vitabaks avatar Dec 01 '23 11:12 vitabaks

Test: Deploying to DigitalOcean with backups (WAL-G)

export DO_API_TOKEN=*********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='digitalocean' \
     servers_count=1 \
     server_type='g-2vcpu-8gb' \
     server_image='ubuntu-22-04-x64' \
     server_location='nyc1' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test2-pgcluster \
     wal_g_install=true \
     digital_ocean_spaces_region=nyc3 \
     AWS_ACCESS_KEY_ID='*******' \
     AWS_SECRET_ACCESS_KEY='*********'"

Note: Generate an access key and a secret key to access the Spaces API. If you don’t already have keys, you can generate by visiting the control panel’s API page - https://cloud.digitalocean.com/account/api/spaces

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] ***************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get fingerprint for SSH key 'vitaliy'] ************************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] ******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitaliy-test2-pgcluster'] ***********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify public firewall] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Postgres cluster firewall] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] *************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01)

TASK [cloud-resources : Set variable: droplet_result] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create Spaces Bucket 'backups-vitaliy-test2-pgcluster'] *******************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] *******************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] ******************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-storage)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=67.205.134.114)

TASK [cloud-resources : Show Droplet info] **********************************************************************************************************************************************************************
ok: [localhost] => (item=67.205.134.114) => {
    "msg": [
        "ID: 387857092",
        "Name: vitaliy-test2-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 67.205.134.114",
        "Private IP: 10.116.0.3"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in DigitalOcean Spaces Object Storage] ***********************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Check if WAL-G is already installed] **************************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Install lib dependencies to build WAL-G] **********************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Check the installed Go version] *******************************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Check the latest available Go version] ************************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Download Go v1.21.4] ******************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Install Go] ***************************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Download WAL-G v2.0.1 source code] ****************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Build WAL-G deps] *********************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Build and install WAL-G] **************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Add WAL-G cron jobs] ******************************************************************************************************************************************************************************
changed: [10.116.0.3] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [10.116.0.3] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

root@vitaliy-test2-pgcluster-pgnode01:~# su - postgres
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat .walg.json 
{
  "AWS_ACCESS_KEY_ID": "******",
  "AWS_SECRET_ACCESS_KEY": "*********",
  "AWS_ENDPOINT": "https://nyc3.digitaloceanspaces.com",
  "AWS_REGION": "nyc3",
  "AWS_S3_FORCE_PATH_STYLE": "True",
  "WALG_S3_PREFIX": "s3://backups-vitaliy-test2-pgcluster",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test2-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}
postgres@vitaliy-test2-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-12-01 11:57:20.657624+00 |            0 |                 |                  | 2023-12-01 11:56:25.051441+00
(1 row)

postgres=# \q
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster
INFO: 2023/12/01 11:57:39.466278 Couldn't find previous backup. Doing full backup.
INFO: 2023/12/01 11:57:39.477022 Calling pg_start_backup()
INFO: 2023/12/01 11:57:39.620068 Starting a new tar bundle
INFO: 2023/12/01 11:57:39.620108 Walking ...
INFO: 2023/12/01 11:57:39.620422 Starting part 1 ...
INFO: 2023/12/01 11:57:39.912292 Packing ...
INFO: 2023/12/01 11:57:39.913277 Finished writing part 1.
INFO: 2023/12/01 11:57:39.985622 Starting part 2 ...
INFO: 2023/12/01 11:57:39.985652 /global/pg_control
INFO: 2023/12/01 11:57:39.986493 Finished writing part 2.
INFO: 2023/12/01 11:57:39.986617 Calling pg_stop_backup()
INFO: 2023/12/01 11:57:40.030028 Starting part 3 ...
INFO: 2023/12/01 11:57:40.030118 backup_label
INFO: 2023/12/01 11:57:40.030134 tablespace_map
INFO: 2023/12/01 11:57:40.047375 Finished writing part 3.
INFO: 2023/12/01 11:57:40.125614 Wrote backup with name base_000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-12-01T11:57:40Z 000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$

image image

passed

vitabaks avatar Dec 01 '23 12:12 vitabaks

Test: Amazon Elastic Load Balancer (ELB)

Commit: 12b662206cbb7aa33e2aa64bfd9804673d4d8c4c aa0c104d4bc8ef8cb947ee4265e6979158bd4576

export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=******

docker run --rm -it \
  --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
  --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  vitabaks/postgresql_cluster:cloud \
    ansible-playbook deploy_pgcluster.yml --extra-vars \
      "ansible_user=ubuntu \
       cloud_provider='aws' \
       servers_count=3 \
       server_type=m5.xlarge \
       server_image=ami-0f1eef1771f0c6b68 \
       server_location=us-east-2 \
       volume_size=100 \
       ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
       pgbouncer_install=true \
       postgresql_version=16 \
       patroni_cluster_name=vitabaks-pgcluster \
       cloud_load_balancer=true"

Result:

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'python3-pip' package is present on controlling host] ***********************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Remove temporary SSH key 'ssh_key_tmp_rsqjych' from cloud (if any)] ****************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Add SSH key 'ssh_key_tmp_rsqjych' to cloud] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Gather information about default VPC] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] ***************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] ******************************************************************************************************************************************
ok: [localhost] => (item=3.21.158.196)
ok: [localhost] => (item=3.144.85.6)
ok: [localhost] => (item=3.21.105.210)

TASK [cloud-resources : AWS: Create or modify Elastic Load Balancer (ELB)] **************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Show EC2 instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=3.21.158.196) => {
    "msg": [
        "ID: i-00ad8e2bd22e27a9a",
        "Name: vitabaks-pgcluster-pgnode01",
        "Image: ami-0f1eef1771f0c6b68",
        "Type: m5.xlarge",
        "Volume Size: 100 GB",
        "Public IP: 3.21.158.196",
        "Private IP: 172.31.5.183"
    ]
}
ok: [localhost] => (item=3.144.85.6) => {
    "msg": [
        "ID: i-03c9c4021aa958e79",
        "Name: vitabaks-pgcluster-pgnode02",
        "Image: ami-0f1eef1771f0c6b68",
        "Type: m5.xlarge",
        "Volume Size: 100 GB",
        "Public IP: 3.144.85.6",
        "Private IP: 172.31.0.82"
    ]
}
ok: [localhost] => (item=3.21.105.210) => {
    "msg": [
        "ID: i-06e840e3b99754b83",
        "Name: vitabaks-pgcluster-pgnode03",
        "Image: ami-0f1eef1771f0c6b68",
        "Type: m5.xlarge",
        "Volume Size: 100 GB",
        "Public IP: 3.21.105.210",
        "Private IP: 172.31.7.30"
    ]
}

...

TASK [deploy-finish : Postgres list of users] *******************************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] ***************************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "dbs_result.stdout_lines": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] ********************************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitabaks-pgcluster (7328427680838806347) ---+----+-----------+",
        "| Member          | Host         | Role    | State     | TL | Lag in MB |",
        "+-----------------+--------------+---------+-----------+----+-----------+",
        "| ip-172-31-0-82  | 172.31.0.82  | Replica | streaming |  1 |         0 |",
        "| ip-172-31-5-183 | 172.31.5.183 | Leader  | running   |  1 |           |",
        "| ip-172-31-7-30  | 172.31.7.30  | Replica | streaming |  1 |         0 |",
        "+-----------------+--------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Postgres Cluster connection info] *********************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "msg": [
        "+-------------------------------------------------------------------------+",
        "DNS (primary): internal-vitabaks-pgcluster-primary-1466392758.us-east-2.elb.amazonaws.com ",
        "DNS (replica): internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com ",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: ZBhh9eDRVUtj6f2tzqeIS8xKxfUvGHaK",
        "+-------------------------------------------------------------------------+"
    ]
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
172.31.0.82                : ok=112  changed=65   unreachable=0    failed=0    skipped=367  rescued=0    ignored=0   
172.31.5.183               : ok=130  changed=68   unreachable=0    failed=0    skipped=409  rescued=0    ignored=0   
172.31.7.30                : ok=112  changed=65   unreachable=0    failed=0    skipped=367  rescued=0    ignored=0   
localhost                  : ok=25   changed=6    unreachable=0    failed=0    skipped=157  rescued=0    ignored=0

Check primary:

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-primary-1466392758.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting     |  role   
------------------------+---------
 172.31.5.183,127.0.0.1 | primary
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-primary-1466392758.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting     |  role   
------------------------+---------
 172.31.5.183,127.0.0.1 | primary
(1 row)

Check replicas:

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.0.82,127.0.0.1 | replica
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.7.30,127.0.0.1 | replica
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.0.82,127.0.0.1 | replica
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.7.30,127.0.0.1 | replica
(1 row)

image image

passed

vitabaks avatar Jan 26 '24 15:01 vitabaks

Test: DigitalOcean Load Balancer

commit: 55fd8e50f92092f6db52281052af087ffb0a630f 9f95b16d40ab93fb3ac7c7c1a086dc4135501896

export DO_API_TOKEN=*************

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider=digitalocean \
     server_count=3 \
     server_location=nyc1 \
     server_type=g-2vcpu-8gb \
     server_image=ubuntu-22-04-x64 \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\"
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitabaks-pgcluster \
     cloud_load_balancer=true"

Result:


PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get fingerprint for all SSH keys] ****************************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] *****************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitabaks-pgcluster'] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify public firewall] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Postgres cluster firewall] ******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] ************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Set variable: droplet_result] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] ******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] *****************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=67.205.177.108)
ok: [localhost] => (item=134.209.64.218)
ok: [localhost] => (item=157.230.92.31)

TASK [cloud-resources : Set variable: digital_ocean_load_balancer_port] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: digital_ocean_load_balancer_target_port] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Load Balancer] ******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : DigitalOcean: Gather information about load balancers] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Show Droplet info] *********************************************************************************************************************************************************************
ok: [localhost] => (item=67.205.177.108) => {
    "msg": [
        "ID: 403793886",
        "Name: vitabaks-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 67.205.177.108",
        "Private IP: 10.116.0.2"
    ]
}
ok: [localhost] => (item=134.209.64.218) => {
    "msg": [
        "ID: 403793973",
        "Name: vitabaks-pgcluster-pgnode02",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 134.209.64.218",
        "Private IP: 10.116.0.3"
    ]
}
ok: [localhost] => (item=157.230.92.31) => {
    "msg": [
        "ID: 403794046",
        "Name: vitabaks-pgcluster-pgnode03",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 157.230.92.31",
        "Private IP: 10.116.0.4"
    ]
}

...

TASK [deploy-finish : Postgres list of users] ******************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] **************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "dbs_result.stdout_lines": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] *******************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitabaks-pgcluster (7340630136958481870) -+-----------+----+-----------+",
        "| Member                      | Host       | Role    | State     | TL | Lag in MB |",
        "+-----------------------------+------------+---------+-----------+----+-----------+",
        "| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |",
        "| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |",
        "| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |",
        "+-----------------------------+------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Postgres Cluster connection info] ********************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "msg": [
        "+-------------------------------------------------------------------------+",
        "IP Address (primary): 143.244.212.53 ",
        "IP Address (replica): 167.172.12.113 ",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: 4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP",
        "+-------------------------------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.116.0.2                 : ok=130  changed=65   unreachable=0    failed=0    skipped=414  rescued=0    ignored=0   
10.116.0.3                 : ok=111  changed=62   unreachable=0    failed=0    skipped=372  rescued=0    ignored=0   
10.116.0.4                 : ok=111  changed=62   unreachable=0    failed=0    skipped=372  rescued=0    ignored=0   
localhost                  : ok=25   changed=4    unreachable=0    failed=0    skipped=172  rescued=0    ignored=0

Check primary:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.2,127.0.0.1 | primary
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.2,127.0.0.1 | primary
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.2,127.0.0.1 | primary
(1 row)

Check replicas:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.4,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.3,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.4,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.3,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.4,127.0.0.1 | replica
(1 row)

Switchover:

root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+-----------+----+-----------+
| Member                      | Host       | Role    | State     | TL | Lag in MB |
+-----------------------------+------------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |
+-----------------------------+------------+---------+-----------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# patronictl switchover
Current cluster topology
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+-----------+----+-----------+
| Member                      | Host       | Role    | State     | TL | Lag in MB |
+-----------------------------+------------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |
+-----------------------------+------------+---------+-----------+----+-----------+
Primary [vitabaks-pgcluster-pgnode01]: 
Candidate ['vitabaks-pgcluster-pgnode02', 'vitabaks-pgcluster-pgnode03'] []: vitabaks-pgcluster-pgnode02
When should the switchover take place (e.g. 2024-02-28T13:42 )  [now]: 
Are you sure you want to switchover cluster vitabaks-pgcluster, demoting current leader vitabaks-pgcluster-pgnode01? [y/N]: y
2024-02-28 12:42:21.57382 Successfully switched over to "vitabaks-pgcluster-pgnode02"
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+---------+----+-----------+
| Member                      | Host       | Role    | State   | TL | Lag in MB |
+-----------------------------+------------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Replica | stopped |    |   unknown |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Leader  | running |  1 |           |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | running |  1 |         0 |
+-----------------------------+------------+---------+---------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# 
root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+---------+----+-----------+
| Member                      | Host       | Role    | State   | TL | Lag in MB |
+-----------------------------+------------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Replica | running |  1 |         0 |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Leader  | running |  2 |           |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | running |  1 |         0 |
+-----------------------------+------------+---------+---------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# 

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.3,127.0.0.1 | primary
(1 row)

image image image

passed

vitabaks avatar Feb 28 '24 12:02 vitabaks

Test: Hetzner Load Balancer

Commit: 43022caeb072bb2389ea46f5af49c45404252aab 33c93a3562716ad6ae2b5e7c9c2b50d66fbe9b77

export HCLOUD_API_TOKEN=*********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider=hetzner \
     server_count=3 \
     server_type=CCX13 \
     server_image=ubuntu-22.04 \
     server_location=ash \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\"
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitabaks-pgcluster \
     cloud_load_balancer=true \
     database_public_access=false"

Result:


PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'hcloud' dependency is present on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Hetzner Cloud: Gather information about SSH key 'vitaliy'] ******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: ssh_key_names] ************************************************************************************************************************************************************
ok: [localhost] => (item=None)
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Gather information about network zones] **********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Extract network zone for server_location] ********************************************************************************************************************************
ok: [localhost] => (item=network_zone: us-east)

TASK [cloud-resources : Hetzner Cloud: Gather information about networks] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a network 'postgres-cluster-network-us-east'] *********************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a subnetwork in network 'postgres-cluster-network-us-east'] *******************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify public firewall] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify Postgres cluster firewall] ******************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify server] *************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Add server to network 'postgres-cluster-network-us-east'] ****************************************************************************************************************
ok: [localhost] => (item=vitabaks-pgcluster-pgnode01)
ok: [localhost] => (item=vitabaks-pgcluster-pgnode02)
ok: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Create or modify volume] *************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=5.161.59.224)
ok: [localhost] => (item=5.161.207.40)
ok: [localhost] => (item=5.161.181.46)

TASK [cloud-resources : Hetzner Cloud: Create or modify Load Balancer] ******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Configure Load Balancer service] *****************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Add Load Balancer to network 'postgres-cluster-network-us-east'] *********************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Disable public interface for Load Balancer] ******************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Add servers to Load Balancer (use label_selector 'cluster=vitabaks-pgcluster')] ******************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Gather information about Load Balancers] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Show Server info] ***********************************************************************************************************************************************************************
ok: [localhost] => (item=5.161.59.224) => {
    "msg": [
        "ID: 44673953",
        "Name: vitabaks-pgcluster-pgnode01",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.59.224",
        "Private IP: 10.0.1.1"
    ]
}
ok: [localhost] => (item=5.161.207.40) => {
    "msg": [
        "ID: 44673966",
        "Name: vitabaks-pgcluster-pgnode02",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.207.40",
        "Private IP: 10.0.1.2"
    ]
}
ok: [localhost] => (item=5.161.181.46) => {
    "msg": [
        "ID: 44674000",
        "Name: vitabaks-pgcluster-pgnode03",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.181.46",
        "Private IP: 10.0.1.3"
    ]
}

...


TASK [deploy-finish : Postgres list of users] *******************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] ***************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "dbs_result.stdout_lines": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] ********************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+",
        "| Member                      | Host     | Role    | State     | TL | Lag in MB |",
        "+-----------------------------+----------+---------+-----------+----+-----------+",
        "| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |",
        "| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |",
        "| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |",
        "+-----------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Postgres Cluster connection info] *********************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "msg": [
        "+-------------------------------------------------------------------------+",
        "IP Address (primary): 10.0.1.4 ",
        "IP Address (replica): 10.0.1.5 ",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: kSVSQIWwUiZKY6N5iyJP9ikcaMHDhFXZ",
        "+-------------------------------------------------------------------------+"
    ]
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
10.0.1.1                   : ok=133  changed=69   unreachable=0    failed=0    skipped=408  rescued=0    ignored=0   
10.0.1.2                   : ok=113  changed=66   unreachable=0    failed=0    skipped=366  rescued=0    ignored=0   
10.0.1.3                   : ok=113  changed=66   unreachable=0    failed=0    skipped=366  rescued=0    ignored=0   
localhost                  : ok=29   changed=9    unreachable=0    failed=0    skipped=175  rescued=0    ignored=0

Check primary:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=kSVSQIWwUiZKY6N5iyJP9ikcaMHDhFXZ psql -h 10.0.1.4 -p 6432 -U postgres -d postgres -c "select pg_is_in_recovery()"
 pg_is_in_recovery 
-------------------
 f
(1 row)

Check replica:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=kSVSQIWwUiZKY6N5iyJP9ikcaMHDhFXZ psql -h 10.0.1.5 -p 6432 -U postgres -d postgres -c "select pg_is_in_recovery()"
 pg_is_in_recovery 
-------------------
 t
(1 row)

image image image

Switchover:

root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# patronictl switchover vitabaks-pgcluster
Current cluster topology
+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+
Primary [vitabaks-pgcluster-pgnode01]: 
Candidate ['vitabaks-pgcluster-pgnode02', 'vitabaks-pgcluster-pgnode03'] []: vitabaks-pgcluster-pgnode02    
When should the switchover take place (e.g. 2024-03-15T14:37 )  [now]: 
Are you sure you want to switchover cluster vitabaks-pgcluster, demoting current leader vitabaks-pgcluster-pgnode01? [y/N]: y
2024-03-15 13:37:33.96476 Successfully switched over to "vitabaks-pgcluster-pgnode02"
+ Cluster: vitabaks-pgcluster (7346582435914755030) ---------+----+-----------+
| Member                      | Host     | Role    | State   | TL | Lag in MB |
+-----------------------------+----------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Replica | stopped |    |   unknown |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Leader  | running |  1 |           |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | running |  1 |         0 |
+-----------------------------+----------+---------+---------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# 
root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Replica | streaming |  2 |         0 |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Leader  | running   |  2 |           |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  2 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+

image

Check Replica load balancer

image

passed

vitabaks avatar Mar 15 '24 13:03 vitabaks

Test: GCP Load Balancer

Commit: https://github.com/vitabaks/postgresql_cluster/pull/464/commits/2b98634aff57a5711e7e865f8f5a3882e81cf211

Note: Currently, only Global External classic proxy Network Load Balancer is supported.

ansible-playbook deploy_pgcluster.yml --extra-vars \
  "ansible_user=root \
   cloud_provider='gcp' \
   cloud_load_balancer=true \
   server_spot=true \
   server_count=3 \
   server_type='e2-medium' \
   server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
   server_location='us-east1' \
   system_volume_size=50 \
   volume_size=50 \
   postgresql_version=16 \
   pgbouncer_install=true \
   patroni_cluster_name=vitabaks-pgcluster \
   ssh_public_keys='ssh-rsa AAAAB3NzaC1yc2EAAA*****3wX2uPq35NlVL6Bn/whzcMINzKKCc7AVGbki'"

Result:


PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] ********************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Lookup the GCP_SERVICE_ACCOUNT_CONTENTS environmental variable] *************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_service_account_contents] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Gather information about project] **************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Gather information about network] **************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Extract ip_range for network 'default'] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify SSH public firewall rule] *****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify Postgres cluster firewall rule] ***********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create health checks and LB firewall rule] *****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify VM instance] ******************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : GCP: [Load Balancer] Create instance group] *********************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: [Load Balancer] Create health check] ***********************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary-hc)
changed: [localhost] => (item=vitabaks-pgcluster-replica-hc)

TASK [cloud-resources : GCP: [Load Balancer] Create backend service] ********************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : GCP: [Load Balancer] Create target TCP proxy] *******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary-proxy)
changed: [localhost] => (item=vitabaks-pgcluster-replica-proxy)

TASK [cloud-resources : GCP: [Load Balancer] Reserve static IP address] *****************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary-ip)
changed: [localhost] => (item=vitabaks-pgcluster-replica-ip)

TASK [cloud-resources : GCP: [Load Balancer] Create forwarding rule] ********************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary-fr)
changed: [localhost] => (item=vitabaks-pgcluster-replica-fr)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=34.75.150.237)
ok: [localhost] => (item=34.74.24.56)
ok: [localhost] => (item=34.148.172.232)

TASK [cloud-resources : Server info] ****************************************************************************************************************************************************************************
ok: [localhost] => (item=34.75.150.237) => {
    "msg": {
        "id": "7188753037207374462",
        "image": "ubuntu-2204-lts",
        "name": "vitabaks-pgcluster-pgnode01",
        "private_ip": "10.142.0.16",
        "public_ip": "34.75.150.237",
        "type": "e2-medium",
        "volume_size": "50 GB"
    }
}
ok: [localhost] => (item=34.74.24.56) => {
    "msg": {
        "id": "4278690618588367475",
        "image": "ubuntu-2204-lts",
        "name": "vitabaks-pgcluster-pgnode02",
        "private_ip": "10.142.0.18",
        "public_ip": "34.74.24.56",
        "type": "e2-medium",
        "volume_size": "50 GB"
    }
}
ok: [localhost] => (item=34.148.172.232) => {
    "msg": {
        "id": "2008758823073974856",
        "image": "ubuntu-2204-lts",
        "name": "vitabaks-pgcluster-pgnode03",
        "private_ip": "10.142.15.229",
        "public_ip": "34.148.172.232",
        "type": "e2-medium",
        "volume_size": "50 GB"
    }
}
...

TASK [deploy-finish : Postgres list of users] *******************************************************************************************************************************************************************
ok: [10.142.0.16] => {
    "msg": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] ***************************************************************************************************************************************************************
ok: [10.142.0.16] => {
    "msg": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] ********************************************************************************************************************************************************************
ok: [10.142.0.16] => {
    "msg": [
        "+ Cluster: vitabaks-pgcluster (7380334332110398205) ----+-----------+----+-----------+",
        "| Member                      | Host          | Role    | State     | TL | Lag in MB |",
        "+-----------------------------+---------------+---------+-----------+----+-----------+",
        "| vitabaks-pgcluster-pgnode01 | 10.142.0.16   | Leader  | running   |  1 |           |",
        "| vitabaks-pgcluster-pgnode02 | 10.142.0.18   | Replica | streaming |  1 |         0 |",
        "| vitabaks-pgcluster-pgnode03 | 10.142.15.229 | Replica | streaming |  1 |         0 |",
        "+-----------------------------+---------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Connection info] **************************************************************************************************************************************************************************
ok: [10.142.0.16] => {
    "msg": {
        "address": {
            "primary": "34.160.248.138",
            "replica": "34.160.84.41"
        },
        "password": "j0C2fajhB9NFCrGlePz7r0DwOk7TzaYG",
        "port": "6432",
        "superuser": "postgres"
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
10.142.0.16                : ok=123  changed=62   unreachable=0    failed=0    skipped=419  rescued=0    ignored=0   
10.142.0.18                : ok=104  changed=59   unreachable=0    failed=0    skipped=381  rescued=0    ignored=0   
10.142.15.229              : ok=104  changed=59   unreachable=0    failed=0    skipped=381  rescued=0    ignored=0   
localhost                  : ok=30   changed=11   unreachable=0    failed=0    skipped=186  rescued=0    ignored=0

Screenshots:

image

image

  • 1 healthy only - primary

image

  • 2 healthy - replicas

Check Primary

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=j0C2fajhB9NFCrGlePz7r0DwOk7TzaYG psql -h 34.160.248.138 -p 6432 -U postgres -d postgres -c "select pg_is_in_recovery()"
 pg_is_in_recovery 
-------------------
 f
(1 row)

Check Replica

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=j0C2fajhB9NFCrGlePz7r0DwOk7TzaYG psql -h 34.160.84.41 -p 6432 -U postgres -d postgres -c "select pg_is_in_recovery()"
 pg_is_in_recovery 
-------------------
 t
(1 row)

Switchover

postgres@vitabaks-pgcluster-pgnode01:~$ patronictl switchover vitabaks-pgcluster
Current cluster topology
+ Cluster: vitabaks-pgcluster (7380334332110398205) ----+-----------+----+-----------+
| Member                      | Host          | Role    | State     | TL | Lag in MB |
+-----------------------------+---------------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.142.0.16   | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.142.0.18   | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.142.15.229 | Replica | streaming |  1 |         0 |
+-----------------------------+---------------+---------+-----------+----+-----------+
Primary [vitabaks-pgcluster-pgnode01]: 
Candidate ['vitabaks-pgcluster-pgnode02', 'vitabaks-pgcluster-pgnode03'] []: vitabaks-pgcluster-pgnode02
When should the switchover take place (e.g. 2024-06-14T13:33 )  [now]: 
Are you sure you want to switchover cluster vitabaks-pgcluster, demoting current leader vitabaks-pgcluster-pgnode01? [y/N]: y
2024-06-14 12:33:20.17774 Successfully switched over to "vitabaks-pgcluster-pgnode02"
+ Cluster: vitabaks-pgcluster (7380334332110398205) ----+---------+----+-----------+
| Member                      | Host          | Role    | State   | TL | Lag in MB |
+-----------------------------+---------------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.142.0.16   | Replica | stopped |    |   unknown |
| vitabaks-pgcluster-pgnode02 | 10.142.0.18   | Leader  | running |  1 |           |
| vitabaks-pgcluster-pgnode03 | 10.142.15.229 | Replica | running |  1 |         0 |
+-----------------------------+---------------+---------+---------+----+-----------+

result:

              now              | pg_is_in_recovery 
-------------------------------+-------------------
 2024-06-14 12:33:15.424919+00 | f
(1 row)

              now              | pg_is_in_recovery 
-------------------------------+-------------------
 2024-06-14 12:33:16.487728+00 | f
(1 row)

             now              | pg_is_in_recovery 
------------------------------+-------------------
 2024-06-14 12:33:17.54989+00 | f
(1 row)

psql: error: connection to server at "34.160.248.138", port 6432 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
psql: error: connection to server at "34.160.248.138", port 6432 failed: FATAL:  server login has been failing, try again later (server_login_retry)
psql: error: connection to server at "34.160.248.138", port 6432 failed: FATAL:  server login has been failing, try again later (server_login_retry)
psql: error: connection to server at "34.160.248.138", port 6432 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
psql: error: connection to server at "34.160.248.138", port 6432 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
psql: error: connection to server at "34.160.248.138", port 6432 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
psql: error: connection to server at "34.160.248.138", port 6432 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
              now              | pg_is_in_recovery 
-------------------------------+-------------------
 2024-06-14 12:33:35.953383+00 | f
(1 row)

             now              | pg_is_in_recovery 
------------------------------+-------------------
 2024-06-14 12:33:37.01567+00 | f
(1 row)
  • the connection was automatically redirected to the new Primary server

passed

vitabaks avatar Jun 14 '24 12:06 vitabaks

Test: Azure Load Balancer

ansible-playbook deploy_pgcluster.yml --extra-vars \
  "ansible_user=azureadmin \
    cloud_provider='azure' \
    cloud_load_balancer=true \
    database_public_access=false \
    server_spot=true \
    server_count=3 \
    server_type='Standard_D2s_v5' \
    server_location='eastus' \
    system_volume_size=50 \
    volume_size=50 \
    pgbouncer_install=true \
    postgresql_version=16 \
    patroni_cluster_name=vitabaks-pgcluster \
    pgbackrest_auto_conf=false \
    ssh_public_keys='ssh-rsa AAAAB3NzaC1yc2EAAAA************KKCc7AVGbk='"

result:

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] ******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Check if Azure CLI is installed] *******************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Gather information about network] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-public-ip)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-public-ip)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-network-interface)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-network-interface)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-network-interface)

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Azure: Create or modify Load Balancer] *************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Extract virtual machine private IPs] ***************************************************************************************************************************************************
ok: [localhost] => (item=10.0.1.4)
ok: [localhost] => (item=10.0.1.5)
ok: [localhost] => (item=10.0.1.6)

TASK [cloud-resources : Azure: Add virtual machine IP addresses to Load Balancer backend pool] *****************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary-backend)
changed: [localhost] => (item=vitabaks-pgcluster-replica-backend)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=vitabaks-pgcluster-pgnode01)
ok: [localhost] => (item=vitabaks-pgcluster-pgnode02)
ok: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Server info] ***************************************************************************************************************************************************************************
ok: [localhost] => (item=vitabaks-pgcluster-pgnode01) => {
    "msg": {
        "id": "/subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group-eastus/providers/Microsoft.Compute/virtualMachines/vitabaks-pgcluster-pgnode01",
        "image": {
            "exact_version": "22.04.202408010",
            "offer": "0001-com-ubuntu-server-jammy",
            "publisher": "Canonical",
            "sku": "22_04-lts-gen2",
            "version": "22.04.202408010"
        },
        "name": "vitabaks-pgcluster-pgnode01",
        "private_ip": "10.0.1.4",
        "public_ip": "40.71.77.217",
        "type": "Standard_D2s_v5",
        "volume_size": "50 GB",
        "volume_type": "StandardSSD_LRS"
    }
}
ok: [localhost] => (item=vitabaks-pgcluster-pgnode02) => {
    "msg": {
        "id": "/subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group-eastus/providers/Microsoft.Compute/virtualMachines/vitabaks-pgcluster-pgnode02",
        "image": {
            "exact_version": "22.04.202408010",
            "offer": "0001-com-ubuntu-server-jammy",
            "publisher": "Canonical",
            "sku": "22_04-lts-gen2",
            "version": "22.04.202408010"
        },
        "name": "vitabaks-pgcluster-pgnode02",
        "private_ip": "10.0.1.5",
        "public_ip": "40.71.93.63",
        "type": "Standard_D2s_v5",
        "volume_size": "50 GB",
        "volume_type": "StandardSSD_LRS"
    }
}
ok: [localhost] => (item=vitabaks-pgcluster-pgnode03) => {
    "msg": {
        "id": "/subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group-eastus/providers/Microsoft.Compute/virtualMachines/vitabaks-pgcluster-pgnode03",
        "image": {
            "exact_version": "22.04.202408010",
            "offer": "0001-com-ubuntu-server-jammy",
            "publisher": "Canonical",
            "sku": "22_04-lts-gen2",
            "version": "22.04.202408010"
        },
        "name": "vitabaks-pgcluster-pgnode03",
        "private_ip": "10.0.1.6",
        "public_ip": "52.234.209.131",
        "type": "Standard_D2s_v5",
        "volume_size": "50 GB",
        "volume_type": "StandardSSD_LRS"
    }
}

...

TASK [deploy-finish : Postgres list of users] ******************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] **************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] *******************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": [
        "+ Cluster: vitabaks-pgcluster (7404174849573692915) -----------+----+-----------+",
        "| Member                      | Host     | Role    | State     | TL | Lag in MB |",
        "+-----------------------------+----------+---------+-----------+----+-----------+",
        "| vitabaks-pgcluster-pgnode01 | 10.0.1.4 | Leader  | running   |  1 |           |",
        "| vitabaks-pgcluster-pgnode02 | 10.0.1.5 | Replica | streaming |  1 |         0 |",
        "| vitabaks-pgcluster-pgnode03 | 10.0.1.6 | Replica | streaming |  1 |         0 |",
        "+-----------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Connection info] *************************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": {
        "address": {
            "primary": "10.0.1.7",
            "replica": "10.0.1.8"
        },
        "password": "FKE06XeePXDJAWMuGpRvUBkAh5auxdAD",
        "port": "6432",
        "superuser": "postgres"
    }
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.0.1.4                   : ok=125  changed=60   unreachable=0    failed=0    skipped=422  rescued=0    ignored=0   
10.0.1.5                   : ok=104  changed=57   unreachable=0    failed=0    skipped=380  rescued=0    ignored=0   
10.0.1.6                   : ok=104  changed=57   unreachable=0    failed=0    skipped=380  rescued=0    ignored=0   
localhost                  : ok=28   changed=9    unreachable=0    failed=0    skipped=196  rescued=0    ignored=0

image image

Check Primary

azureadmin@test-client-pgnode01:~$ PGPASSWORD=FKE06XeePXDJAWMuGpRvUBkAh5auxdAD psql -h 10.0.1.7 -p 6432 -U postgres -d postgres -c "select now(), pg_is_in_recovery()";
              now              | pg_is_in_recovery 
-------------------------------+-------------------
 2024-08-17 18:30:10.132455+00 | f
(1 row)

Check Replica

azureadmin@test-client-pgnode01:~$ PGPASSWORD=FKE06XeePXDJAWMuGpRvUBkAh5auxdAD psql -h 10.0.1.8 -p 6432 -U postgres -d postgres -c "select now(), pg_is_in_recovery()";
              now              | pg_is_in_recovery 
-------------------------------+-------------------
 2024-08-17 18:30:21.691474+00 | t
(1 row)

Test switchover/failover

postgres@vitabaks-pgcluster-pgnode01:~$ patronictl switchover vitabaks-pgcluster
Current cluster topology
+ Cluster: vitabaks-pgcluster (7404174849573692915) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.4 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.0.1.5 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.6 | Replica | streaming |  1 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+
Primary [vitabaks-pgcluster-pgnode01]: 
Candidate ['vitabaks-pgcluster-pgnode02', 'vitabaks-pgcluster-pgnode03'] []: vitabaks-pgcluster-pgnode03
When should the switchover take place (e.g. 2024-08-17T19:33 )  [now]: 
Are you sure you want to switchover cluster vitabaks-pgcluster, demoting current leader vitabaks-pgcluster-pgnode01? [y/N]: y
2024-08-17 18:33:58.22043 Successfully switched over to "vitabaks-pgcluster-pgnode03"
+ Cluster: vitabaks-pgcluster (7404174849573692915) ---------+----+-----------+
| Member                      | Host     | Role    | State   | TL | Lag in MB |
+-----------------------------+----------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.4 | Replica | stopped |    |   unknown |
| vitabaks-pgcluster-pgnode02 | 10.0.1.5 | Replica | running |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.6 | Leader  | running |  1 |           |
+-----------------------------+----------+---------+---------+----+-----------+
postgres@vitabaks-pgcluster-pgnode01:~$ patronictl list
+ Cluster: vitabaks-pgcluster (7404174849573692915) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.4 | Replica | streaming |  2 |         0 |
| vitabaks-pgcluster-pgnode02 | 10.0.1.5 | Replica | streaming |  2 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.6 | Leader  | running   |  2 |           |
+-----------------------------+----------+---------+-----------+----+-----------+

check:

azureadmin@test-client-pgnode01:~$ for i in {1..600}; do PGPASSWORD=FKE06XeePXDJAWMuGpRvUBkAh5auxdAD psql -h 10.0.1.7 -p 6432 -U postgres -d postgres -tAXc "select now(), pg_is_in_recovery()"; sleep 1; done
2024-08-17 18:33:32.651116+00|f
2024-08-17 18:33:33.699763+00|f
2024-08-17 18:33:34.75129+00|f
2024-08-17 18:33:35.800452+00|f
2024-08-17 18:33:36.84978+00|f
2024-08-17 18:33:37.904403+00|f
2024-08-17 18:33:38.954316+00|f
2024-08-17 18:33:40.004996+00|f
2024-08-17 18:33:41.055422+00|f
2024-08-17 18:33:42.110512+00|f
2024-08-17 18:33:43.160571+00|f
2024-08-17 18:33:44.211893+00|f
2024-08-17 18:33:45.268437+00|f
2024-08-17 18:33:46.32029+00|f
2024-08-17 18:33:47.374771+00|f
2024-08-17 18:33:48.427755+00|f
2024-08-17 18:33:49.480862+00|f
2024-08-17 18:33:50.532497+00|f
2024-08-17 18:33:51.587349+00|f
2024-08-17 18:33:52.644482+00|f
2024-08-17 18:33:53.695687+00|f
2024-08-17 18:33:54.755684+00|f
2024-08-17 18:33:55.81036+00|f
2024-08-17 18:34:12.092555+00|t
2024-08-17 18:34:13.146098+00|f
2024-08-17 18:34:14.195198+00|f
2024-08-17 18:34:15.251599+00|f
2024-08-17 18:34:16.30437+00|f
2024-08-17 18:34:17.362922+00|f
2024-08-17 18:34:18.422107+00|f
2024-08-17 18:34:19.472027+00|f
2024-08-17 18:34:20.52696+00|f
2024-08-17 18:34:21.585346+00|f
2024-08-17 18:34:22.635402+00|f
2024-08-17 18:34:23.693377+00|f
2024-08-17 18:34:24.74441+00|f
  • switching to the new primary has been completed
  • the downtime is ~16 seconds (18:33:55 - 18:34:12)

image

passed

Note: Azure load balancer doesn’t support the scenario of accessing the frontend of internal load balancer from the participating backend pool virtual machine. Please refer this document for detailed information.
This way, you can successfully connect to the database using the IP address of the load balancer from any other virtual machines, but not from the database server itself.

vitabaks avatar Aug 17 '24 18:08 vitabaks