Issue with Upgrading Zabbix from 6.0.2 to 6.4: Database Upgrade Failing in HA Mode

[alemsas]

I am currently running Zabbix 6.0.2 in a Docker environment and attempting to upgrade to Zabbix 6.4. My setup includes Zabbix server and agent containers, with the Zabbix server using MySQL (Percona XtraDB Cluster). I have followed the upgrade instructions, but the upgrade process fails because the Zabbix server still attempts to operate in HA mode during the database upgrade.

Current Setup:

Database: Percona XtraDB Cluster

My configuration:

cat docker-compose.yml
version: "3.9"

services:
  zabbix-server-1:
    image: 'zabbix/zabbix-server-mysql:ubuntu-6.4-latest'
#    ports:
#      - "10051:10051"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/lib/snmp/mibs:/var/lib/zabbix/mibs:ro
      - './usr/lib/zabbix/alertscripts:/usr/lib/zabbix/alertscripts:ro'
      - './usr/lib/zabbix/externalscripts:/usr/lib/zabbix/externalscripts:ro'
    environment:
      - DB_SERVER_HOST=172.31.63.1
      - DB_SERVER_PORT=3306
      - ZBX_HANODENAME=zabbix-server-1
      - ZBX_NODEADDRESS=172.31.63.1
    env_file:
      - ./envs/.env_mysql
      - ./envs/.env_server
    container_name: zabbix-server-1
    hostname: zbxSRV-1
    restart: unless-stopped
    network_mode: host
    ulimits:
      nproc: 65535
      nofile:
        soft: 20000
        hard: 40000
    stop_grace_period: 30s
  zabbix-agent-1:
    image: 'zabbix/zabbix-agent:ubuntu-latest'
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - './zabbix-server/zabbix_agentd.d:/etc/zabbix/zabbix_agentd.d:ro'
      - './zabbix-server/var/lib/zabbix:/var/lib/zabbix:ro'
    env_file:
     - ./envs/.env_agent
    environment:
      - ZBX_SERVER_HOST=127.0.0.1,172.31.63.1
      - ZBX_ACTIVE_ALLOW=false
    privileged: true
    container_name: zabbix-agent-1
    hostname: zbxAgent-1
    restart: unless-stopped
    pid: "host"
    network_mode: host
    stop_grace_period: 5s

also here is my docker logs for zabbix-server

docker logs -f --tail=30 zabbix-server-1
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherPSK13": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSKeyFile": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKIdentity": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKFile": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "ServiceManagerSyncFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "HANodeName": 'zabbix-server-1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "NodeAddress": '172.31.63.1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "User": 'zabbix'...updated
Starting Zabbix Server. Zabbix 6.4.14 (revision 0a50e61).
Press Ctrl+C to exit.

     7:20240516:232828.012 Starting Zabbix Server. Zabbix 6.4.14 (revision 0a50e61).
     7:20240516:232828.012 ****** Enabled features ******
     7:20240516:232828.012 SNMP monitoring:           YES
     7:20240516:232828.012 IPMI monitoring:           YES
     7:20240516:232828.012 Web monitoring:            YES
     7:20240516:232828.012 VMware monitoring:         YES
     7:20240516:232828.012 SMTP authentication:       YES
     7:20240516:232828.012 ODBC:                      YES
     7:20240516:232828.012 SSH support:               YES
     7:20240516:232828.012 IPv6 support:              YES
     7:20240516:232828.012 TLS support:               YES
     7:20240516:232828.012 ******************************
     7:20240516:232828.012 using configuration file: /etc/zabbix/zabbix_server.conf
     7:20240516:232828.044 Zabbix supports only "utf8_bin,utf8mb3_bin,utf8mb4_bin" collation(s). Database "zabbix" has default collation "utf8mb4_0900_ai_ci"
     7:20240516:232828.080 current database version (mandatory/optional): 06000000/06000020
     7:20240516:232828.080 required mandatory version: 06040000
     7:20240516:232828.080 mandatory patches were found
     7:20240516:232828.082 cannot perform database upgrade in HA mode: all nodes need to be stopped and Zabbix server started in standalone mode for the time of upgrade.
     7:20240516:232828.082 Zabbix Server stopped. Zabbix 6.4.14 (revision 0a50e61).

Steps Taken:

1. ​Dynamically in MySQL:SET GLOBAL log_bin_trust_function_creators = 1;
2. Updated Docker Image in docker-compose.yml:
3. Cleared the ha_node Table:​ ​4.Set Environment Variable to Disable HA Mode with remove below into my compose!

- ZBX_HANODENAME=zabbix-server-1
- ZBX_NODEADDRESS=172.31.63.1

Request for Help: What additional steps or configurations are necessary to ensure the Zabbix server starts in standalone mode to complete the database upgrade? Are there any known issues or additional configurations required when upgrading from 6.0.2 to 6.4 that I might have overlooked?​