[BUG] Update of 1 node with no replication to a few replicated nodes leads to data loss

[atsikham]

Describe the bug This bug is the result of #1497 spike. Update of 1 node with no replication to a few replicated nodes leads to data loss. Also no rabbitmqctl forget_cluster_node <node> executed during downscale, so it leads to some issues, such of removed node disk presence in cluster information.

To Reproduce Steps to reproduce the behavior:

1. deploy 1 rabbitmq node with disabled clustering (default configuration)
2. create some entities: users/queues/vhosts (see More information part with useful commands)
3. extend config file with a part from Config files
4. apply configuration with 3 instances (count: 3)
5. check created data, it's been lost
6. downscale to 2 instances, data is in place

Expected behavior

• rabbitmqctl forget_cluster_node <node> should be executed during downscale to be able avoid some results of #1574
• Nothing should be changed in the cluster when applying the same configuration.
• Only specified changes should be added to the cluster when applying new configuration, upscale and downscale.
• Documentation about scaling process should be updated.

Config files

---
kind: configuration/rabbitmq
title: "RabbitMQ"
name: default
provider: azure
specification:
  cluster:
    is_clustered: true

Cloud Environment (please complete the following information): Azure, but should also be tested with other providers.

Additional context Found in version 0.7.1.

More information Verification commands

# check cluster status
rabbitmq-diagnostics cluster_status

# create user
rabbitmqctl add_user testuser testpassword
rabbitmqctl set_user_tags testuser administrator
rabbitmqctl set_permissions -p / testuser ".*" ".*" ".*"

# create vhost
rabbitmqctl add_vhost Some_Virtual_Host
rabbitmqctl set_permissions -p Some_Virtual_Host guest ".*" ".*" ".*"

# create queue
rabbitmq-plugins enable rabbitmq_management
wget http://127.0.0.1:15672/cli/rabbitmqadmin
chmod +x rabbitmqadmin
./rabbitmqadmin declare queue --vhost=Some_Virtual_Host name=some_outgoing_queue durable=true

# create exchange
./rabbitmqadmin declare exchange --vhost=Some_Virtual_Host name=some_exchange type=direct

# create binding
./rabbitmqadmin --vhost=Some_Virtual_Host declare binding source=some_exchange destination_type=queue destination=some_outgoing_queue routing_key=some_routing_key

# check queues and vhosts count
rabbitmqctl status

# list
rabbitmqctl list_bindings --vhost "Some_Virtual_Host"
rabbitmqctl list_vhosts
rabbitmqctl list_queues --vhost "Some_Virtual_Host"

---

DoD checklist

• Changelog
  • [ ] updated
  • [ ] not needed
• COMPONENTS.md
  • [ ] updated
  • [ ] not needed
• Schema
  • [ ] updated
  • [ ] not needed
• Backport tasks
  • [ ] created
  • [ ] not needed
• Documentation
  • [ ] added
  • [ ] updated
  • [ ] not needed
• [ ] Feature has automated tests
• [ ] Automated tests passed (QA pipelines)
  • [ ] apply
  • [ ] upgrade
  • [ ] backup/restore
• [ ] Idempotency tested
• [ ] All conversations in PR resolved