HA configuration with enabled persistence

[hshmilo]

Hi Victor, It is great chart! Have you tested it for the cases with the enabled persistence? I tried to configure it for HA with the enabled persistence but I faced with the following issue: I configured my cluster for one master and one slave servers. I realized that the only one PVC instance is created for all replicas (masters and slaves) that lead to conflicts in the journals and replicated data. Master logs:

2018-07-05 12:55:19,109 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2018-07-05 12:55:20,311 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /var/lib/artemis/data/bindings to /var/lib/artemis/data/bindings/oldreplica.1
2018-07-05 12:55:20,322 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /var/lib/artemis/data/journal to /var/lib/artemis/data/journal/oldreplica.2
2018-07-05 12:55:20,376 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /var/lib/artemis/data/paging to /var/lib/artemis/data/paging/oldreplica.1

Slave logs:

2018-07-05 13:11:16,209 INFO  [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2018-07-05 13:11:16,240 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /var/lib/artemis/data/bindings to /var/lib/artemis/data/bindings/oldreplica.2
2018-07-05 13:11:16,257 INFO  [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /var/lib/artemis/data/journal/oldreplica.1
2018-07-05 13:11:16,266 INFO  [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /var/lib/artemis/data/journal to /var/lib/artemis/data/journal/oldreplica.3

In the logs we may see that the slave broker deletes data of the master broker: "There were too many old replicated folders upon startup, removing /var/lib/artemis/data/journal/oldreplica.1".

The documentation says:

Data Replication When using replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network. Therefore all (persistent) data received by the live server will be duplicated to the backup. (https://activemq.apache.org/artemis/docs/2.4.0/ha.html)

Warning Once a cluster node has been configured it is common to simply copy that configuration to other nodes to produce a symmetric cluster. However, care must be taken when copying the Apache ActiveMQ Artemis files. Do not copy the Apache ActiveMQ Artemis data (i.e. the bindings, journal, and large-messages directories) from one node to another. When a node is started for the first time and initializes its journal files it also persists a sp thecial identifier to the journal directory. This id must be unique among nodes in the cluster or the cluster will not form properly. (https://activemq.apache.org/artemis/docs/2.4.0/clusters.html)

May be I miss something in the configuration? Or should I adjust the chart to create separate PVC for the master and slave nodes in the cluster? The another possible workaround is to use shared-store ha-policy.

Best regards, Hanna

[vromero]

Hi @hshmilo, Apologies for belated reply, somehow I missed the github notification.

Honestly I havent tried it, I'll give it a try, hopefully adding a test for it too.