hashicorp
Two leaders reported at the same time when using Consul service registration
Hi,
Running Vault 1.11.4 cluster with 5 nodes, using integrated storage. Also running a 5 node Consul cluster v1.13.3, on the same hosts. Each Vault cluster node has service registration configured to report to the adjacent/local Consul node.
After a network partition that affected a couple of nodes, and after recovery of that situation, two nodes are reporting as leader/active to consul. Consequently, Consul service check metrics report two leaders, and DNS responses alternate between one leader and another.
All Vault nodes, using command line, report only one leader, and everything seems fine in that regard, via status, /sys/health, /sys/leader, everything normal, there's only one active/leader node.
Restarting the Consul cluster did not resolve the situation. That's why I'm reporting as regarding to Vault. I believe the service registration routine in the fake leader insists in reporting as leader to Consul.
I'm trying to avoid restarting the fake leader (or forcing a step-down) until this behaviour is better understand.
The service registation config is as such:
"service_registration": { "consul": { "address": "<myfqdn>:8501", "scheme": "https", "service_tags": "env:prod,node:node-X", "tls_min_version": "tls12", "tls_client_ca_file": "/etc/vault/tls/tls.ca", "tls_ca_file": "/etc/vault/tls/tls.ca", "tls_cert_file": "/etc/vault/tls/tls.crt", "tls_key_file": "/etc/vault/tls/tls.key", "tls_skip_verify": "false" } }
Expected behavior Only one leader reported to Consul.
Environment: Vault 1.11.4 server/cli Consul 1.13.3 CentOS 7.9
Thanks for filing this ticket. Would you please provide us with Vault configs and also Vault/Consul logs related to reported issue? Would it be possible to also share reproduction steps?
Hi, since it's been a while since we've heard from you on this issue, I'm going to go ahead and close it. Please feel free to re-open it if you are able to provide more details as per Hamid's request. Thanks!
