Consul upgrade from 1.0.6 to 1.8.0 on a single node cluster

[JeevanAleti]

Hi, I am trying to upgrade consul from 1.0.6 to 1.8.0 using docker-compose file. After upgrading I done see the consul UI, it just shows empty UI with a word Consul

docker-compose file

consul: logging: <<: *others-logging hostname: ${HOST_NAME} image: consul:1.0.6 restart: ${RESTART_POLICY} #ports: #- "8500:8500" #- "8300-8302:8300-8302" # - 8600:8600 # - 8600:8600/udp # - 8300-8302:8300-8302 # - 8301-8302:8301-8302/udp volumes: - ${HOST_PATH}/config/localtime:/etc/localtime:ro - ${HOST_PATH}/config/timezone:/etc/timezone:ro - ${HOST_PATH}/config/consul/consul.hcl:/etc/consul/consul.hcl - consul-data:/consul/data environment: - "TZ=America/Toronto" command: "agent -config-dir=/etc/consul -client 0.0.0.0"

consul.hcl file

bind_addr = "0.0.0.0" skip_leave_on_interrupt = true node_name = "test" server = true bootstrap_expect = 1 ui = true data_dir = "/consul" ports = { #http = 8100 http = 8500 } addresses = { http = "0.0.0.0" https = "0.0.0.0" }

container logs

2022-07-22T19:13:20.099241745Z BootstrapExpect is set to 1; this is the same as Bootstrap mode. 2022-07-22T19:13:20.099268838Z bootstrap = true: do not enable unless necessary 2022-07-22T19:13:20.099306417Z ==> Starting Consul agent... 2022-07-22T19:13:20.099585407Z Version: 'v1.8.0' 2022-07-22T19:13:20.099592436Z Node ID: '0a28f8c4-bd16-42af-ba55-5b5f9a337a18' 2022-07-22T19:13:20.099596687Z Node name: 'test' 2022-07-22T19:13:20.099600909Z Datacenter: 'dc1' (Segment: '') 2022-07-22T19:13:20.099605342Z Server: true (Bootstrap: true) 2022-07-22T19:13:20.099623316Z Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) 2022-07-22T19:13:20.099627942Z Cluster Addr: 172.18.0.35 (LAN: 8301, WAN: 8302) 2022-07-22T19:13:20.099632102Z Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false 2022-07-22T19:13:20.099647584Z 2022-07-22T19:13:20.099651953Z ==> Log data will now stream in as it occurs: 2022-07-22T19:13:20.099656160Z 2022-07-22T19:13:20.109459273Z 2022-07-22T19:13:20.109Z [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0a28f8c4-bd16-42af-ba55-5b5f9a337a18 Address:172.18.0.35:8300}]" 2022-07-22T19:13:20.109599691Z 2022-07-22T19:13:20.109Z [INFO] agent.server.raft: entering follower state: follower="Node at 172.18.0.35:8300 [Follower]" leader= 2022-07-22T19:13:20.110114806Z 2022-07-22T19:13:20.110Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: test.dc1 172.18.0.35 2022-07-22T19:13:20.110686224Z 2022-07-22T19:13:20.110Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: test 172.18.0.35 2022-07-22T19:13:20.110828772Z 2022-07-22T19:13:20.110Z [INFO] agent.server: Handled event for server in area: event=member-join server=test.dc1 area=wan 2022-07-22T19:13:20.111393326Z 2022-07-22T19:13:20.111Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp 2022-07-22T19:13:20.111404084Z 2022-07-22T19:13:20.111Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp 2022-07-22T19:13:20.111765254Z 2022-07-22T19:13:20.111Z [INFO] agent.server: Adding LAN server: server="test (Addr: tcp/172.18.0.35:8300) (DC: dc1)" 2022-07-22T19:13:20.112345586Z 2022-07-22T19:13:20.112Z [INFO] agent: Started HTTP server: address=[::]:8500 network=tcp 2022-07-22T19:13:20.112418295Z 2022-07-22T19:13:20.112Z [INFO] agent: started state syncer 2022-07-22T19:13:20.112427614Z ==> Consul agent running! 2022-07-22T19:13:27.242633937Z 2022-07-22T19:13:27.242Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader" 2022-07-22T19:13:28.240006000Z 2022-07-22T19:13:28.239Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2022-07-22T19:13:28.240024086Z 2022-07-22T19:13:28.239Z [INFO] agent.server.raft: entering candidate state: node="Node at 172.18.0.35:8300 [Candidate]" term=2 2022-07-22T19:13:28.243150914Z 2022-07-22T19:13:28.243Z [INFO] agent.server.raft: election won: tally=1 2022-07-22T19:13:28.243169880Z 2022-07-22T19:13:28.243Z [INFO] agent.server.raft: entering leader state: leader="Node at 172.18.0.35:8300 [Leader]" 2022-07-22T19:13:28.243309158Z 2022-07-22T19:13:28.243Z [INFO] agent.server: cluster leadership acquired 2022-07-22T19:13:28.243427076Z 2022-07-22T19:13:28.243Z [INFO] agent.server: New leader elected: payload=test 2022-07-22T19:13:28.246336529Z 2022-07-22T19:13:28.246Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2022-07-22T19:13:28.246344852Z 2022-07-22T19:13:28.246Z [INFO] agent.leader: started routine: routine="federation state pruning" 2022-07-22T19:13:28.246349276Z 2022-07-22T19:13:28.246Z [INFO] agent.leader: started routine: routine="CA root pruning" 2022-07-22T19:13:28.246409664Z 2022-07-22T19:13:28.246Z [INFO] agent.server: member joined, marking health alive: member=test 2022-07-22T19:13:28.250119346Z 2022-07-22T19:13:28.250Z [INFO] agent.server: federation state anti-entropy synced 2022-07-22T19:13:31.213233546Z 2022-07-22T19:13:31.213Z [INFO] agent: Synced node info 2022-07-22T19:13:37.802041725Z 2022-07-22T19:13:37.801Z [ERROR] agent: Failed to check for updates: error="Get "https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=ecbe25b9-0903-72ac-9b66-4c8c60627069&version=1.8.0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

from the container:

/ # consul members Node Address Status Type Build Protocol DC Segment test 172.18.0.35:8301 alive server 1.8.0 2 dc1

the log shows the member is added but still I dont see anything in the UI

nginx container logs:

2022-07-22T15:19:07.100674547-04:00 2022/07/22 15:19:07 [error] 7#7: *16442 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.6.71, server: _, request: "GET /ui/assets/vendor-992465b8b230f89d4adce5f016543c4d.css HTTP/1.1", upstream: "http://172.18.0.92:8082/ui/assets/vendor-992465b8b230f89d4adce5f016543c4d.css", host: "test.dev.io", referrer: "https://test.dev.io/consul/ui/"

[jkirschner-hashicorp]

Hi @JeevanAleti,

It looks like this is your first post on the consul repo - welcome!

That sounds like the behavior that occurs when the UI isn't enabled. More recent versions of Consul provide a more actionable message when this occurs, per this PR: https://github.com/hashicorp/consul/pull/11820

Per the docs, setting ui = true looks correct for your version of Consul though... so I'm not sure what's wrong. (ui is replaced with ui_config.enabled starting in Consul 1.9.0)

A few follow-up questions that may help:

• Are you sure that config file with ui = true is being used/loaded?
• Does consul validate suggest any issues with the config file?
• What URL are you using to try to access the UI? Did this work on 1.0.6? (I see port 8082 in your logs, it that the right port?)

[JeevanAleti]

Thanks for your response.

1. yes the config file is the one that is being used, also as an alternative I have add the required config in docker file but still the UI doesn’t load.
2. consul validate shows configuration is valid
3. the url is configured in Nginx as https://test.dev.io/consul. This url works fine with 1.0.6 version but displays empty UI with 1.8.0

is there anything to do with docker volume that was being used for 1.0.6 version.

If there any other way to verify the working model other than from UI, thank you.

[seanamos]

Just to rule it out, did you clear your browser cache/do a hard refresh?

[JeevanAleti]

yes, I did everything but no luck. I see the below error in the logs and console Looks like the error is due to nginx proxy I have nginx reverse proxy defined as:

location = /consul { port_in_redirect off; rewrite /.* /consul/ui/# redirect; } location = /consul/ { port_in_redirect off; rewrite /.* /consul/ui/# redirect; } location ~ ^/consul/.* { root /usr/share/nginx/html; #auth_basic "Administrator Area"; #auth_basic_user_file conf/.htpasswd; # This will change "Location" and "Refresh" header fields # in the proxied server response proxy_redirect / /consul; # apply substitution in javascript files also sub_filter_types application/x-javascript; sub_filter '"/v1/' '"/consul/v1/'; sub_filter_once off; rewrite ^/consul/(.*) /$1 break; proxy_pass http://consul; }