dkron icon indicating copy to clipboard operation
dkron copied to clipboard

Failed to access the dashboard of leader node in a three-node cluster on a single machine

Open rukolahasser opened this issue 4 years ago • 1 comments

Describe the bug I built a three-node cluster on a single machine, but constantly failed to access the access the dashboard of leader node/

To Reproduce Steps to reproduce the behavior:

  1. Bootstrap three nodes on the same machine with the following command:
node1:
dkron agent --node-name=node1 --server --http-addr=:6080 --bootstrap-expect=3

node2:
dkron agent --server --node-name=node2 --bind-addr=0.0.0.0:8947 --rpc-port=6869 --http-addr=:6081 --bootstrap-expect=3 --join=127.0.0.1:8946

node3:
dkron agent --server --node-name=node3 --bind-addr=0.0.0.0:8948 --rpc-port=6870 --http-addr=:6082 --bootstrap-expect=3 --join=127.0.0.1:8946
  1. If the leader is node2, then accessing node2 at port 6081 will throw the exception, but node1 and node3 are fine.
  2. Exception:
2020/07/03 15:28:56 [Recovery] 2020/07/03 - 15:28:56 panic recovered:
GET /dashboard/ HTTP/1.1
Host: 192.168.14.127:6081
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en,zh-CN;q=0.9,zh;q=0.8
Connection: keep-alive
Cookie: _ga=GA1.1.1116449916.1591763526; Hm_lvt_17a3a88cbe9f9c8808943e8ed1c7155a=1591763526; Hm_lpvt_17a3a88cbe9f9c8808943e8ed1c7155a=1591763526
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36


runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:212 (0x449a29)
/usr/local/go/src/runtime/signal_unix.go:695 (0x449878)
/go/pkg/mod/github.com/hashicorp/[email protected]/api.go:663 (0xa7a05b)
/dkron/dkron/agent.go:542 (0x1f03fb6)
/dkron/dkron/dashboard.go:35 (0x1f0d6c2)
/dkron/dkron/dashboard.go:77 (0x1f0e0c8)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:156 (0xbec12a)
/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83 (0xbff97f)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:156 (0xbec12a)
/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:241 (0xbfeab0)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:156 (0xbec12a)
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:409 (0xbf6015)
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:367 (0xbf56ed)
/usr/local/go/src/net/http/server.go:2807 (0x7096c2)
/usr/local/go/src/net/http/server.go:1895 (0x70503b)
/usr/local/go/src/runtime/asm_amd64.s:1373 (0x4635a0)

Expected behavior Expected to access dashboard of any node.

Screenshots Screen Shot 2020-07-03 at 15 53 06

** Specifications:**

  • OS: [Ubuntu 12.04.5]
  • Version [Dkron 2.2.2]

Additional context Add any other context about the problem here.

rukolahasser avatar Jul 03 '20 07:07 rukolahasser

This is due to the nodes being competing for the same local db file. I'm not sure if this is a regression, but you can workaround by specifying a different --data-dir for each node --data-dir=node2 for example.

vcastellm avatar Jul 23 '20 15:07 vcastellm