consul
consul copied to clipboard
Published
20 hours ago •
hashicorp
hashicorp
Crash on startup of consul on 1.12.x (up to latest 1.14.4)
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
Overview of the Issue
When installing consul agent on a new host using k8s auto server discovery (see config below) we get a golang crash. This happens on any version 1.12.x and above, but works on 1.11.x. If you install 1.11.x then upgrade to anything 1.12+, everything works fine, even if you leave the consul cluster.
Operating system and Environment details
OS, Architecture, and any other information you can provide about the environment.
OS: CentOS 7 Infra: GCP Virtual Machine
Config Files
config.json
{
"addresses": {
"http": "127.0.0.1"
},
"advertise_addr": "XXX",
"client_addr": "0.0.0.0",
"data_dir": "/opt/consul",
"datacenter": "us-east1-dev",
"disable_update_check": true,
"enable_script_checks": true,
"log_level": "INFO",
"node_name": "dns-us-east1-app-dev-1",
"retry_join": [
"provider=k8s kubeconfig=/var/lib/consul/.kube/config namespace=consul label_selector=\"app=consul-helm,component=server\""
],
"server": false,
"telemetry": {
"statsd_address": "127.0.0.1:8125"
},
"ui_dir": ""
}
Log Fragments
==> Starting Consul agent...
Version: '1.14.3'
Build Date: '2022-12-13 17:13:55 +0000 UTC'
Node ID: '78aae51b-ab71-6a0f-07f6-db3f96ac677e'
Node name: 'dns-us-east1-app-dev-1'
Datacenter: 'us-east1-dev' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: -1, DNS: 8600)
Cluster Addr: XXX (LAN: 8301, WAN: 8302)
Gossip Encryption: false
Auto-Encrypt-TLS: false
HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
==> Log data will now stream in as it occurs:
2023-02-03T00:00:04.662Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2023-02-03T00:00:04.662Z [WARN] agent: The 'ui_dir' field is deprecated. Use the 'ui_config.dir' field instead.
2023-02-03T00:00:04.662Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/
2023-02-03T00:00:04.669Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2023-02-03T00:00:04.669Z [WARN] agent.auto_config: The 'ui_dir' field is deprecated. Use the 'ui_config.dir' field instead.
2023-02-03T00:00:04.669Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/
2023-02-03T00:00:04.672Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: dns-us-east1-app-dev-1 XXX
2023-02-03T00:00:04.672Z [INFO] agent.router: Initializing LAN area manager
2023-02-03T00:00:04.673Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2023-02-03T00:00:04.673Z [WARN] agent.client.serf.lan: serf: Failed to re-join any previously known node
2023-02-03T00:00:04.674Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2023-02-03T00:00:04.674Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
2023-02-03T00:00:04.674Z [INFO] agent: started state syncer
2023-02-03T00:00:04.674Z [INFO] agent: Consul agent running!
2023-02-03T00:00:04.674Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2023-02-03T00:00:04.674Z [INFO] agent: Joining cluster...: cluster=LAN
2023-02-03T00:00:04.714Z [WARN] agent.router.manager: No servers available
2023-02-03T00:00:04.714Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2023-02-03T00:00:04.933Z [WARN] agent.router.manager: No servers available
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x469b9f]
goroutine 75 [running]:
runtime.throw({0x35633df?, 0xc000d398d8?})
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/panic.go:1047 +0x5d fp=0xc000d397c8 sp=0xc000d39798 pc=0x437f1d
runtime.sigpanic()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/signal_unix.go:842 +0x2c5 fp=0xc000d39818 sp=0xc000d397c8 pc=0x44eb65
aeshashbody()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1366 +0x39f fp=0xc000d39820 sp=0xc000d39818 pc=0x469b9f
runtime.mapiternext(0xc0008ceb80)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/map.go:936 +0x2eb fp=0xc000d39890 sp=0xc000d39820 pc=0x41004b
runtime.mapiterinit(0xc000d0ad00?, 0xc000d42f60?, 0x2a42751?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/map.go:863 +0x236 fp=0xc000d398b0 sp=0xc000d39890 pc=0x40fd16
reflect.mapiterinit(0x8?, 0x38?, 0xc000d39a40?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/map.go:1375 +0x19 fp=0xc000d398d8 sp=0xc000d398b0 pc=0x465d99
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
/home/runner/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:112
github.com/json-iterator/go.(*sortKeysMapEncoder).Encode(0xc000d02ba0, 0xc000cf7018, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_map.go:291 +0x236 fp=0xc000d39a40 sp=0xc000d398d8 pc=0xf44c56
github.com/json-iterator/go.(*placeholderEncoder).Encode(0xc000d39ad0?, 0xf53e8f?, 0xc000c96318?)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:332 +0x22 fp=0xc000d39a68 sp=0xc000d39a40 pc=0xf3db42
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000d02e10, 0x2a2ceca?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x56 fp=0xc000d39ae0 sp=0xc000d39a68 pc=0xf51f96
github.com/json-iterator/go.(*structEncoder).Encode(0xc000d02e70, 0x700?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x652 fp=0xc000d39bb8 sp=0xc000d39ae0 pc=0xf52892
github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc000c96300?, 0xc000cfaae0?, 0xc000d39c48?)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_optional.go:74 +0xb0 fp=0xc000d39c08 sp=0xc000d39bb8 pc=0xf49ef0
github.com/json-iterator/go.(*placeholderEncoder).Encode(0xc000d39c98?, 0xf53e8f?, 0xc000c96318?)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:332 +0x22 fp=0xc000d39c30 sp=0xc000d39c08 pc=0xf3db42
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000d42300, 0x2b33e69?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x56 fp=0xc000d39ca8 sp=0xc000d39c30 pc=0xf51f96
github.com/json-iterator/go.(*structEncoder).Encode(0xc000d423c0, 0xc0001ad580?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x652 fp=0xc000d39d80 sp=0xc000d39ca8 pc=0xf52892
github.com/json-iterator/go.(*placeholderEncoder).Encode(0xc000d39e10?, 0xf53e8f?, 0xc000c96318?)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:332 +0x22 fp=0xc000d39da8 sp=0xc000d39d80 pc=0xf3db42
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000d42630, 0x2a2ccd2?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x56 fp=0xc000d39e20 sp=0xc000d39da8 pc=0xf51f96
github.com/json-iterator/go.(*structEncoder).Encode(0xc000d42690, 0x40aad5?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x652 fp=0xc000d39ef8 sp=0xc000d39e20 pc=0xf52892
github.com/json-iterator/go.(*sliceEncoder).Encode(0xc000cf7cb0, 0xc00104c318, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_slice.go:38 +0x2cb fp=0xc000d39fa8 sp=0xc000d39ef8 pc=0xf4a98b
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000d431a0, 0x2a42453?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x56 fp=0xc000d3a020 sp=0xc000d39fa8 pc=0xf51f96
github.com/json-iterator/go.(*structEncoder).Encode(0xc000d43290, 0x0?, 0xc000c96300)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x652 fp=0xc000d3a0f8 sp=0xc000d3a020 pc=0xf52892
github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc00044dcc0?, 0x0?, 0x0?)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_optional.go:74 +0xb0 fp=0xc000d3a148 sp=0xc000d3a0f8 pc=0xf49ef0
github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc000d0ae60, 0xc00104c2c0, 0xc000ca2e40?)
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:219 +0x82 fp=0xc000d3a180 sp=0xc000d3a148 pc=0xf3d102
github.com/json-iterator/go.(*Stream).WriteVal(0xc000c96300, {0x310abc0, 0xc00104c2c0})
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:98 +0x166 fp=0xc000d3a1f0 sp=0xc000d3a180 pc=0xf3c426
github.com/json-iterator/go.(*frozenConfig).Marshal(0xc00044dcc0, {0x310abc0, 0xc00104c2c0})
/home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/config.go:299 +0xc9 fp=0xc000d3a288 sp=0xc000d3a1f0 pc=0xf336a9
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0x29df45b?, {0x4150470?, 0xc00104c2c0?}, {0x41195a0, 0xc000c7b050})
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:305 +0x6d fp=0xc000d3a320 sp=0xc000d3a288 pc=0x10978cd
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc000489450, {0x4150470, 0xc00104c2c0}, {0x41195a0, 0xc000c7b050})
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:300 +0xfc fp=0xc000d3a380 sp=0xc000d3a320 pc=0x10977fc
k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc00066e1e0, {0x4150420?, 0xc000c96240}, {0x41195a0, 0xc000c7b050})
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/versioning/versioning.go:244 +0x946 fp=0xc000d3a708 sp=0xc000d3a380 pc=0x109c866
k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc00066e1e0, {0x4150420, 0xc000c96240}, {0x41195a0, 0xc000c7b050})
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/versioning/versioning.go:184 +0x106 fp=0xc000d3a768 sp=0xc000d3a708 pc=0x109bec6
k8s.io/apimachinery/pkg/runtime.Encode({0x7f90a8a476d8, 0xc00066e1e0}, {0x4150420, 0xc000c96240})
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:50 +0x64 fp=0xc000d3a7a8 sp=0xc000d3a768 pc=0xf905c4
k8s.io/client-go/tools/clientcmd.Write(...)
/home/runner/go/pkg/mod/k8s.io/[email protected]/tools/clientcmd/loader.go:469
k8s.io/client-go/tools/clientcmd.WriteToFile({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0xc000c7a7e0}, 0xc000c7a810, 0xc000c7a840, 0xc000c7a870, {0xc000ca4ac0, ...}, ...}, ...)
/home/runner/go/pkg/mod/k8s.io/[email protected]/tools/clientcmd/loader.go:422 +0xa8 fp=0xc000d3a820 sp=0xc000d3a7a8 pc=0x28a3548
k8s.io/client-go/tools/clientcmd.ModifyConfig({0x4166c80, 0xc000c18230}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0xc000cebb90}, 0xc000cebbc0, 0xc000cebbf0, ...}, ...)
/home/runner/go/pkg/mod/k8s.io/[email protected]/tools/clientcmd/config.go:280 +0xcd8 fp=0xc000d3af48 sp=0xc000d3a820 pc=0x289f898
k8s.io/client-go/tools/clientcmd.(*persister).Persist(0xc000c839a0, 0xc000ceb110)
/home/runner/go/pkg/mod/k8s.io/[email protected]/tools/clientcmd/config.go:363 +0x11a fp=0xc000d3b038 sp=0xc000d3af48 pc=0x28a023a
k8s.io/client-go/plugin/pkg/client/auth/gcp.(*cachedTokenSource).Token(0xc000e1ccc0)
/home/runner/go/pkg/mod/k8s.io/[email protected]/plugin/pkg/client/auth/gcp/gcp.go:225 +0xe9 fp=0xc000d3b0a8 sp=0xc000d3b038 pc=0x28b9d09
golang.org/x/oauth2.(*Transport).RoundTrip(0xc000c95c80, 0xc000b7bd00)
/home/runner/go/pkg/mod/golang.org/x/[email protected]/transport.go:45 +0xd6 fp=0xc000d3b1c0 sp=0xc000d3b0a8 pc=0x15bf5d6
k8s.io/client-go/plugin/pkg/client/auth/gcp.(*conditionalTransport).RoundTrip(0xc000c95c60, 0xc000b7bd00)
/home/runner/go/pkg/mod/k8s.io/[email protected]/plugin/pkg/client/auth/gcp/gcp.go:369 +0x5b fp=0xc000d3b210 sp=0xc000d3b1c0 pc=0x28bb11b
k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc000c95ca0, 0xc000b7bc00)
/home/runner/go/pkg/mod/k8s.io/[email protected]/transport/round_trippers.go:159 +0x350 fp=0xc000d3b308 sp=0xc000d3b210 pc=0x15c2a70
net/http.send(0xc000b7bc00, {0x4134160, 0xc000c95ca0}, {0x34a02e0?, 0x1?, 0x0?})
/opt/hostedtoolcache/go/1.19.4/x64/src/net/http/client.go:251 +0x5f7 fp=0xc000d3b500 sp=0xc000d3b308 pc=0x730417
net/http.(*Client).send(0xc000c11830, 0xc000b7bc00, {0x0?, 0x40d7df?, 0x0?})
/opt/hostedtoolcache/go/1.19.4/x64/src/net/http/client.go:175 +0x9b fp=0xc000d3b578 sp=0xc000d3b500 pc=0x72fc9b
net/http.(*Client).do(0xc000c11830, 0xc000b7bc00)
/opt/hostedtoolcache/go/1.19.4/x64/src/net/http/client.go:715 +0x8fc fp=0xc000d3b768 sp=0xc000d3b578 pc=0x73201c
net/http.(*Client).Do(...)
/opt/hostedtoolcache/go/1.19.4/x64/src/net/http/client.go:581
k8s.io/client-go/rest.(*Request).request(0xc000463b20, {0x4164570, 0xc0000760a0}, 0xc000cb8930?)
/home/runner/go/pkg/mod/k8s.io/[email protected]/rest/request.go:855 +0x513 fp=0xc000d3b948 sp=0xc000d3b768 pc=0x15d7cd3
k8s.io/client-go/rest.(*Request).Do(0x20?, {0x4164570?, 0xc0000760a0?})
/home/runner/go/pkg/mod/k8s.io/[email protected]/rest/request.go:928 +0xa7 fp=0xc000d3b9e0 sp=0xc000d3b948 pc=0x15d8907
k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List(0xc000cdb200, {0x4164570, 0xc0000760a0}, {{{0x0, 0x0}, {0x0, 0x0}}, {0xc000a60661, 0x20}, {0x0, ...}, ...})
/home/runner/go/pkg/mod/k8s.io/[email protected]/kubernetes/typed/core/v1/pod.go:95 +0x176 fp=0xc000d3bac0 sp=0xc000d3b9e0 pc=0x15edf56
github.com/hashicorp/go-discover/provider/k8s.(*Provider).Addrs(0xc000c181e0?, 0x414bfb0?, 0x60f70a8?)
/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/provider/k8s/k8s_discover.go:109 +0x598 fp=0xc000d3bc88 sp=0xc000d3bac0 pc=0x28bf118
github.com/hashicorp/go-discover.(*Discover).Addrs(0xc000ca25a0, {0xc000f85880?, 0x356bcf8?}, 0x9?)
/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/discover.go:185 +0x251 fp=0xc000d3bd38 sp=0xc000d3bc88 pc=0x28936d1
github.com/hashicorp/consul/agent.retryJoinAddrs(0xc000ca25a0, {0x3562561, 0x4}, {0x35610e3, 0x3}, {0xc000b2fc80?, 0x1, 0xc000cc0ed8?}, {0x4177e60, 0xc000ca2540})
/home/runner/work/consul/consul/agent/retry_join.go:134 +0x218 fp=0xc000d3be18 sp=0xc000d3bd38 pc=0x293d3d8
github.com/hashicorp/consul/agent.(*retryJoiner).retryJoin(0xc000d3bf50)
/home/runner/work/consul/consul/agent/retry_join.go:229 +0x2d7 fp=0xc000d3bf10 sp=0xc000d3be18 pc=0x293dc77
github.com/hashicorp/consul/agent.(*Agent).retryJoinLAN(0xc000f66e00)
/home/runner/work/consul/consul/agent/retry_join.go:30 +0x1aa fp=0xc000d3bfc8 sp=0xc000d3bf10 pc=0x293c7ca
github.com/hashicorp/consul/agent.(*Agent).Start.func12()
/home/runner/work/consul/consul/agent/agent.go:765 +0x26 fp=0xc000d3bfe0 sp=0xc000d3bfc8 pc=0x28d8126
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000d3bfe8 sp=0xc000d3bfe0 pc=0x46c661
created by github.com/hashicorp/consul/agent.(*Agent).Start
/home/runner/work/consul/consul/agent/agent.go:765 +0x19b1
goroutine 1 [select]:
runtime.gopark(0xc00129f9b8?, 0x5?, 0x1?, 0x30?, 0xc00129f72e?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc00129f4a8 sp=0xc00129f488 pc=0x43ab36
runtime.selectgo(0xc00129f9b8, 0xc00129f724, 0x4?, 0x0, 0x4?, 0x1)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/select.go:328 +0x7bc fp=0xc00129f5e8 sp=0xc00129f4a8 pc=0x44ac7c
github.com/hashicorp/consul/command/agent.(*cmd).run(0xc000765600, {0xc0000721a0, 0x1, 0x1})
/home/runner/work/consul/consul/command/agent/agent.go:297 +0x1868 fp=0xc00129fca0 sp=0xc00129f5e8 pc=0x295dea8
github.com/hashicorp/consul/command/agent.(*cmd).Run(0xc000765600, {0xc0000721a0?, 0xffffffffffffffff?, 0x0?})
/home/runner/work/consul/consul/command/agent/agent.go:70 +0x27 fp=0xc00129fcf0 sp=0xc00129fca0 pc=0x295b927
github.com/mitchellh/cli.(*CLI).Run(0xc000a6ba40)
/home/runner/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:260 +0x5f8 fp=0xc00129fe20 sp=0xc00129fcf0 pc=0x57dc58
main.realMain()
/home/runner/work/consul/consul/main.go:53 +0x40b fp=0xc00129ff68 sp=0xc00129fe20 pc=0x29bc3cb
main.main()
/home/runner/work/consul/consul/main.go:23 +0x19 fp=0xc00129ff80 sp=0xc00129ff68 pc=0x29bbf99
runtime.main()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:250 +0x212 fp=0xc00129ffe0 sp=0xc00129ff80 pc=0x43a772
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00129ffe8 sp=0xc00129ffe0 pc=0x46c661
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000096fb0 sp=0xc000096f90 pc=0x43ab36
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:369
runtime.forcegchelper()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:302 +0xad fp=0xc000096fe0 sp=0xc000096fb0 pc=0x43a9cd
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x46c661
created by runtime.init.6
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:290 +0x25
goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000097790 sp=0xc000097770 pc=0x43ab36
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000977c8 sp=0xc000097790 pc=0x425237
runtime.gcenable.func1()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgc.go:178 +0x26 fp=0xc0000977e0 sp=0xc0000977c8 pc=0x419e86
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000977e8 sp=0xc0000977e0 pc=0x46c661
created by runtime.gcenable
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgc.go:178 +0x6b
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000ae000?, 0x4113c08?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000097f70 sp=0xc000097f50 pc=0x43ab36
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x60c2720)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000097fa0 sp=0xc000097f70 pc=0x423293
runtime.bgscavenge(0x0?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000097fc8 sp=0xc000097fa0 pc=0x423885
runtime.gcenable.func2()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgc.go:179 +0x26 fp=0xc000097fe0 sp=0xc000097fc8 pc=0x419e26
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000097fe8 sp=0xc000097fe0 pc=0x46c661
created by runtime.gcenable
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgc.go:179 +0xaa
goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0xc000096670?, 0x2b?, 0x73?, 0xc000096770?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000096628 sp=0xc000096608 pc=0x43ab36
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:369
runtime.runfinq()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mfinal.go:180 +0x10f fp=0xc0000967e0 sp=0xc000096628 pc=0x418f8f
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x46c661
created by runtime.createfing
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mfinal.go:157 +0x45
goroutine 10 [chan receive]:
runtime.gopark(0xc0000986d8?, 0x44099b?, 0x20?, 0x87?, 0x457c85?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0000986c8 sp=0xc0000986a8 pc=0x43ab36
runtime.chanrecv(0xc0000b3020, 0xc0000987a0, 0x1)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/chan.go:583 +0x49b fp=0xc000098758 sp=0xc0000986c8 pc=0x406d5b
runtime.chanrecv2(0x12a05f200?, 0x36ac1aa?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/chan.go:447 +0x18 fp=0xc000098780 sp=0xc000098758 pc=0x406898
k8s.io/klog.(*loggingT).flushDaemon(0xc0001b8090?)
/home/runner/go/pkg/mod/k8s.io/[email protected]/klog.go:1010 +0x6a fp=0xc0000987c8 sp=0xc000098780 pc=0xef12ea
k8s.io/klog.init.0.func1()
/home/runner/go/pkg/mod/k8s.io/[email protected]/klog.go:411 +0x26 fp=0xc0000987e0 sp=0xc0000987c8 pc=0xeef186
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x46c661
created by k8s.io/klog.init.0
/home/runner/go/pkg/mod/k8s.io/[email protected]/klog.go:411 +0xef
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x1f6d15efb2?, 0x0?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000098f50 sp=0xc000098f30 pc=0x43ab36
runtime.gcBgMarkWorker()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc000098fe0 sp=0xc000098f50 pc=0x41bfd1
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000098fe8 sp=0xc000098fe0 pc=0x46c661
created by runtime.gcBgMarkStartWorkers
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/mgc.go:1159 +0x25
goroutine 12 [select]:
runtime.gopark(0xc000093788?, 0x3?, 0x20?, 0x39?, 0xc000093772?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0000935f8 sp=0xc0000935d8 pc=0x43ab36
runtime.selectgo(0xc000093788, 0xc00009376c, 0xc0001fe180?, 0x0, 0x0?, 0x1)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/select.go:328 +0x7bc fp=0xc000093738 sp=0xc0000935f8 pc=0x44ac7c
go.opencensus.io/stats/view.(*worker).start(0xc0001fe180)
/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:276 +0xad fp=0xc0000937c8 sp=0xc000093738 pc=0x2543d2d
go.opencensus.io/stats/view.init.0.func1()
/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x26 fp=0xc0000937e0 sp=0xc0000937c8 pc=0x2542fa6
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000937e8 sp=0xc0000937e0 pc=0x46c661
created by go.opencensus.io/stats/view.init.0
/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d
goroutine 16 [chan receive]:
runtime.gopark(0x510fc5?, 0x36c7766?, 0x4?, 0x0?, 0x4113a90?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000092ef0 sp=0xc000092ed0 pc=0x43ab36
runtime.chanrecv(0xc00126c360, 0x0, 0x1)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/chan.go:583 +0x49b fp=0xc000092f80 sp=0xc000092ef0 pc=0x406d5b
runtime.chanrecv1(0x510ac5?, 0xc0000bea80?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/chan.go:442 +0x18 fp=0xc000092fa8 sp=0xc000092f80 pc=0x406858
github.com/hashicorp/consul/command.MakeShutdownCh.func1()
/home/runner/work/consul/consul/command/registry.go:282 +0x2d fp=0xc000092fe0 sp=0xc000092fa8 pc=0x29bbf0d
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x46c661
created by github.com/hashicorp/consul/command.MakeShutdownCh
/home/runner/work/consul/consul/command/registry.go:280 +0xe5
goroutine 14 [select, locked to thread]:
runtime.gopark(0xc000093fa8?, 0x2?, 0x12?, 0xad?, 0xc000093fa4?)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000093e18 sp=0xc000093df8 pc=0x43ab36
runtime.selectgo(0xc000093fa8, 0xc000093fa0, 0x0?, 0x0, 0x0?, 0x1)
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/select.go:328 +0x7bc fp=0xc000093f58 sp=0xc000093e18 pc=0x44ac7c
runtime.ensureSigM.func1()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/signal_unix.go:991 +0x1b0 fp=0xc000093fe0 sp=0xc000093f58 pc=0x44f0b0
runtime.goexit()
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x46c661
created by runtime.ensureSigM
/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/signal_unix.go:974 +0xbd
There's more to the stack trace, but character limit blocks me from posting it all, can provide more if required
Thanks for reporting this, @sharkannon . I failed to reproduce the panic with the config on a local environment with Kind cluster.
Have you observed the same panic on any non-GCP VM?
No, we don’t have any nonGCP VM’s that can contact our Kube cluster. Guess I could locally, but haven’t
Get Outlook for iOShttps://aka.ms/o0ukef
From: cskh @.> Sent: Friday, February 3, 2023 12:47:55 PM To: hashicorp/consul @.> Cc: Stephen Herd @.>; Mention @.> Subject: Re: [hashicorp/consul] Crash on startup of consul on 1.12.x (up to latest 1.14.4) (Issue #16144)
Thanks for reporting this, @sharkannonhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsharkannon&data=05%7C01%7C%7C945855a15d7c498df3fe08db0627ee22%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638110540814613858%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v7Yob4CLUQhzMRCy%2Fp9fSb4w93VzWnZgK2krS%2FEOA5U%3D&reserved=0 . I failed to reproduce the panic with the config on a local environment with Kind cluster.
Have you observed the same panic on any non-GCP VM?
— Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhashicorp%2Fconsul%2Fissues%2F16144%23issuecomment-1416386883&data=05%7C01%7C%7C945855a15d7c498df3fe08db0627ee22%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638110540814613858%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cDbzWTpKYjOHA9aTGz%2BTn8mFef%2BmrSJfegxpjPNI648%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAFL5X6GTPW7TDM7EX37M73WVVVHXANCNFSM6AAAAAAUPU6FHM&data=05%7C01%7C%7C945855a15d7c498df3fe08db0627ee22%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638110540814613858%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j9JTr34mVWTuFZTIJrCXlok9rbnAnoFnvrCY4DvYgTc%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>
Here's the TRACE log level, but nothing to much different there it seems.
==> Starting Consul agent...
Version: '1.14.3'
Build Date: '2022-12-13 17:13:55 +0000 UTC'
Node ID: '78aae51b-ab71-6a0f-07f6-db3f96ac677e'
Node name: 'dns-us-east1-app-dev-1'
Datacenter: 'us-east1-dev' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: -1, DNS: 8600)
Cluster Addr: XXX (LAN: 8301, WAN: 8302)
Gossip Encryption: false
Auto-Encrypt-TLS: false
HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
==> Log data will now stream in as it occurs:
2023-02-06T18:14:23.889Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2023-02-06T18:14:23.889Z [WARN] agent: The 'ui_dir' field is deprecated. Use the 'ui_config.dir' field instead.
2023-02-06T18:14:23.889Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/
2023-02-06T18:14:23.889Z [TRACE] agent.tlsutil: Update: version=1
2023-02-06T18:14:23.889Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=1
2023-02-06T18:14:23.889Z [TRACE] agent.tlsutil: OutgoingALPNRPCWrapper: version=1
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] Channel created
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] original dial target is: "consul://us-east1-dev.78aae51b-ab71-6a0f-07f6-db3f96ac677e/server.us-east1-dev"
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] parsed dial target is: {Scheme:consul Authority:us-east1-dev.78aae51b-ab71-6a0f-07f6-db3f96ac677e Endpoint:server.us-east1-dev URL:{Scheme:consul Opaque: User: Host:us-east1-dev.78aae51b-ab71-6a0f-07f6-db3f96ac677e Path:/server.us-east1-dev RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] Channel authority set to "server.us-east1-dev"
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] Resolver state updated: {
"Addresses": null,
"ServiceConfig": null,
"Attributes": null
} ()
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] Channel switches to new LB policy "pick_first"
2023-02-06T18:14:23.889Z [TRACE] agent: [core]pickfirstBalancer: ResolverError called with error produced zero addresses
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] Channel Connectivity change to TRANSIENT_FAILURE
2023-02-06T18:14:23.889Z [TRACE] agent: [core][Channel #1] Resolver state updated: {
"Addresses": null,
"ServiceConfig": null,
"Attributes": null
} ()
2023-02-06T18:14:23.889Z [TRACE] agent: [core]pickfirstBalancer: ResolverError called with error produced zero addresses
2023-02-06T18:14:23.896Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
2023-02-06T18:14:23.896Z [WARN] agent.auto_config: The 'ui_dir' field is deprecated. Use the 'ui_config.dir' field instead.
2023-02-06T18:14:23.896Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/
2023-02-06T18:14:23.896Z [TRACE] agent.tlsutil: Update: version=2
2023-02-06T18:14:23.896Z [TRACE] agent.tlsutil: IncomingGRPConfig: version=2
2023-02-06T18:14:23.896Z [TRACE] agent: [core][Server #2] Server created
2023-02-06T18:14:23.897Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: dns-us-east1-app-dev-1 XXX
2023-02-06T18:14:23.897Z [INFO] agent.router: Initializing LAN area manager
2023-02-06T18:14:23.897Z [TRACE] agent.tlsutil: OutgoingTLSConfigForCheck: version=2
2023-02-06T18:14:23.898Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2023-02-06T18:14:23.899Z [WARN] agent.client.serf.lan: serf: Failed to re-join any previously known node
2023-02-06T18:14:23.899Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
2023-02-06T18:14:23.899Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2023-02-06T18:14:23.899Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
2023-02-06T18:14:23.899Z [INFO] agent: started state syncer
2023-02-06T18:14:23.899Z [INFO] agent: Consul agent running!
2023-02-06T18:14:23.900Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2023-02-06T18:14:23.900Z [INFO] agent: Joining cluster...: cluster=LAN
2023-02-06T18:14:23.900Z [DEBUG] agent: discover: Using provider "k8s": cluster=LAN
2023-02-06T18:14:23.906Z [WARN] agent.router.manager: No servers available
2023-02-06T18:14:23.907Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
unexpected fault address 0x0
fatal error: fault
I've figured out the problem. Seems that the issue is related to the google-cloud-sdk-gke-gcloud-auth-plugin not being installed. in recent google changes, it's required for kube auth, and seems to cause the crash in 1.12.x+ if it's not there. May want to look into adding a try/catch to capture the fault, and throw some logging related to this.
@sharkannon , thanks for the info. We will look into this issue and improve the UX.
Looks like 1.14.4 doesn't crash on missing the auth plugin
2023-02-19T00:40:09.399-0500 [ERROR] agent: Cannot discover address: cluster=LAN
address="provider=k8s kubeconfig=/.kube/config namespace=consul label_selector=\"app=consul-helm,component=server\"" error="discover-k8s: error listing pods: Get \"https://14.132.14.116/api/v1/namespaces/consul/pods?labelSelector=app%3Dconsul-helm%2Ccomponent%3Dserver\": getting credentials: exec: exec: \"gke-gcloud-auth-plugin\": executable file not found in $PATH"
2023-02-19T00:40:09.399-0500 [WARN] agent: Join cluster failed, will retry: cluster=LAN retry_interval=30s error="No servers to join"
