netchecker: Error occurred while checking the agents. Details: unknown (get agents.network-checker.ext netchecker-agent-xxxxx)

[DevSecTim]

While using the Kubespray tool to deploy netchecker I experience the above error, here is the issue for it over in the Kubespray repo - please see it for the details.

TL;DR:

curl http://localhost:31081/api/v1/connectivity_check produces the following error:

Error occurred while checking the agents. Details: unknown (get agents.network-checker.ext netchecker-agent-xxxxx)

The netchecker-server log has this repeating:

E0910 17:15:25.308402       1 storer_k8s.go:110] unknown (get agents.network-checker.ext netchecker-agent-hostnet-2b4hm)
I0910 17:15:25.310800       1 storer_k8s.go:129] Updated agent netchecker-agent-hostnet-2b4hm unknown (put agents.network-checker.ext netchecker-agent-hostnet-2b4hm)
E0910 17:15:25.310846       1 storer_k8s.go:133] unknown (put agents.network-checker.ext netchecker-agent-hostnet-2b4hm)
[negroni] 2018-09-10T17:15:25Z | 0 | 	 5.088171ms | netchecker-service:8081 | POST /api/v1/agents/netchecker-agent-hostnet-2b4hm 
[negroni] 2018-09-10T17:15:25Z | 0 | 	 20.881µs | netchecker-service:8081 | GET /api/v1/ping 

[Demon-DK]

--- Kubespray deployment --- $ kubectl version --short Client Version: v1.11.2 Server Version: v1.11.2
image: "mirantis/k8s-netchecker-agent:v1.2.2" image: "mirantis/k8s-netchecker-server:v1.2.2"

Have the same issue. Noted also that related to both type of pings - as from internal as from hostnet pods

# kubectl logs netchecker-server-69c9f498d8-5q85b

negroni | netchecker-service:8081 | GET /api/v1/ping
negroni | netchecker-service:8081 | POST /api/v1/agents/netchecker-agent-hostnet-l94lk
E0918   | unknown (get agents.network-checker.ext netchecker-agent-hostnet-l94lk)
I0918   | Updated agent netchecker-agent-hostnet-l94lk unknown (put agents.network-checker.ext netchecker-agent-hostnet-l94lk)
E0918   | unknown (put agents.network-checker.ext netchecker-agent-hostnet-l94lk)

negroni | netchecker-service:8081 | GET /api/v1/ping
negroni | netchecker-service:8081 | POST /api/v1/agents/netchecker-agent-6872t
E0918   | unknown (get agents.network-checker.ext netchecker-agent-6872t)
I0918   | Updated agent netchecker-agent-6872t unknown (put agents.network-checker.ext netchecker-agent-6872t)
E0918   | unknown (put agents.network-checker.ext netchecker-agent-6872t)

Same time everything is clear in pod's logs

# kubectl logs netchecker-agent-xxx

I0918  Send payload via URL: http://netchecker-service:8081/api/v1/agents/netchecker-agent-6872t
I0918  Response status code: 200
I0918  Sleep for 15 second(s)
I0918  HTTP Probe (http://netchecker-service:8081/api/v1/ping):
       HTTPCode: 200;
       Total: 9223372036854 ms;
       ContentTransfer: 9223372036854 ms;
       Connect: 1 ms; DNSLookup: 1 ms;
       ServerProcessing: 1 ms;
       TCPConnection: 0 ms;

# kubectl logs netchecker-agent-hostnet-xxx

I0918  Send payload via URL: http://netchecker-service:8081/api/v1/agents/netchecker-agent-hostnet-l94lk
I0918  Response status code: 200
I0918  Sleep for 15 second(s)
I0918  HTTP Probe (http://netchecker-service:8081/api/v1/ping):
       HTTPCode: 200;
       Total: 9223372036854 ms;
       ContentTransfer: 9223372036854 ms;
       Connect: 1 ms; DNSLookup: 0 ms;
       ServerProcessing: 0 ms;
       TCPConnection: 0 ms;

Some additional observations # curl -L netchecker-service:8081/api/v1/agents/ {} *returned nothing

[AlexeyKasatkin]

Sample exchange with k8s-api server:

I0508 15:05:58.209270 1 round_trippers.go:405] GET https://10.233.0.1:443/apis/network-checker.ext/v1/namespaces/default/agents/netchecker-agent-hostnet-khfjn 403 Forbidden in 1 milliseconds

I0508 15:05:58.209334 1 round_trippers.go:411] Response Headers: I0508 15:05:58.209358 1 round_trippers.go:414] Content-Type: application/json I0508 15:05:58.209406 1 round_trippers.go:414] X-Content-Type-Options: nosniff I0508 15:05:58.209434 1 round_trippers.go:414] Content-Length: 441 I0508 15:05:58.209459 1 round_trippers.go:414] Date: Wed, 08 May 2019 15:05:59 GMT

I0508 15:05:58.209515 1 request.go:991] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"agents.network-checker.ext "netchecker-agent-hostnet-khfjn" is forbidden: User "system:serviceaccount:default:netchecker-server" cannot get resource "agents" in API group "network-checker.ext" in the namespace "default"","reason":"Forbidden","details":{"name":"netchecker-agent-hostnet-khfjn","group":"network-checker.ext","kind":"agents"},"code":403}

E0508 15:05:58.209583 1 storer_k8s.go:110] unknown (get agents.network-checker.ext netchecker-agent-hostnet-khfjn)

So, netchecker-server user does not have rights to access created resources.