Errors in agent's Runc topology probe on Openshift

[dubek]

We're running Skydive on Openshift 3.11 on IKS.

Here's the Skydive agent log:

2019-09-05T08:37:31.789Z        INFO    agent/agent.go:41 glob..func1   kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Skydive Agent _2019-09-02T05:45:19 starting...
2019-09-05T08:37:31.791Z        INFO    http/server.go:104 (*Server).Listen     kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Listening on socket 0.0.0.0:8081
2019-09-05T08:37:31.796Z        DEBUG   websocket/pool.go:102 (*Pool).AddClient kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: AddClient kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm for pool AnalyzerClientPool type : [*websocket.Pool]
2019-09-05T08:37:31.797Z        INFO    agent/probes.go:44 NewTopologyProbeBundle       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Topology probes: [ovsdb docker runc]
2019-09-05T08:37:31.797Z        INFO    probes/probes.go:70 NewFlowProbeBundle  kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Flow probes: [pcapsocket ovssflow sflow gopacket dpdk ebpf ovsmirror ovsnetflow]
2019-09-05T08:37:31.798Z        INFO    probes/probes.go:114 NewFlowProbeBundle kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Not compiled with dpdk support, skipping it
2019-09-05T08:37:31.798Z        INFO    agent/agent.go:51 glob..func1   kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Skydive Agent started
2019-09-05T08:37:31.798Z        DEBUG   netns/netns.go:306 (*ProbeHandler).start        kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: ProbeHandler initialized
2019-09-05T08:37:31.799Z        WARN    ovsdb/ovsdb.go:544 (*OvsMonitor).startMonitoring        kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Could not connect to OVSDB(dial unix /var/run/openvswitch/db.sock: connect: connection refused), will retry every 4s
2019-09-05T08:37:31.799Z        DEBUG   docker/docker.go:184 (*ProbeHandler).Do kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Connecting to Docker daemon: unix:///var/run/docker.sock
2019-09-05T08:37:31.800Z        DEBUG   runc/runc.go:366 (*ProbeHandler).initialize.func1       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Probe initialized for /var/run/containerd/runc
2019-09-05T08:37:31.800Z        INFO    websocket/client.go:507 (*Client).Connect       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Connecting to ws://172.21.140.14:8082/ws/agent/topology
2019-09-05T08:37:31.801Z        DEBUG   runc/runc.go:366 (*ProbeHandler).initialize.func1       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Probe initialized for /var/run/runc
2019-09-05T08:37:31.803Z        DEBUG   runc/runc.go:261 (*ProbeHandler).registerContainer      kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Register runc container 0068a34d4245d32ae7861ee67ab74472fb1f0421120b5a1145cc477bf642f9a6 and PID 18067
2019-09-05T08:37:31.803Z        DEBUG   netns/netns.go:135 (*ProbeHandler).Register     kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Register network namespace: /proc/18067/ns/net
2019-09-05T08:37:31.803Z        DEBUG   netns/netns.go:174 (*ProbeHandler).Register     kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Network namespace added: 3,4026532724
2019-09-05T08:37:31.803Z        DEBUG   netns/netns.go:192 (*ProbeHandler).Register     kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Registering namespace: 3,4026532724
2019-09-05T08:37:31.805Z        ERROR   runc/runc.go:346 (*ProbeHandler).initializeFolder       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: no such file or directory
2019-09-05T08:37:31.805Z        ERROR   runc/runc.go:346 (*ProbeHandler).initializeFolder       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: no such file or directory
2019-09-05T08:37:31.806Z        DEBUG   runc/runc.go:261 (*ProbeHandler).registerContainer      kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: Register runc container 0ce1bbc1db5ab0260c5df542b1fd69d54eef6458074ad957e7e891d03e226736 and PID 20213
2019-09-05T08:37:31.807Z        ERROR   runc/runc.go:346 (*ProbeHandler).initializeFolder       kube-wdc04-crac1766060dcf4354816f6b171eae7a7b-w1.cloud.ibm: no such file or directory

Note the no such file or directory errors which don't mention which file is missing; that's tricky to debug.