Missing pod view all of a sudden

[paulmorabito]

Hi,

I'm using Weave Scope 1.10.2 on Kubernetes 1.13.5. This is on a bare metal cluster running Debian.

Previously, when accessing Weave Scope I would see a Pod filter option alongside Processes, Containers and Hosts at the top of the Web UI. This has now suddenly vanished.

If I log into the Kubernetes Dashboard or access via kubectl, I can see my pods by namespace but it is only not displaying within Weave Scope.

I don't recall doing anything to the UI view and nothing in the cluster has changed recently.

Any ideas how this can be restored?

[qiell]

Can you please share the logs of weave-scope-agent and weave-scope-app pods?

[paulmorabito]

I am using kubetail to get the logs and only this line appears in weave-scope-agent:

Error from server: Get https://192.168.1.113:10250/containerLogs/weave/weave-scope-agent-cpxq9/scope-agent?follow=true&sinceSeconds=10: dial tcp 192.168.1.113:10250: connect: no route to host

And this is because one of my nodes is down at the moment. It's a manager node but the other two are up and running.

There is nothing appearing in the logs for weave-scope-app

Is there any way I can grab the logs or query for them?

On Tue, 30 Apr 2019 at 22:30, Akash Srivastava [email protected] wrote:

Can you please share the logs of weave-scope-agent and weave-scope-app pods?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/weaveworks/scope/issues/3597#issuecomment-487952919, or mute the thread https://github.com/notifications/unsubscribe-auth/ADSGX3YSSE3DRINP7BZC7QLPTBCYLANCNFSM4HJLT2BQ .

[paulmorabito]

This the logs for one of the weave-scope-agent pods.

<probe> INFO: 2019/04/29 00:23:01.741729 Pipe pipe-3566376904579192063 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:23:04.464767 Pipe pipe-3566376904579192063 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/29 00:23:12.114149 Pipe pipe-2906881728982743550 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:23:13.505390 Pipe pipe-2906881728982743550 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/29 00:25:49.424499 Pipe pipe-5906913349546651901 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:25:58.973341 Pipe pipe-5906913349546651901 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/29 00:28:08.475549 Restarting container b105500426cf0871771e95548b014069b4e85752f3136b194cb8c9cbb76613fa
<probe> INFO: 2019/04/29 00:28:30.675054 Pipe pipe-6743805560493990469 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:28:43.704271 Pipe pipe-6743805560493990469 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/29 00:52:51.510626 Pipe pipe-4120662372944844622 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:52:59.085255 Pipe pipe-4120662372944844622 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/29 00:53:18.158542 Pipe pipe-1204611376668093057 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:54:57.386998 Pipe pipe-1204611376668093057 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> ERRO: 2019/04/29 00:54:57.387241 Error waiting on attachment to container 8941079550c5496be8d94044684540e525a494d041b13c779943d06ee7f48c83: io: read/write on closed pipe
<probe> INFO: 2019/04/29 00:55:26.908806 Pipe pipe-7678798738439803381 connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/29 00:55:32.980394 Pipe pipe-7678798738439803381 connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> WARN: 2019/04/30 00:00:12.870448 Dropping report to weave-scope-app.weave.svc.cluster.local
<probe> ERRO: 2019/04/30 01:55:22.217213 Error doing controls for weave-scope-app.weave.svc.cluster.local, backing off 1s: dial tcp 10.233.14.101:80: connect: connection refused
<probe> ERRO: 2019/04/30 01:55:22.335321 Error doing publish for weave-scope-app.weave.svc.cluster.local, backing off 1s: Post http://10.233.14.101:80/api/report: dial tcp 10.233.14.101:80: connect: connection refused
<probe> INFO: 2019/04/30 01:55:28.077231 Control connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2019/04/30 01:55:28.077251 Publish loop for weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/30 01:55:28.077741 Control connection to weave-scope-app.weave.svc.cluster.local exiting
<probe> INFO: 2019/04/30 01:55:30.621391 Publish loop for weave-scope-app.weave.svc.cluster.local starting
<probe> WARN: 2019/04/30 12:34:27.585560 background /proc reader: full pass took 8m29.165762021s: 50% more than expected (10s)
<probe> WARN: 2019/04/30 12:46:20.788750 background /proc reader: full pass took 7m41.763712891s: 50% more than expected (10s)
<probe> WARN: 2019/05/01 00:00:04.386374 Dropping report to weave-scope-app.weave.svc.cluster.local
<probe> WARN: 2019/05/01 00:00:04.416681 Dropping report to weave-scope-app.weave.svc.cluster.local

weave-scope-app:

<app> INFO: 2019/04/30 01:55:23.128577 app starting, version 1.10.2, ID 440dfbc307bf246f
<app> INFO: 2019/04/30 01:55:23.131489 command line args: --mode=app
<app> INFO: 2019/04/30 01:55:23.133859 Basic authentication disabled
<app> INFO: 2019/04/30 01:55:23.133958 listening on :4040
<app> WARN: 2019/04/30 01:55:23.145326 Error updating weaveDNS, backing off 20s: Error running weave ps: exit status 1: "Link not found\n". If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<app> WARN: 2019/04/30 01:55:43.157167 Error updating weaveDNS, backing off 40s: Error running weave ps: exit status 1: "Link not found\n". If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<app> ERRO: 2019/04/30 12:10:00.870552 err:read tcp 10.233.64.4:4040->10.233.64.1:34676: i/o timeout
<app> ERRO: 2019/04/30 12:18:43.495420 err:read tcp 10.233.64.4:4040->10.233.64.1:39298: use of closed network connection
<app> ERRO: 2019/04/30 12:18:43.495477 cannot serialize topology diff: write tcp 10.233.64.4:4040->10.233.64.1:39298: i/o timeout
<app> ERRO: 2019/04/30 13:13:25.251548 cannot serialize topology diff: websocket: close sent
<app> ERRO: 2019/04/30 13:17:24.150946 cannot serialize topology diff: websocket: close sent
<app> ERRO: 2019/04/30 13:39:53.229548 err:read tcp 10.233.64.4:4040->10.233.64.1:32790: use of closed network connection
<app> ERRO: 2019/04/30 13:39:53.229613 cannot serialize topology diff: write tcp 10.233.64.4:4040->10.233.64.1:32790: i/o timeou

As I said in my previous comment, one of my nodes is currently down. I'm trying to restore it but I have two others running fine and everything else in the cluster is fine.

[paulmorabito]

I've now fixed the node that was down but the node is constantly in "NotReady" status and the above issue is also not fixed.

The weave scope agent reports:

Error from server: Get https://192.168.1.113:10250/containerLogs/weave/weave-scope-agent-cpxq9/scope-agent: dial tcp 192.168.1.113:10250: connect: connection refused

[paulmorabito]

Node is now online fully. Same missing pod view remains.

[paulmorabito]

logs for weave scope agent on previously failing node are:

time="2019-05-02T13:24:05Z" level=info msg="publishing to: weave-scope-app.weave.svc.cluster.local:80"
<probe> INFO: 2019/05/02 13:24:05.931548 Basic authentication disabled
<probe> INFO: 2019/05/02 13:24:16.056542 command line args: --mode=probe --probe-only=true --probe.docker=true --probe.docker.bridge=docker0 --probe.kubernetes.role=host weave-scope-app.weave.svc.cluster.local:80
<probe> INFO: 2019/05/02 13:24:16.057107 probe starting, version 1.10.2, ID 5843f87aed5ecc83
<probe> WARN: 2019/05/02 13:24:16.112354 Cannot resolve 'scope.weave.local.': dial tcp 172.17.0.1:53: connect: connection refused
<probe> INFO: 2019/05/02 13:24:16.251475 Control connection to weave-scope-app.weave.svc.cluster.local starting
<probe> WARN: 2019/05/02 13:24:16.343102 Error setting up the eBPF tracker, falling back to proc scanning: error while loading "kprobe/tcp_v4_connect" (invalid argument):

<probe> INFO: 2019/05/02 13:24:16.388575 Success collecting weave status
<probe> INFO: 2019/05/02 13:24:16.426414 Publish loop for weave-scope-app.weave.svc.cluster.local starting

[bboreham]

This line seems the most indicative of a problem:

<probe> WARN: 2019/04/30 12:34:27.585560 background /proc reader: full pass took 8m29.165762021s: 50% more than expected (10s)

the question is, why is it taking 8 minutes to scan /proc ? Either there is a huge amount of info to scan, or the probe is very constrained.

[paulmorabito]

I've re-checked the logs and that error only occurred in those two instances on 04/30 on the one node and never since across any nodes in the cluster. The node is also healthy, or at least seems to be. The kubernetes dashboard can display pods fine.

Is there something I can do to restore the pod view? Because of this one error, does it mean the pod view is no longer ever available?

[vaibhavjain882]

Guys any luck here? I am into the same issue from the past 1 year, tired of restarting cluster agent every time to make it work.

[george-mcintyre]

Same here! Have to restart weave-scope-cluster-agent to make it work. Using weave-scope 1.12.0

[raydeo]

OK, here is how I solve this issue - which has been a real pain for me previously. I found running the weavescope command in a script with port forwarding offered more stability and has always kept working for me ever since doing it like this, on numerous servers.

Remember to set IP ADDRESS

`` DASHBOARDS Weave Dash SCRIPT root@kubernetes-4vcpu-8gb:~/services# cat weavedash.sh

#!/bin/bash kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')" sleep 3 kubectl port-forward --address 0.0.0.0 svc/weave-scope-app 8081:80 -n weave & http:// IP ADDRESSS :8081

COMMANDS kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')" kubectl port-forward --address 0.0.0.0 svc/weave-scope-app 8081:80 -n weave & http:// IP ADDRESSS :8081

``

I hope it works for you too, let me know.

I have not been using WeaveScope for a while now ...but can spin up a server and check it again if you are still stuck.