containers icon indicating copy to clipboard operation
containers copied to clipboard

Published 20 hours ago • verified publisher icon solarwinds

Can't see container CPU/Memory usage.

Open shabbirkagalwala opened this issue 7 years ago • 8 comments

I started the service for swi-ui and containers for the agent using docker run on each node of the swarm. I was also able to add the add nodes using the web-ui but now for some reason, the container stats i.e CPU and Memory just show 0.00% and null values. (see screenshot)

My service create command: docker service create \ --name swi-ui \ --replicas 1 \ --network webhelpdesk --network proxy \ --label com.df.notify=true --label com.df.distribute=true \ --label com.df.servicePath=/ --label com.df.port=80 \ solarwinds/container-ui

Docker run command: docker run -d --name swi-agent --privileged --net=host --restart always -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro solarwinds/container-agent

capture1

Port 9090 is open and I can see ./agent using this port via the command netstat -tuplen.

Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 12060078 15412/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 17088 2406/master tcp6 0 0 :::9090 :::* LISTEN 0 237464630 6037/./agent tcp6 0 0 :::9000 :::* LISTEN 0 219767543 21355/dockerd tcp6 0 0 :::4040 :::* LISTEN 0 29844626 17522/scope-app tcp6 0 0 :::7946 :::* LISTEN 0 29240890 21355/dockerd tcp6 0 0 :::80 :::* LISTEN 0 114601075 21355/dockerd tcp6 0 0 :::8081 :::* LISTEN 0 196936481 11774/docker-proxy tcp6 0 0 :::8082 :::* LISTEN 0 79188916 28217/docker-proxy tcp6 0 0 :::22 :::* LISTEN 0 12060080 15412/sshd tcp6 0 0 :::8089 :::* LISTEN 0 103626226 2435/docker-proxy tcp6 0 0 ::1:25 :::* LISTEN 0 17089 2406/master tcp6 0 0 :::443 :::* LISTEN 0 114602087 21355/dockerd tcp6 0 0 :::8443 :::* LISTEN 0 79188905 28205/docker-proxy udp 0 0 0.0.0.0:4789 0.0.0.0:* 0 29240926 - udp6 0 0 :::7946 :::* 0 29240891 21355/dockerd

Are there any additional ports that are being used to display the container stats? If not, what may be causing the issue here?

shabbirkagalwala avatar Jun 30 '17 21:06 shabbirkagalwala

What version of docker are you running?

Can you take a post logs from the agent and the ui containers. to get the logs you can run

docker logs swi-ui

docker logs swi-agent

derhally avatar Jun 30 '17 22:06 derhally

This is the log for the agent:

[root@ip-192-168-4-4 centos]# docker logs f32 starting agent network mode[host] container: f32ddf3afb81292fbdaf0a6f5491c93692c158dd675870521cebee361cfcc34d pid: 14537 inventory - NetworkMode is host skipping container f32ddf3afb81292fbdaf0a6f5491c93692c158dd675870521cebee361cfcc34d network mode[default] container: d7be3eca828ec06cfb3f065e197ef7097b886f93289f026235b0b10b1d439648 pid: 3739 network mode[default] container: bdc64876969c184c3c8bffa7f9073522665685c76ebdff468a7c64b17faf4d07 pid: 2380 network mode[default] container: f0a44f06dcdd5a69c4a232193954a21d82caba418c18d1617e59e3c0aebac065 pid: 2408 2017/07/01 02:37:21 serving on port 9090 2017/07/01 02:37:34 Client connecting from '192.168.0.108:56082' message received [{"type":"subscription","modules":["api.Pcap", "api.Iperf", "api.Inventory", "api.StatsSummary", "api.Event", "api.PingResult"]}]

2017/07/01 02:19:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:19:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:19:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:19:57 192.168.3.78 ping 1ms 2017/07/01 02:19:57 192.168.4.4 ping 1ms 2017/07/01 02:19:57 192.168.3.131 ping 1ms 2017/07/01 02:20:27 Running ping against 3 hosts 2017/07/01 02:20:27 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:20:27 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:20:27 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:20:27 192.168.3.78 ping 0ms 2017/07/01 02:20:27 192.168.3.131 ping 1ms 2017/07/01 02:20:27 192.168.4.4 ping 2ms 2017/07/01 02:20:57 Running ping against 3 hosts 2017/07/01 02:20:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:20:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:20:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:20:57 192.168.3.131 ping 1ms 2017/07/01 02:20:57 192.168.4.4 ping 1ms 2017/07/01 02:20:57 192.168.3.78 ping 4ms 2017/07/01 02:21:27 Running ping against 3 hosts 2017/07/01 02:21:27 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:21:27 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:21:27 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:21:27 192.168.4.4 ping 2ms 2017/07/01 02:21:27 192.168.3.131 ping 2ms 2017/07/01 02:21:27 192.168.3.78 ping 3ms 2017/07/01 02:21:57 Running ping against 3 hosts 2017/07/01 02:21:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:21:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:21:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:21:57 192.168.3.78 ping 0ms 2017/07/01 02:21:57 192.168.4.4 ping 1ms 2017/07/01 02:21:57 192.168.3.131 ping 1ms 2017/07/01 02:22:27 Running ping against 3 hosts 2017/07/01 02:22:27 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:22:27 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:22:27 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:22:27 192.168.3.131 ping 1ms 2017/07/01 02:22:27 192.168.3.78 ping 1ms 2017/07/01 02:22:27 192.168.4.4 ping 1ms 2017/07/01 02:22:57 Running ping against 3 hosts 2017/07/01 02:22:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:22:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:22:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:22:57 192.168.3.131 ping 1ms 2017/07/01 02:22:57 192.168.4.4 ping 1ms 2017/07/01 02:22:57 192.168.3.78 ping 4ms 2017/07/01 02:23:27 Running ping against 3 hosts 2017/07/01 02:23:27 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:23:27 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:23:27 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:23:27 192.168.4.4 ping 3ms 2017/07/01 02:23:27 192.168.3.131 ping 4ms 2017/07/01 02:23:27 192.168.3.78 ping 4ms 2017/07/01 02:23:57 Running ping against 3 hosts 2017/07/01 02:23:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:23:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:23:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:23:57 192.168.3.131 ping 0ms 2017/07/01 02:23:57 192.168.3.78 ping 1ms 2017/07/01 02:23:57 192.168.4.4 ping 1ms 2017/07/01 02:24:27 Running ping against 3 hosts 2017/07/01 02:24:27 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:24:27 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:24:27 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:24:27 192.168.3.78 ping 1ms 2017/07/01 02:24:27 192.168.3.131 ping 1ms 2017/07/01 02:24:27 192.168.4.4 ping 2ms 2017/07/01 02:24:57 Running ping against 3 hosts 2017/07/01 02:24:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:24:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:24:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:24:57 192.168.3.131 ping 1ms 2017/07/01 02:24:57 192.168.3.78 ping 1ms 2017/07/01 02:24:57 192.168.4.4 ping 1ms 2017/07/01 02:25:27 Running ping against 3 hosts 2017/07/01 02:25:27 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:25:27 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:25:27 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:25:27 192.168.4.4 ping 3ms 2017/07/01 02:25:27 192.168.3.131 ping 4ms 2017/07/01 02:25:27 192.168.3.78 ping 5ms 2017/07/01 02:25:57 Running ping against 3 hosts 2017/07/01 02:25:57 Sending request to http://192.168.4.4:9090/ping 2017/07/01 02:25:57 Sending request to http://192.168.3.78:9090/ping 2017/07/01 02:25:57 Sending request to http://192.168.3.131:9090/ping 2017/07/01 02:25:57 192.168.3.131 ping 0ms 2017/07/01 02:25:57 192.168.3.78 ping 1ms 2017/07/01 02:25:57 192.168.4.4 ping 1ms

For the web ui:

[root@ip-192-168-0-108 centos]# docker logs e6d Hosting environment: Production Content root path: /app Now listening on: http://+:80 Application started. Press Ctrl+C to shut down. info: CMonService.Services.AgentConnectionManager[0] Added listener for host 'ip-192-168-0-108.us-west-2.compute.internal' info: CMonService.Services.AgentConnectionManager[0] Added listener for host 'ip-192-168-3-131.us-west-2.compute.internal' info: CMonService.Services.AgentConnectionManager[0] Added listener for host 'ip-192-168-3-78.us-west-2.compute.internal' info: CMonService.Services.AgentConnectionManager[0] Added listener for host 'ip-192-168-4-4.us-west-2.compute.internal' fail: CMonService.Agent.AgentEventListener[0] Error reading from socket for 'ws://192.168.4.4:9090/ws/stream'. System.Net.WebSockets.WebSocketException: The 'System.Net.WebSockets.InternalClientWebSocket' instance cannot be used for communication because it has been transitioned into the 'Aborted' state. ---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'ClientWebSocket'. at System.Net.WebSockets.ManagedWebSocket.<EnsureBufferContainsAsync>d__69.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.ManagedWebSocket.<ReceiveAsyncPrivate>d__60.MoveNext() --- End of inner exception stack trace --- at System.Net.WebSockets.ManagedWebSocket.<ReceiveAsyncPrivate>d__60.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CMonService.Agent.AgentEventListener.<GetData>d__12.MoveNext() in D:\source\cmontestui\src\CMonService\Agent\AgentEventListener.cs:line 100 fail: CMonService.Services.AgentRegistry[0] Error adding agent on host '192.168.4.4:9090' System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CMonService.Agent.AgentClient.<GetEnvironmentInfoAsync>d__3.MoveNext() in D:\source\cmontestui\src\CMonService\Agent\AgentClient.cs:line 32 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CMonService.Services.AgentRegistry.<AddAgentAsync>d__5.MoveNext() in D:\source\cmontestui\src\CMonService\Services\AgentRegistry.cs:line 46 info: CMonService.Services.AgentConnectionManager[0] Added listener for host 'ip-192-168-4-4.us-west-2.compute.internal' fail: CMonService.Agent.AgentEventListener[0] Error reading from socket for 'ws://192.168.4.4:9090/ws/stream'. System.Net.WebSockets.WebSocketException: The 'System.Net.WebSockets.InternalClientWebSocket' instance cannot be used for communication because it has been transitioned into the 'Aborted' state. ---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'AsyncEventArgsNetworkStream'. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.ManagedWebSocket.<EnsureBufferContainsAsync>d__69.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.ManagedWebSocket.<ReceiveAsyncPrivate>d__60.MoveNext() --- End of inner exception stack trace --- at System.Net.WebSockets.ManagedWebSocket.<ReceiveAsyncPrivate>d__60.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CMonService.Agent.AgentEventListener.<GetData>d__12.MoveNext() in D:\source\cmontestui\src\CMonService\Agent\AgentEventListener.cs:line 100 info: CMonService.Services.AgentConnectionManager[0] Added listener for host 'ip-192-168-4-4.us-west-2.compute.internal'

Docker Version

Client: Version: 17.03.1-ce API version: 1.27 Go version: go1.7.5 Git commit: c6d412e Built: Mon Mar 27 17:05:44 2017 OS/Arch: linux/amd64

Server: Version: 17.03.1-ce API version: 1.27 (minimum version 1.12) Go version: go1.7.5 Git commit: c6d412e Built: Mon Mar 27 17:05:44 2017 OS/Arch: linux/amd64 Experimental: false

OS Version

CentOS Linux release 7.3.1611 (Core)

shabbirkagalwala avatar Jul 01 '17 02:07 shabbirkagalwala

Looks like the UI container's connection to the agents is failing for some reason. Does the issue resolve itself if you stop and restart the UI container?

On Fri, Jun 30, 2017 at 10:43 PM, Shabbir Kagalwala < [email protected]> wrote:

Reopened #3 https://github.com/solarwinds/containers/issues/3.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/solarwinds/containers/issues/3#event-1146826928, or mute the thread https://github.com/notifications/unsubscribe-auth/AMcvTMe-iflk04wY3Gbla5M_eEIIYgp1ks5sJbJsgaJpZM4OLDpP .

derhally avatar Jul 03 '17 15:07 derhally

I am using a service for the web ui, i did try removing the service, adding it again, adding the nodes/hosts again but still couldn't see the CPU and memory usage. It is strange as I am able to add nodes/hosts, see the containers running on the nodes, can even see the topology (without any connections) but can't see the CPU and memory usage.

Are there any additional ports that the web ui uses to communicate with the agent containers except for 9090?

shabbirkagalwala avatar Jul 05 '17 14:07 shabbirkagalwala

Port 9090 is the only port that needs to be open between the UI service and the agents. The log shows that the websocket is being closed right after it is established. The UI tries to maintain a persistent websocket connection with the agent in order to receive the stats.

derhally avatar Jul 05 '17 21:07 derhally

So, what could be causing this issue? I have 9090 open in the security groups and can also add nodes.

Is it because I am running the web-ui as a service?

Even when the UI and Agent are on the same host, i cant see the stats for the containers running on the same host.

Thank you for the help so far, I appreciate it!

On Jul 5, 2017, at 4:55 PM, derhally [email protected] wrote:

Port 9090 is the only port that needs to be open between the UI service and the agents. The log shows that the websocket is being closed right after it is established. The UI tries to maintain a persistent websocket connection with the agent in order to receive the stats.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

shabbirkagalwala avatar Jul 05 '17 22:07 shabbirkagalwala

@shabbirkagalwala you bet. Here's another troubleshooting tactic - you can test for connectivity as well as hunt for the statistics you're missing for via the Agent's REST API by pointing your browser at any of these endpoints:

  • http://a-host-running-the-agent:9090/environment
  • http://a-host-running-the-agent:9090/networks
  • http://a-host-running-the-agent:9090/containers
  • http://a-host-running-the-agent:9090/containers/container-id

@derhally can point to which of these should have the CPU & memory statistics.

leecalcote avatar Jul 05 '17 23:07 leecalcote

@leecalcote Thank you for the response.

I tried all the links as you mentioned and I can see the output. For example:

{ "hostname": "ip-192-168-3-131.us-west-2.compute.internal", "architecture": "x86_64", "totalmemory": 7933190144, "numberofcpu": 2, "ostype": "linux", "osversion": "CentOS Linux 7 (Core)", "containerplatform": "Docker", "containerplatformversion": "17.03.1-ce", "containercount": 5, "containerspaused": 0, "containersrunning": 5, "containersstopped": 0, "numberofimages": 71, "typesofnetworks": [ "bridge", "host", "macvlan", "null", "overlay" ]

I guess as mentioned by @derhally the agents not being able to communicate with the web-ui might be the issue. I am going to try tweaking my security groups and Network ACL's to see if anything helps and will get back here to report on what i find.

Thank you for all the help so far, truly appreciate it!

EDIT

Looking at the :9090/containers/container-id, this is what i found, it says 404 page not found at the end, could that be an issue?

{ "id": "b752b3b32d2849608e31d0a1413425782c19cd957846ad04998d736a1ec6141f", "Names": [ "/swi-agent" ], "Image": "solarwinds/container-agent", "ImageID": "sha256:fb5347c04f225d0bae6ef2a42f5ba79f9c08c66886e537cd661586c34d25b864", "Created": 1498850039, "Ports": [], "Labels": {}, "State": "running", "Status": "Up 5 days", "NetworkSettings": { "Networks": { "host": { "Links": null, "Aliases": null, "NetworkID": "33811c2c6a416be08e65dfd67f3a16ee336833ff441a792002a0e4c4f4075f76", "EndpointID": "962f69808fa9122e46e11fdc958102a01a2edd8eba86c7f01187298b82dae104", "IPAddress": "", "IPPrefixLen": 0, "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "" } } } }404 page not found

EDIT 2

I opened up all ports and allowed traffic still cant see the stats. I think the ports aren't the issue now, let me know if I need to do anything else to troubleshoot.

shabbirkagalwala avatar Jul 06 '17 14:07 shabbirkagalwala