nerdctl icon indicating copy to clipboard operation
nerdctl copied to clipboard

Published 20 hours ago • verified publisher icon containerd

The output of `nerdctl stats` disappears after a few minutes

Open AkihiroSuda opened this issue 3 years ago • 10 comments

Description

The output of nerdctl stats disappears after a few minutes

Steps to reproduce the issue

  1. nerdctl run -d alpine sleep infinity
  2. nerdctl stats
  3. Wait for a few minute

Describe the results you received and expected

Received:

CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
ce15db2ec247   alpine-ce15d   --        -- / --             --        --        --          --

Expected: some stats

What version of nerdctl are you using?

v0.22.0-42-g4c37225

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

No response

Host information

Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.6.6
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file
  Storage: native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-41-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.59GiB
...

AkihiroSuda avatar Jul 30 '22 06:07 AkihiroSuda

@fahedouch PTAL

AkihiroSuda avatar Jul 30 '22 06:07 AkihiroSuda

@AkihiroSuda I tested stats for 15 minutes and didn't get this behavior. I got an empty stats only when I killed the container after this 15 minutes of fetching stats

extract of my logs :

CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   99.81%    0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   100.21%   0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   99.40%    0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   100.41%   0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   100.15%   0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   99.85%    0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   98.59%    0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   101.43%   0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   100.11%   0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   99.95%    0B / 0B             0.00%     920B / 962B   0B / 0B     3
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   40.71%    0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.09%     0B / 0B             0.00%     920B / 962B   0B / 0B     2
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   1.16%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   0.00%     0B / 0B             0.00%     920B / 962B   0B / 0B     1
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   --        -- / --             --        --        --          --
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   --        -- / --             --        --        --          --
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   --        -- / --             --        --        --          --
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   --        -- / --             --        --        --          --
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   --        -- / --             --        --        --          --
CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
24c458aa71b2   alpine-24c45   --        -- / --             --        --        --          --

Please check if your container is still alive

fahedouch avatar Jul 31 '22 10:07 fahedouch

I can't reproduce too.

junnplus avatar Jul 31 '22 10:07 junnplus

@fahedouch @junnplus

I still hit this issue. Could you test it on a cgroup v2 host such as Ubuntu 22.04 ?

AkihiroSuda avatar Aug 01 '22 04:08 AkihiroSuda

Sorry, I retried a few times to make sure it's reproducible.

$ sudo nerdctl info
Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.6.4
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file
  Storage: native overlayfs stargz
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-41-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.74GiB
 Name: lima-default
 ID: 2eef127c-1a0f-466b-b5eb-364a761abbdf

junnplus avatar Aug 01 '22 05:08 junnplus

Looks like I can hit the issue with cgroup v1 mode too

Server:
 Server Version: v1.6.6
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file
  Storage: native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.0-43-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.59GiB

AkihiroSuda avatar Aug 01 '22 05:08 AkihiroSuda

I got on cgroup v1:

route ip+net: netlinkrib: too many open files

It's maybe due to the (netns?) FD leak: https://github.com/containerd/nerdctl/blob/4c37225fa31a63e32162f34bdbc26554430e6dc6/cmd/nerdctl/stats.go#L438

And here, when error occurred, should print it to screen? https://github.com/containerd/nerdctl/blob/4c37225fa31a63e32162f34bdbc26554430e6dc6/cmd/nerdctl/stats.go#L474-L478

And this can fix the FD leak:

diff --git a/cmd/nerdctl/stats_linux.go b/cmd/nerdctl/stats_linux.go
index 7279c87..aae77c8 100644
--- a/cmd/nerdctl/stats_linux.go
+++ b/cmd/nerdctl/stats_linux.go
@@ -62,11 +62,13 @@ func setContainerStatsAndRenderStatsEntry(previousStats *statsutil.ContainerStat
                if err != nil {
                        return statsutil.StatsEntry{}, fmt.Errorf("failed to retrieve the statistics in netns %s: %v", ns, err)
                }
+               defer ns.Close()
 
                nlHandle, err = netlink.NewHandleAt(ns)
                if err != nil {
                        return statsutil.StatsEntry{}, fmt.Errorf("failed to retrieve the statistics in netns %s: %v", ns, err)
                }
+               defer nlHandle.Close()
 
                for _, v := range interfaces {
                        nlink, err = nlHandle.LinkByIndex(v.Index)

liubin avatar Aug 03 '22 09:08 liubin

Thank you @liubin , would you mind submitting a PR?

AkihiroSuda avatar Aug 04 '22 07:08 AkihiroSuda

@AkihiroSuda @liubin I am working on PR to fix this

fahedouch avatar Aug 04 '22 07:08 fahedouch

@fahedouch Please go on with your work. I only did some verify locally.

liubin avatar Aug 04 '22 11:08 liubin

Closed by https://github.com/containerd/nerdctl/pull/1298

fahedouch avatar Aug 25 '22 09:08 fahedouch