htop-dev
Load average - display CPU load and I/O load separately
On my Synology DS220+ (great device) I found that htop is somehow improved? It's not only displaying the row like
- Load average: 0.03 0.03 0.03
... but also two more very useful rows:
- SYNOIO Load average: 0.02 0.02 0.02
- SYNOCPU Load average: 0.01 0.01 0.01
Is it possible to add this functionality to the htop? I think it's the most useful information to know what part of load is caused by CPU .. and what part of I/O (read/write to disc).
@Advoka looks like Synology ship a modified htop - is the source code available somewhere?
It looks like they modified htop. Based on the "SYNO" keywords (SYNOIO, SYNOCPU) displayed. I have no idea about source codes availability. I found this discussion https://www.reddit.com/r/synology/comments/cn9qnd/what_distribution_of_linux_is_synology_using/ where it's mentioned Synology's OS: "it's a very heavily customized Debian fork". Btw. my Synology DS220+ NAS device is this one https://www.synology.com/en-global/products/DS220+
IIUC, you may want to look at the PSI meters - they aren't completely equivalent to load average, but may be what you want.
Synology is forced to publish source code for GPL licensed programs they use. Go to Synology Archive, pick the latest DSM version then look for htop for the platform you are interested.
It would be so great to add this feature to htop core!
Thank you very much for that link.
Do you have some information where to find the exact platform version for the NAS from its model name?
Edit 1: Found something here …
Edit 2: Good news and bad news. The good news is, I located, where the code for those meters is. It's basically the following block of code:
diff --git a/LoadAverageMeter.c b/LoadAverageMeter.c
index e29433f1..eb034ea4 100644
--- a/LoadAverageMeter.c
+++ b/LoadAverageMeter.c
@@ -1,3 +1,6 @@
+#ifndef MY_ABC_HERE
+#define MY_ABC_HERE
+#endif
/*
htop - LoadAverageMeter.c
(C) 2004-2011 Hisham H. Muhammad
@@ -9,6 +12,7 @@ in the source distribution for its full text.
#include "CRT.h"
#include "Platform.h"
+#include "syno.h" // SYNO
/*{
#include "Meter.h"
@@ -17,6 +21,14 @@ in the source distribution for its full text.
int LoadAverageMeter_attributes[] = {
LOAD_AVERAGE_ONE, LOAD_AVERAGE_FIVE, LOAD_AVERAGE_FIFTEEN
};
+#ifdef MY_ABC_HERE
+int SYNOIO_LoadAverageMeter_attributes[] = {
+ LOAD_AVERAGE_ONE, LOAD_AVERAGE_FIVE, LOAD_AVERAGE_FIFTEEN
+};
+int SYNOCPU_LoadAverageMeter_attributes[] = {
+ LOAD_AVERAGE_ONE, LOAD_AVERAGE_FIVE, LOAD_AVERAGE_FIFTEEN
+};
+#endif /* MY_ABC_HERE */
int LoadMeter_attributes[] = { LOAD };
@@ -25,6 +37,17 @@ static void LoadAverageMeter_updateValues(Meter* this, char* buffer, int size) {
xSnprintf(buffer, size, "%.2f/%.2f/%.2f", this->values[0], this->values[1], this->values[2]);
}
+#ifdef MY_ABC_HERE
+static void SYNOIO_LoadAverageMeter_updateValues(Meter* this, char* buffer, int size) {
+ Platform_SYNOIO_getLoadAverage(&this->values[0], &this->values[1], &this->values[2]);
+ xSnprintf(buffer, size, "%.2f/%.2f/%.2f", this->values[0], this->values[1], this->values[2]);
+}
+static void SYNOCPU_LoadAverageMeter_updateValues(Meter* this, char* buffer, int size) {
+ Platform_SYNOCPU_getLoadAverage(&this->values[0], &this->values[1], &this->values[2]);
+ xSnprintf(buffer, size, "%.2f/%.2f/%.2f", this->values[0], this->values[1], this->values[2]);
+}
+#endif /* MY_ABC_HERE */
+
static void LoadAverageMeter_display(Object* cast, RichString* out) {
Meter* this = (Meter*)cast;
char buffer[20];
@@ -85,3 +108,38 @@ MeterClass LoadMeter_class = {
.description = "Load: average of ready processes in the last minute",
.caption = "Load: "
};
+#ifdef MY_ABC_HERE
+MeterClass SYNOIO_LoadAverageMeter_class = {
+ .super = {
+ .extends = Class(Meter),
+ .delete = Meter_delete,
+ .display = LoadAverageMeter_display,
+ },
+ .updateValues = SYNOIO_LoadAverageMeter_updateValues,
+ .defaultMode = TEXT_METERMODE,
+ .maxItems = 3,
+ .total = 100.0,
+ .attributes = SYNOIO_LoadAverageMeter_attributes,
+ .name = "SYNOIO_LoadAverage",
+ .uiName = "SYNOIO Load average",
+ .description = "Load averages: 1 minute, 5 minutes, 15 minutes",
+ .caption = "SYNOIO Load average: "
+};
+
+MeterClass SYNOCPU_LoadAverageMeter_class = {
+ .super = {
+ .extends = Class(Meter),
+ .delete = Meter_delete,
+ .display = LoadAverageMeter_display,
+ },
+ .updateValues = SYNOCPU_LoadAverageMeter_updateValues,
+ .defaultMode = TEXT_METERMODE,
+ .maxItems = 3,
+ .total = 100.0,
+ .attributes = SYNOCPU_LoadAverageMeter_attributes,
+ .name = "SYNOCPU_LoadAverage",
+ .uiName = "SYNOCPU Load average",
+ .description = "Load averages: 1 minute, 5 minutes, 15 minutes",
+ .caption = "SYNOCPU Load average: "
+};
+#endif /* MY_ABC_HERE */
diff --git a/LoadAverageMeter.h b/LoadAverageMeter.h
index bd18f4d0..6c10d2e0 100644
--- a/LoadAverageMeter.h
+++ b/LoadAverageMeter.h
@@ -1,3 +1,6 @@
+#ifndef MY_ABC_HERE
+#define MY_ABC_HERE
+#endif
/* Do not edit this file. It was automatically generated. */
#ifndef HEADER_LoadAverageMeter
@@ -10,6 +13,7 @@ in the source distribution for its full text.
*/
#include "Meter.h"
+#include "syno.h" // SYNO
extern int LoadAverageMeter_attributes[];
@@ -19,4 +23,11 @@ extern MeterClass LoadAverageMeter_class;
extern MeterClass LoadMeter_class;
+#ifdef MY_ABC_HERE
+extern int SYNOIO_LoadAverageMeter_attributes[];
+extern int SYNOCPU_LoadAverageMeter_attributes[];
+extern MeterClass SYNOIO_LoadAverageMeter_class;
+extern MeterClass SYNOCPU_LoadAverageMeter_class;
+#endif /* MY_ABC_HERE */
+
#endif
diff --git a/linux/Platform.c b/linux/Platform.c
index ab90ca74..fe4d235a 100644
--- a/linux/Platform.c
+++ b/linux/Platform.c
@@ -1,3 +1,6 @@
+#ifndef MY_ABC_HERE
+#define MY_ABC_HERE
+#endif
/*
htop - linux/Platform.c
(C) 2014 Hisham H. Muhammad
@@ -22,6 +25,7 @@ in the source distribution for its full text.
#include "ClockMeter.h"
#include "HostnameMeter.h"
#include "LinuxProcess.h"
+#include "syno.h" // SYNO
#include <math.h>
#include <assert.h>
@@ -113,6 +117,10 @@ MeterClass* Platform_meterTypes[] = {
&ClockMeter_class,
&LoadAverageMeter_class,
&LoadMeter_class,
+#ifdef MY_ABC_HERE
+ &SYNOIO_LoadAverageMeter_class,
+ &SYNOCPU_LoadAverageMeter_class,
+#endif /* MY_ABC_HERE */
&MemoryMeter_class,
&SwapMeter_class,
&TasksMeter_class,
@@ -153,6 +161,34 @@ void Platform_getLoadAverage(double* one, double* five, double* fifteen) {
}
}
+#ifdef MY_ABC_HERE
+void Platform_SYNOIO_getLoadAverage(double* IO_one, double* IO_five, double* IO_fifteen) {
+ *IO_one = 0; *IO_five = 0; *IO_fifteen = 0;
+ double CPU_one = 0, CPU_five = 0, CPU_fifteen = 0;
+ FILE *fd = fopen(PROCDIR "/syno_loadavg", "r");
+ if (fd) {
+ int total = fscanf(fd, "%32lf %32lf %32lf %32lf %32lf %32lf", IO_one, IO_five, IO_fifteen,
+ &CPU_one, &CPU_five, &CPU_fifteen);
+ (void) total;
+ assert(total == 6);
+ fclose(fd);
+ }
+}
+
+void Platform_SYNOCPU_getLoadAverage(double* CPU_one, double* CPU_five, double* CPU_fifteen) {
+ *CPU_one = 0; *CPU_five = 0; *CPU_fifteen = 0;
+ double IO_one = 0, IO_five = 0, IO_fifteen = 0;
+ FILE *fd = fopen(PROCDIR "/syno_loadavg", "r");
+ if (fd) {
+ int total = fscanf(fd, "%32lf %32lf %32lf %32lf %32lf %32lf", &IO_one, &IO_five, &IO_fifteen,
+ CPU_one, CPU_five, CPU_fifteen);
+ (void) total;
+ assert(total == 6);
+ fclose(fd);
+ }
+}
+#endif /* MY_ABC_HERE */
+
int Platform_getMaxPid() {
FILE* file = fopen(PROCDIR "/sys/kernel/pid_max", "r");
if (!file) return -1;
diff --git a/linux/Platform.h b/linux/Platform.h
index b0456e5b..ca3e606d 100644
--- a/linux/Platform.h
+++ b/linux/Platform.h
@@ -1,3 +1,6 @@
+#ifndef MY_ABC_HERE
+#define MY_ABC_HERE
+#endif
/* Do not edit this file. It was automatically generated. */
#ifndef HEADER_Platform
@@ -14,6 +17,7 @@ in the source distribution for its full text.
#include "BatteryMeter.h"
#include "LinuxProcess.h"
#include "SignalsPanel.h"
+#include "syno.h" // SYNO
#ifndef CLAMP
#define CLAMP(x,low,high) (((x)>(high))?(high):(((x)<(low))?(low):(x)))
@@ -34,6 +38,11 @@ extern MeterClass* Platform_meterTypes[];
int Platform_getUptime();
void Platform_getLoadAverage(double* one, double* five, double* fifteen);
+#ifdef MY_ABC_HERE
+void Platform_SYNOIO_getLoadAverage(double* IO_one, double* IO_five, double* IO_fifteen);
+
+void Platform_SYNOCPU_getLoadAverage(double* CPU_one, double* CPU_five, double* CPU_fifteen);
+#endif /* MY_ABC_HERE */
int Platform_getMaxPid();
diff --git a/syno.h b/syno.h
new file mode 100644
index 00000000..05141b7e
--- /dev/null
+++ b/syno.h
@@ -0,0 +1,10 @@
+#ifndef MY_ABC_HERE
+#define MY_ABC_HERE
+#endif
+/*
+ * Copyright (C) 2020 Synology Inc. All rights reserved.
+ */
+
+/*
+ * SYNO Load Average show in htop
+ */
Now for the bad news:
#ifdef MY_ABC_HERE
void Platform_SYNOIO_getLoadAverage(double* IO_one, double* IO_five, double* IO_fifteen) {
*IO_one = 0; *IO_five = 0; *IO_fifteen = 0;
double CPU_one = 0, CPU_five = 0, CPU_fifteen = 0;
FILE *fd = fopen(PROCDIR "/syno_loadavg", "r");
if (fd) {
int total = fscanf(fd, "%32lf %32lf %32lf %32lf %32lf %32lf", IO_one, IO_five, IO_fifteen,
&CPU_one, &CPU_five, &CPU_fifteen);
(void) total;
assert(total == 6);
fclose(fd);
}
}
void Platform_SYNOCPU_getLoadAverage(double* CPU_one, double* CPU_five, double* CPU_fifteen) {
*CPU_one = 0; *CPU_five = 0; *CPU_fifteen = 0;
double IO_one = 0, IO_five = 0, IO_fifteen = 0;
FILE *fd = fopen(PROCDIR "/syno_loadavg", "r");
if (fd) {
int total = fscanf(fd, "%32lf %32lf %32lf %32lf %32lf %32lf", &IO_one, &IO_five, &IO_fifteen,
CPU_one, CPU_five, CPU_fifteen);
(void) total;
assert(total == 6);
fclose(fd);
}
}
#endif /* MY_ABC_HERE */
This code accesses a file PROCDIR "/syno_loadavg" (AKA /proc/syno_loadavg), that is specific to Synology NAS and does not exist on normal Linux kernels.
Thus, unless we can gather those values from elsewhere, this ain't gonna work …
Thanks for looking into it.
Do you have some information where to find the exact platform version for the NAS from its model name?
Edit 1: Found something here …
Exactly this link.
Here is the output of /proc/syno_loadavg for the reference
$ cat /proc/syno_loadavg
4.22 4.48 4.80 0.54 0.60 0.77
Only /usr/local/packages/@appdata/ActiveInsight/collectors_all/current/cpu_loadavg_collector.py mentions that file (search excluded binaries of course), but it is only reading it as far as I can understand.
#!/usr/bin/env python
from prometheus_client.metrics_core import *
import subprocess
def read_oneline_split(filepath):
with open(filepath, "r") as f:
return f.readline().split()
class LoadAvgCollector(object):
def __init__(self, registry):
self._prefix = "cpu_"
register_config = {
"collect_func": [{
"interval_sec": 6,
"func": self.collect_6s,
"metrics": [self._prefix + "loadavg",
self._prefix + "io_loadavg",
self._prefix + "cpu_loadavg",
self._prefix + "user_utilization",
self._prefix + "system_utilization"]
}]
}
if registry:
registry.register(self, register_config)
def get_snmp_cpu_util(self, mib):
argument = ['/bin/snmpwalk', '-v', '2c', '-c', 'syno', 'localhost', mib]
p = subprocess.Popen(argument, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if 0 != p.returncode:
raise Exception("Fail to call snmpwalk subprocess")
return out.split()[-1]
def collect_6s(self):
loadavg = GaugeMetricFamily(self._prefix + "loadavg", "Overall Load Average", labels=["mins"])
io_loadavg = GaugeMetricFamily(self._prefix + "io_loadavg", "IO Load Average", labels=["mins"])
cpu_loadavg = GaugeMetricFamily(self._prefix + "cpu_loadavg", "CPU Load Average", labels=["mins"])
cpu_user_utilization = GaugeMetricFamily(self._prefix + "user_utilization", "CPU User Utilization", labels=[])
cpu_system_utilization = GaugeMetricFamily(self._prefix + "system_utilization", "CPU System Utilization", labels=[])
try:
data = read_oneline_split("/proc/loadavg")
loadavg.add_metric(["1"], float(data[0]))
loadavg.add_metric(["5"], float(data[1]))
loadavg.add_metric(["15"], float(data[2]))
except Exception as e:
raise Exception("Failed to read loadavg")
try:
data = read_oneline_split("/proc/syno_loadavg")
io_loadavg.add_metric(["1"], float(data[0]))
io_loadavg.add_metric(["5"], float(data[1]))
io_loadavg.add_metric(["15"], float(data[2]))
cpu_loadavg.add_metric(["1"], float(data[3]))
cpu_loadavg.add_metric(["5"], float(data[4]))
cpu_loadavg.add_metric(["15"], float(data[5]))
except Exception as e:
raise Exception("Failed to read syno_loadavg")
try:
user_util = float(self.get_snmp_cpu_util('1.3.6.1.4.1.2021.11.9'))
system_util = float(self.get_snmp_cpu_util('1.3.6.1.4.1.2021.11.10'))
if user_util < 0.0 or system_util < 0.0:
raise Exception("Fail to get cpu utilization from snmp")
cpu_user_utilization.add_metric([], user_util/100)
cpu_system_utilization.add_metric([], system_util/100)
except Exception as e:
raise Exception("Failed to read cpu load")
return [loadavg, io_loadavg, cpu_loadavg, cpu_user_utilization, cpu_system_utilization]
I dug a bit deeper even, and also located the kernel code (from the Synology NAS) that generates those additional io/cpu load averages. Unfortunately this needs some quite tight integration in the system kernel, as the code integrates directly in the scheduler (basically the code paths also taken by the normal loadavg metrics). This also explains, why those numbers line up so nicely: They are generated from the same data source.
I dug a bit deeper even, and also located the kernel code (from the Synology NAS)
Have you found their kernel source code? Could you share a link please?
Or is it actually these archives?
Yes, it's in the Linux archives.
Take a look at linux-4.4.x/kernel/sched/loadavg.c, i particular the sections marked by the MY_ABC_HERE defines. The following variables are the important ones to look for:
#ifdef MY_ABC_HERE
atomic_long_t calc_io_load_tasks;
atomic_long_t calc_cpu_load_tasks;
#endif /* MY_ABC_HERE */
unsigned long calc_load_update;
unsigned long avenrun[3];
#ifdef MY_ABC_HERE
unsigned long avenrun_io[3];
unsigned long avenrun_cpu[3];
#endif /* MY_ABC_HERE */