htop-dev
Corrections to the MemoryMeter for all non-Linux platforms
@Explorer09 @PierreMarieBaty it's a complicated situation, which Pierre-Marie posted the following to over in issue #1725
Nathan said:
This new interpretation of "buffers" as something that is also "used" isn't really working out too well, IMO from looking at the patch - the increasing ifdef use is a big red flag. I'll update the PR and add some review comments - let's continue this discussion there.
Pierre-Marie replied:
That is because, as I said, each operating system has its own memory classes, with its own terminology for each, and sometimes the meaning of a label (here, "buffers") differs from operating system A to operating system B.
I can't stress this out enough. The meaning (and thus the handling - whether to count it as "used" memory or not, whether to display it or not when "show cache" is ticked) of class "X" is simply not the same on OS A compared to OS B.
For instance, the meaning of "buffered" memory on FreeBSD is not the same as on Linux, and not the same as on e.g. NetBSD. In FreeBSD you count "buffered" memory among used memory, because it's a subclass of the "wired" memory. In Linux you don't count "buffered" memory among the used set, because "buffered" has a different meaning there.
I'm not even talking about other operating systems such as QNX here because you'd probably be pulling your hair...
As I said elsewhere, trying to map each of these OS-specific classes into a unified set of categories is wrong design IMO, and doomed to end like this. Sooner or later you'll hit an OS where the meaning of a label will be totally different, and that class will need a special treatment different of the rest of the crowd. What to do then? Add an OS-specific new memory category? That defeats the purpose of the unified list (portability). Rename the label just for that OS? That's basically moving the #ifdefs elsewhere. Or redesign the whole memory presentation logic in a totally OS-specific way?
Besides, I restate that the purpose of htop is to be an "improved top", and in this regard - in an ideal world - it should be critical that the semantics of the host OS be respected when displaying memory information. Else the users will be confused. For example you'd be tempted on FreeBSD to stuff the "buffered" memory into htop's "cache" category. Yet htop also has a "buffered" category. The FreeBSD user will look at htop's memory readings and think: where's my buffered memory gone?
So, I think we're all cornered to admit the only right choice is the most unpleasant one here...
As for me, I now have a working htop memory meter for FreeBSD and I'm happy with it - even if the code isn't the clean design paradise the maintainers would dream of, and even if the semantics of its memory classes isn't quite the FreeBSD one, it gets the job done and the categories are mapped almost correctly (as far as FreeBSD is concerned, in htop I'd separate "wired" and "active" memory with a red color for the "wired" one as it's almost always kernel pages whereas "active" are userland pages, but that'll be for another day.)
@PierreMarieBaty I'm largely agreeing with you, right up until you say we are cornered and have to accept a mediocre solution for FreeBSD. If we compromise in the way this PR suggests now, the situation is unlikely to improve any time soon.
[...] the semantics of its memory classes isn't quite the FreeBSD one, it gets the job done and the categories are mapped almost correctly (as far as FreeBSD is concerned, in htop I'd separate "wired" and "active" memory with a red color for the "wired" one as it's almost always kernel pages whereas "active" are userland pages, but that'll be for another day.)
Today is another day! Let's improve this for all FreeBSD htop users. As I described earlier, and you wrote here...
What to do then? Add an OS-specific new memory category? That defeats the purpose of the unified list (portability). Rename the label just for that OS? That's basically moving the #ifdefs elsewhere. Or redesign the whole memory presentation logic in a totally OS-specific way?
Yes, these questions are spot on (the answers don't give the complete picture, but no matter).
As I see it, there are two practical approaches:
- add new categories to the MemoryMeter, and adjust the code using isNegative and isPositive such that other platforms continue as they do today, and FreeBSD gets the exact output you describe as optimal. If there are exact category mappings, we reuse those parts of the existing surrounding code, else we add formatting code for the new categories.
- if this approach does not work out and we're unhappy with the resulting combined MemoryMeter, we instead create a freebsd/MemoryMeter.[ch] that implements the optimal FreeBSD memory meter layouts. Then for FreeBSD builds we use this implementation rather than the cross-platform one (via some straightforward Makefile.am changes). We use a similar approach in other places already - see linux/*Meter* source code for example, which provides Linux-specific Meters.
My preference is to start with option 1 and fallback to option 2. Either way, the end result should be the improved Memory reporting on FreeBSD that you described.
@PierreMarieBaty @natoscott I need some more explanation on the differences between "buffer" memory as used in Linux vs. "buffer" memory used in FreeBSD.
For now, your arguments do not convince me. Yes, the same terminology used differently in different operating systems can be confusing, but neither of the solutions you (in particular natoscott) proposed are usable in the long run. No, there shouldn't be any OS-specific category for MemoryMeter. The main reason is that it would be hellish to document them in the man page, where there would be no OS-specific descriptions.
If I understand it correctly, the "buffer" memory in FreeBSD is a subcategory of "used" memory and cannot be freed out by the kernel (on demand), then how about this: Remove the "buffer" memory from display for FreeBSD htop, count the "buffer" memory as part of "used", and add in-code comments to explain the reason this decision.
There are already operating-system specific categories in MemoryMeter (compressed). And FWLIW, "buffers" doesn't mean anything sensible anymore for at least two modern Linux filesystems (btrfs and xfs). There's no reason not to add categories to help people make sense of memory use on FreeBSD.
The main reason is that it would be hellish to document them in the man page, where there would be no OS-specific descriptions.
There is no documentation of the Memory Meter on the man page, or any of the categories of memory it displays.
I need some more explanation on the differences between "buffer" memory as used in Linux vs. "buffer" memory used in FreeBSD.
I don’t know what this means in Linux. In FreeBSD, this is the part of kernel memory where all disk I/O are processed for all filesystems that have a "soft updates"-like caching mechanism - except for ZFS, where FreeBSD’s top maintains a separate line for ZFS stats. Buffer memory is not freeable on FreeBSD and must be counted as "used" memory.
For now, your arguments do not convince me. Yes, the same terminology used differently in different operating systems can be confusing, but neither of the solutions you (in particular natoscott) proposed are usable in the long run. No, there shouldn't be any OS-specific category for MemoryMeter. The main reason is that it would be hellish to document them in the man page, where there would be no OS-specific descriptions.
In my humble opinion, if htop’s man page covers memory readings, then, precisely, the man page should say different things for different operating systems.
The question boils down to this : was htop meant to be an "improved top", or just yet another generic common-ground multiplatform program?
If htop is an improved top, it should not say less than what the OS-specific top says. Else, the promise is not kept.
If I understand it correctly, the "buffer" memory in FreeBSD is a subcategory of "used" memory and cannot be freed out by the kernel (on demand), then how about this: Remove the "buffer" memory from display for FreeBSD htop, count the "buffer" memory as part of "used", and add in-code comments to explain the reason this decision.
That’s doing less than what top says, and that’s doing less than what the patch does, whereas the possibility exists. At this point, why not just leave only 2 categories for all the operating systems, used and free? Everyone would agree, and that would be OS-agnostic and portable.
From the very moment you add extra categories to "used" and "free", you are OS-specific.
Let’s be honest: htop was initially a Linux tool, and the memory classes design, as it is, wasn’t made with portability in mind but simply to reflect Linux’s memory classes, so saying "this and that will make the code not OS-agnostic", on a part of the code that really isn’t from day one, can’t be a real argument.
natoscott said:
we instead create a freebsd/MemoryMeter.[ch] that implements the optimal FreeBSD memory meter layouts. Then for FreeBSD builds we use this implementation rather than the cross-platform one (via some straightforward Makefile.am changes). We use a similar approach in other places already - see linux/Meter source code for example, which provides Linux-specific Meters.
I agree. Every supported OS flavor of htop should do that.
Memory information cannot be OS-agnostic and it is a dead end to continue pushing in the same direction, which has never been an OS-agnostic one but the simple legacy of a Linux-centric tool.
FYI I refreshed my memory on what "buffered" means for FreeBSD vs. Linux. In a nutshell:
FreeBSD buffers == Linux buffers+cache
FreeBSD uses a unified cache and both categories are not separable.
| FreeBSD uses a unified cache and both categories are not separable.
This is often true on Linux also - xfs and btrfs use a unified page cache approach.
TBH, "buffers" could probably be merged into "cached" on Linux as well.
With the additional caveat that on one OS that cache should be counted as "free" memory and on the other OS it should be counted as "used". And on a third one, well.. it’ll depend.
I see no other way out than to go full OS-specific here.
For now, your arguments do not convince me. Yes, the same terminology used differently in different operating systems can be confusing, but neither of the solutions you (in particular natoscott) proposed are usable in the long run. No, there shouldn't be any OS-specific category for MemoryMeter. The main reason is that it would be hellish to document them in the man page, where there would be no OS-specific descriptions.
In my humble opinion, if htop’s man page covers memory readings, then, precisely, the man page should say different things for different operating systems. The question boils down to this : was htop meant to be an "improved top", or just yet another generic common-ground multiplatform program? If htop is an improved top, it should not say less than what the OS-specific top says. Else, the promise is not kept.
I'd say it's the latter - a generic multiplatform program. There's no promise of htop to "not say less than what the OS-specific top says".
Of course you can argue that htop was a Linux utility - I agree, but that doesn't change the fact that there was no promise in the first place.
By the way, there's an entry in htop's FAQ that explains a difference between top on the memory meter display. So htop is not exactly top nor an "improved version" of it.
That’s doing less than what top says, and that’s doing less than what the patch does, whereas the possibility exists.
Ditto. There is no promise.
natoscott said:
we instead create a freebsd/MemoryMeter.[ch] that implements the optimal FreeBSD memory meter layouts. Then for FreeBSD builds we use this implementation rather than the cross-platform one (via some straightforward Makefile.am changes). We use a similar approach in other places already - see linux/Meter source code for example, which provides Linux-specific Meters.
I agree. Every supported OS flavor of htop should do that. Memory information cannot be OS-agnostic and it is a dead end to continue pushing in the same direction, which has never been an OS-agnostic one but the simple legacy of a Linux-centric tool.
Oh hell. I've worked with a mess when GPUMeter was OS-specific before. Someone proposed a GPUMeter for macOS/Darwin, and I had to rework that meter to make it usable OS-agnostic. Please look at the code of GPUMeter.{c,h} (for example cddacadb1ce24b105955e12ed35930b2343450cf) for what I mean.
FreeBSD buffers == Linux buffers+cache FreeBSD uses a unified cache and both categories are not separable.
With the additional caveat that on one OS that cache should be counted as "free" memory and on the other OS it should be counted as "used". And on a third one, well.. it’ll depend.
I see no other way out than to go full OS-specific here.
I kind of understood the situation.
The problem is on the showCachedMemory option itself - it was implemented in an OS-agnostic manner while the reality it should be implemented as OS-specific. So I got the idea of moving the showCachedMemory conditional and handling code to Platform.c for the respective platforms. That way you don't use the #ifdef HTOP_<platform> preprocessor conditional to guard them.
- For Linux, when
showCachedMemoryis off, "buffer" and "cache" would be hidden as if they were "free" memory. - For FreeBSD, when
showCachedMemoryis off, "buffer" will merge with "used" and no longer display as separate fields. - The text meter mode display of the MemoryMeter has to be adjusted accordingly as well.
The FreeBSD package maintainer wrote an incorrect package description then:
[pm@pmbaty (-bash) ~/htop-mine]$ pkg info htop htop-3.4.0 Name : htop Version : 3.4.0 Installed on : Tue Apr 8 14:10:14 2025 CEST Origin : sysutils/htop Architecture : FreeBSD:14:amd64 Prefix : /usr/local Categories : sysutils Licenses : GPLv2 Maintainer : [email protected] WWW : https://htop.dev/ Comment : Better top(1) - interactive process viewer Options : LSOF : off Shared Libs required: libc.so.7 libdevstat.so.7 libexecinfo.so.1 libkvm.so.7 libm.so.5 libncursesw.so.9 libtinfow.so.9 Annotations : FreeBSD_version: 1402000 build_timestamp: 2025-04-04T23:33:28+0000 built_by : poudriere-git-3.4.2-7-gf78625a5 port_checkout_unclean: no port_git_hash : 359bbf7fc ports_top_checkout_unclean: no ports_top_git_hash: 359bbf7fc repo_type : binary repository : FreeBSD Flat size : 267KiB Description : htop is an enhanced version of top, the interactive process viewer, which can display the list of processes in a tree form.
Comparison between 'htop' and 'top'
- In 'htop' you can scroll the list vertically and horizontally to see all processes and full command lines.
- In 'top' you are subject to a delay for each unassigned key you press (especially annoying when multi-key escape sequences are triggered by accident).
- 'htop' starts faster ('top' seems to collect data for a while before displaying anything).
- In 'htop' you don't need to type the process number to kill a process, in 'top' you do.
- In 'htop' you don't need to type the process number or the priority value to renice a process, in 'top' you do.
- In 'htop' you can kill multiple processes at once.
- 'top' is older, hence, more tested.
I thought I read that claim that "htop was an enhanced version of top" elsewhere and that was somewhat endorsed by the htop maintainers. My bad if that's never been the case. I have been misled.
(edit: trying to find where I got this wrong information in the first place, I searched Google for the exact phrase "htop is an enhanced version of top". It yielded 8 pages. It looks like everybody and his cat are firmly convinced of it. Even the Wikipedia article gets it wrong! Alright, I've been misled, but I feel less alone now... 😇)
(edit 2: aha, found it! On Hisham Muhammad's old page @ https://web.archive.org/web/20140121201425/http://hisham.hm/htop/index.php?page=faq : "The long explanation is that what inspired me to write htop was pinfo, an improved man and info reader that adds lots of features (in my machines 'man' is an alias to 'pinfo -m'). It was written by a guy called Przemek Borys. Since 'pinfo' was "a better info" and he named it "pinfo" ("Przemek's Info"), I decided to try to make "a better top" so I called it "htop" ("Hisham's top"). So yes, it is after my own name, but it's also a homage to another nice piece of software!" I'm a very long time htop user, and I've been believing what I read about htop in the 2000s all these 20 last years. I totally concede that it's been removed from the FAQ since and no longer true. Sorry for the nitpicking...)
I kind of understood the situation. The problem is on the showCachedMemory option itself - it was implemented in an OS-agnostic manner while the reality it should be implemented as OS-specific. So I got the idea of moving the showCachedMemory conditional and handling code to Platform.c for the respective platforms. That way you don't use the #ifdef HTOP_
preprocessor conditional to guard them. For Linux, when showCachedMemory is off, "buffer" and "cache" would be hidden as if they were "free" memory. For FreeBSD, when showCachedMemory is off, "buffer" will merge with "used" and no longer display as separate fields. The text meter mode display of the MemoryMeter has to be adjusted accordingly as well.
Anything that doesn't feel like a functional regression from the patch looks good to me. I'm okay with it.
What you did for the GPU meter was a good thing. I don't believe doing the same thing for memory would be more complicated. I might propose a patch doing this for all the platforms I can compile htop on (all except Solaris) if I can secure enough time these days for that.
I kind of understood the situation. The problem is on the showCachedMemory option itself - it was implemented in an OS-agnostic manner while the reality it should be implemented as OS-specific. So I got the idea of moving the showCachedMemory conditional and handling code to Platform.c for the respective platforms. That way you don't use the #ifdef HTOP_ preprocessor conditional to guard them. For Linux, when showCachedMemory is off, "buffer" and "cache" would be hidden as if they were "free" memory. For FreeBSD, when showCachedMemory is off, "buffer" will merge with "used" and no longer display as separate fields. The text meter mode display of the MemoryMeter has to be adjusted accordingly as well.
Anything that doesn't feel like a functional regression from the patch looks good to me. I'm okay with it.
What you did for the GPU meter was a good thing. I don't believe doing the same thing for memory would be more complicated. I might propose a patch doing this for all the platforms I can compile htop on (all except Solaris) if I can secure enough time these days for that.
One more question: Does FreeBSD top or free or what-have-you include the "buffer" memory into the count of "used" space? I would prefer the numbers become separate when the showCachedMemory is on, and the number merged (added together) when showCachedMemory is off. With this design in mind, I think I can propose a patch that you and natoscott can review. It may be applied before or after this PR.
The FreeBSD package maintainer wrote an incorrect package description then:
[pm@pmbaty (-bash) ~/htop-mine]$ pkg info htop htop-3.4.0 … Comment : Better top(1) - interactive process viewer … Description : htop is an enhanced version of top, the interactive process viewer, which can display the list of processes in a tree form. Comparison between 'htop' and 'top' …
You could argue on the semantics here, but "enhanced" only means there are some added features or other changes. Strictly speaking it's not automatically better suited for every user and use case …
I thought I read that claim that "htop was an enhanced version of top" elsewhere and that was somewhat endorsed by the htop maintainers. My bad if that's never been the case. I have been misled.
The origins of htop are rooted in the experiments from a Brazilian developer exploring how top works, guided by a design philosophy we still continue to honor. While htop focuses on exposing system details, it doesn’t strive to be an exact, fully faithful clone.
(edit 2: aha, found it! On Hisham Muhammad's old page @ https://web.archive.org/web/20140121201425/http://hisham.hm/htop/index.php?page=faq : "The long explanation is that what inspired me to write htop was pinfo, an improved man and info reader that adds lots of features (in my machines 'man' is an alias to 'pinfo -m'). It was written by a guy called Przemek Borys. Since 'pinfo' was "a better info" and he named it "pinfo" ("Przemek's Info"), I decided to try to make "a better top" so I called it "htop" ("Hisham's top"). So yes, it is after my own name, but it's also a homage to another nice piece of software!" I'm a very long time htop user, and I've been believing what I read about htop in the 2000s all these 20 last years. I totally concede that it's been removed from the FAQ since and no longer true. Sorry for the nitpicking...)
In the link above @hishamhm talks a bit more about the origins of htop. And it's a mixture of curiosity and wanting to improve over the existing tools. How much of this means following each OS to the letter is practically speaking an open question.
Anything that doesn't feel like a functional regression from the patch looks good to me. I'm okay with it.
That's also mostly my stand on things. The assignments should be reasonable, predictable and sensible. If they furthermore are also somewhat consistent across platforms ("used" doesn't randomly mean "swapped out filesystem cache" on some platform) it's even better. Thus if we can somehow find a good middle ground of categories of memory metrics we want/need to show to the user, this would be ideal.
Looking into things, we can even add the "Wired"/"Fixed" category on Linux, as that's a category of allocations that's currently missing entirely on Linux.
What you did for the GPU meter was a good thing. I don't believe doing the same thing for memory would be more complicated. I might propose a patch doing this for all the platforms I can compile htop on (all except Solaris) if I can secure enough time these days for that.
Looking forward. And no need to hurry.
With some code refactoring, I removed the need of the HTOP_FREEBSD preprocessor conditionals:
https://github.com/Explorer09/htop-1/commits/freebsd-memory/
https://github.com/Explorer09/htop-1/commit/f8e62e6f662145aab4358cca87a98fa2a38cfed3
The "used" numbers in MemoryMeter are now handled in platform-specific code, and the behavior of showCachedMemory is now also platform-specific.
For a UX reason, in FreeBSD, if the showCachedMemory is off, the "buffers" number will add to the "used" memory and not show as a separate number.
IMO this is a good step in the right direction !
About your earlier question, FreeBSD's top has no explicit notion of "used" memory: all the available memory is supposed to be used all the time, even if the reality is more subtle. But it is correct to say that "buffer" memory, being not releasable, counts as "used" memory in the sense any other traditional memory monitoring tool understands it.
@PierreMarieBaty I'm not asking about the "used" memory in FreeBSD top (I know FreeBSD doesn't use that term, but "Active" and "Wired"). My question was whether the "Wired" memory shown there includes the "Buf" memory or not.
@PierreMarieBaty I'm not asking about the "used" memory in FreeBSD top (I know FreeBSD doesn't use that term, but "Active" and "Wired"). My question was whether the "Wired" memory shown there includes the "Buf" memory or not.
Yes. In FreeBSD's top, the number displayed next to "Wired" includes the number displayed next to "Buf". This can be confusing. In other words, "Buf" is a subset of "Wired".
@PierreMarieBaty I'm not asking about the "used" memory in FreeBSD top (I know FreeBSD doesn't use that term, but "Active" and "Wired"). My question was whether the "Wired" memory shown there includes the "Buf" memory or not.
Yes. In FreeBSD's top, the number displayed next to "Wired" includes the number displayed next to "Buf". This can be confusing. In other words, "Buf" is a subset of "Wired".
Oh well. That's gonna be tricky. You can notice that in my implementation (https://github.com/Explorer09/htop-1/commit/f8e62e6f662145aab4358cca87a98fa2a38cfed3), the "used" memory (in Text meter mode) will not include the "buffers" number unless the users turns off the "show cached memory" option. This is to ease the bar drawing. I don't wish users complain about that. htop's memory display cannot be 1:1 matching FreeBSD top's display.
Oh well. That's gonna be tricky. You can notice that in my implementation (Explorer09@f8e62e6), the "used" memory (in Text meter mode) will not include the "buffers" number unless the users turns off the "show cached memory" option. This is to ease the bar drawing. I don't wish users complain about that. htop's memory display cannot be 1:1 matching FreeBSD top's display.
Not being 1:1 is okay (at ~200-300Mb off nobody will complain) but with that behaviour you're likely to get significantly different numbers, and new bug reports. Honestly. Consider the energy you're going to spend to have all these OS-specific subtleties (some of which you might still ignore 😕) fit in your OS-agnostic memory display code, and compare to the energy you'd spend to make the collection of memory stats, classification and display 100% OS-specific like I advised in the first place... In your shoes I'd bite the bullet and go the hard way. It may seem longer and harder, but it has the immense advantage that it's a predictable path that's guaranteed to yield something correct and clean.
Oh well. That's gonna be tricky. You can notice that in my implementation (Explorer09@f8e62e6), the "used" memory (in Text meter mode) will not include the "buffers" number unless the users turns off the "show cached memory" option. This is to ease the bar drawing. I don't wish users complain about that. htop's memory display cannot be 1:1 matching FreeBSD top's display.
Not being 1:1 is okay (at ~200-300Mb off nobody will complain) but with that behaviour you're likely to get significantly different numbers, and new bug reports. Honestly. Consider the energy you're going to spend to have all these OS-specific subtleties (some of which you might still ignore 😕) fit in your OS-agnostic memory display code, and compare to the energy you'd spend to make the collection of memory stats, classification and display 100% OS-specific like I advised in the first place... In your shoes I'd bite the bullet and go the hard way. It may seem longer and harder, but it has the immense advantage that it's a predictable path that's guaranteed to yield something correct and clean.
htop's memory display is not going to be fine-tuned to fit all operating systems' needs. There are UI elements in htop that are going to be OS-agnostic anyway (because otherwise it would become a maintenance hell). For example the fixed coloring of memory bar items. So for the peculiarities of FreeBSD I'm forced to make some compromises. You might notice that the "buffers" memory number will dim or be hidden when the "show cached memory" is off. That behavior has to be OS-agnostic (the option worked for Linux, but was not designed with FreeBSD peculiarities in mind). So for FreeBSD, I think I've done the best compromise that I can.
| htop's memory display cannot be 1:1 matching FreeBSD top's display.
Sure it can. I'm not saying it has to, though (looks like good progress toward the first option I described has been made - so I'm all for that), but if what top displays is the exact information that makes most sense and provides the best insights to performance for FreeBSD users, then this is what htop should display. Its just code, we can change internal mappings, macros, colours, etc as we need to.
What you did for the GPU meter was a good thing. I don't believe doing the same thing for memory would be more complicated. I might propose a patch doing this for all the platforms I can compile htop on (all except Solaris) if I can secure enough time these days for that.
Looking forward. And no need to hurry.
I decided it was best to strike the iron while it's hot, and I'm 90% done. My htop now displays the OS-specific memory classes for Darwin/macOS, DragonflyBSD, FreeBSD, Linux (no aspect change, just refactoring on this one), NetBSD and OpenBSD. This fixed several issues as a side-effect.
In each platform-specific folder 4 files have changed, and this cleans the code a lot. Much less math: the numbers retrieved by sysctls (or from a VM stats struct) are just passed around. And the help page also displays the OS-specific memory classes in the memory bar, selectively whether their OS-specific "this category can be assimilated to a maskable cache" flag is set, and with their OS-specific color. No #ifdefs anywhere. My MemoryMeter.h now looks like this:
#ifndef HEADER_MemoryMeter
#define HEADER_MemoryMeter
/*
htop - MemoryMeter.h
(C) 2004-2011 Hisham H. Muhammad
Released under the GNU GPLv2+, see the COPYING file
in the source distribution for its full text.
*/
#include "Meter.h"
typedef struct MemoryClass_s {
const char *label; // e.g. "used", "shared", "compressed", etc. Should reflect the system-specific 'top' classes
bool countsAsUsed; // whether this memory class counts as "used" memory
bool countsAsCache; // whether this memory class can be reclaimed under pressure (and displayed when "show cached memory" is checked)
ColorElements color; // one of the DYNAMIC_xxx CRT color values
} MemoryClass;
extern const MemoryClass Platform_memoryClasses[]; // defined in the platform-specific code
extern const unsigned int Platform_numberOfMemoryClasses; // defined in the platform-specific code
extern const MeterClass MemoryMeter_class;
#endif
And in each platform's Platform.c, there is something like (here e.g. for Darwin):
const MemoryClass Platform_memoryClasses[] = {
#define MEMORY_CLASS_WIRED 0
{ .label = "wired", .countsAsUsed = true, .countsAsCache = false, .color = DYNAMIC_RED }, // pages wired down to physical memory (kernel)
#define MEMORY_CLASS_SPECULATIVE 1
{ .label = "speculative", .countsAsUsed = true, .countsAsCache = true, .color = DYNAMIC_MAGENTA }, // readahead optimization caches
#define MEMORY_CLASS_ACTIVE 2
{ .label = "active", .countsAsUsed = true, .countsAsCache = false, .color = DYNAMIC_GREEN }, // userland pages actively being used
#define MEMORY_CLASS_PURGEABLE 3
{ .label = "purgeable", .countsAsUsed = false, .countsAsCache = true, .color = DYNAMIC_YELLOW }, // userland pages voluntarily marked "discardable" by apps
#define MEMORY_CLASS_COMPRESSED 4
{ .label = "compressed", .countsAsUsed = true, .countsAsCache = false, .color = DYNAMIC_BLUE }, // userland pages being compressed (means memory pressure++)
#define MEMORY_CLASS_INACTIVE 5
{ .label = "inactive", .countsAsUsed = true, .countsAsCache = true, .color = DYNAMIC_GRAY }, // pages no longer used; macOS counts them as "used" anyway...
}; // N.B. the chart will display categories in this order
const unsigned int Platform_numberOfMemoryClasses = ARRAYSIZE(Platform_memoryClasses);
const int Platform_memoryMeter_attributes[] = {
Platform_memoryClasses[0].color,
Platform_memoryClasses[1].color,
Platform_memoryClasses[2].color,
Platform_memoryClasses[3].color,
Platform_memoryClasses[4].color,
Platform_memoryClasses[5].color
}; // there MUST be as many entries in this attributes array as memory classes
The platform-agnostic MemoryMeter.c is now very simple:
/*
htop - MemoryMeter.c
(C) 2004-2011 Hisham H. Muhammad
Released under the GNU GPLv2+, see the COPYING file
in the source distribution for its full text.
*/
#include "config.h" // IWYU pragma: keep
#include "MemoryMeter.h"
#include <assert.h>
#include <math.h>
#include <stddef.h>
#include "CRT.h"
#include "Macros.h"
#include "Object.h"
#include "Platform.h"
#include "RichString.h"
extern const int Platform_memoryMeter_attributes[]; // OS-specific
static void MemoryMeter_updateValues(Meter* this) {
char* buffer = this->txtBuffer;
size_t size = sizeof(this->txtBuffer);
int written;
Settings *settings = this->host->settings;
/* not all memory classes are supported on all platforms */
for (unsigned int memoryClassIdx = 0; memoryClassIdx < Platform_numberOfMemoryClasses; memoryClassIdx++) {
this->values[memoryClassIdx] = NAN;
}
Platform_setMemoryValues(this);
this->curItems = (uint8_t) Platform_numberOfMemoryClasses;
/* compute the used memory */
double used = 0.0;
for (unsigned int memoryClassIdx = 0; memoryClassIdx < Platform_numberOfMemoryClasses; memoryClassIdx++) {
if (Platform_memoryClasses[memoryClassIdx].countsAsUsed)
used += this->values[memoryClassIdx];
}
/* clear the values we don't want to see */
if ((this->mode == GRAPH_METERMODE || this->mode == BAR_METERMODE) && !settings->showCachedMemory) {
for (unsigned int memoryClassIdx = 0; memoryClassIdx < Platform_numberOfMemoryClasses; memoryClassIdx++) {
if (Platform_memoryClasses[memoryClassIdx].countsAsCache)
this->values[memoryClassIdx] = NAN;
}
}
written = Meter_humanUnit(buffer, used, size);
METER_BUFFER_CHECK(buffer, size, written);
METER_BUFFER_APPEND_CHR(buffer, size, '/');
Meter_humanUnit(buffer, this->total, size);
}
static void MemoryMeter_display(const Object* cast, RichString* out) {
char buffer[50];
const Meter* this = (const Meter*)cast;
const Settings* settings = this->host->settings;
RichString_writeAscii(out, CRT_colors[METER_TEXT], ":");
Meter_humanUnit(buffer, this->total, sizeof(buffer));
RichString_appendAscii(out, CRT_colors[METER_VALUE], buffer);
/* print the OS-specific memory classes in the order supplied by their implementation */
for (unsigned int memoryClassIdx = 0; memoryClassIdx < Platform_numberOfMemoryClasses; memoryClassIdx++) {
if (!settings->showCachedMemory && Platform_memoryClasses[memoryClassIdx].countsAsCache)
continue; // skip reclaimable cache memory classes if "show cached memory" is not ticked
Meter_humanUnit(buffer, this->values[memoryClassIdx], sizeof(buffer));
RichString_appendAscii(out, CRT_colors[METER_TEXT], " ");
RichString_appendAscii(out, CRT_colors[METER_TEXT], Platform_memoryClasses[memoryClassIdx].label);
RichString_appendAscii(out, CRT_colors[METER_TEXT], ":");
RichString_appendAscii(out, CRT_colors[Platform_memoryClasses[memoryClassIdx].color], buffer);
}
}
const MeterClass MemoryMeter_class = {
.super = {
.extends = Class(Meter),
.delete = Meter_delete,
.display = MemoryMeter_display,
},
.updateValues = MemoryMeter_updateValues,
.defaultMode = BAR_METERMODE,
.supportedModes = METERMODE_DEFAULT_SUPPORTED,
.maxItems = NUMBER_OF_DYNAMIC_COLORS, // all the range of DYNAMIC_xxxx colors are allowed
.isPercentChart = true,
.total = 100.0,
.attributes = Platform_memoryMeter_attributes, // OS-specific
.name = "Memory",
.uiName = "Memory",
.caption = "Mem"
};
I'm currently fiddling with an OpenIndiana VM to mount a NFS share on which my htop source tree is, to be able to provide a complete patch that will compile on Solaris. Argh, this is really $*€ me off...
root@openindiana:~# mount -F nfs 192.168.1.102:/Users/pmbaty/Public/htop htop
nfs mount: 192.168.1.102:/Users/pmbaty/Public/htop: Permission denied
Does anyone have some experience with this OS? I could mount that NFS share from all my other Linux and *BSD VMs, but Solaris apparently decided that root was not trustworthy.
@PierreMarieBaty You're over-engineering things. And there's one problem with your approach that, color scheme designers (theme designers) are losing the capability of customizing colors for the memory fields. That's why I said earlier this was a bad idea.
Unlike GPU meter where items are by driver (or, in the user's perspective, GPU vendor), memory meter has common categories that are best colored consistently across platforms. Even we can tune the wording for each category by platform, let's not mess up with the color scheme or the ordering of the categories (e.g. "cached" memory categories should be after the "used" memory categories).
@PierreMarieBaty You're over-engineering things. And there's one problem with your approach that, color scheme designers (theme designers) are losing the capability of customizing colors for the memory fields. That's why I said earlier this was a bad idea.
Nope. The DYNAMIC_xxxx colors are themable - and already themed. Look in CRT.c, it's already done. Even on black/white displays, they fallback to different ASCII characters. I tested it on all the OSes on which I could compile.
Unlike GPU meter where items are by driver (or, in the user's perspective, GPU vendor), memory meter has common categories that are best colored consistently across platforms.
I'm sorry to differ but no it has not. Apart from "used" and "free", it has not. I can only advise you to have a dive in it yourself: install these operating systems in a VM, look at what "top" displays, look at their ways to extract memory information, so as to understand how they do thing. I have the strong feeling that you are under the illusion that every OS must be doing it in a roughly similar way than what Windows and Linux do. Nothing could be more wrong...
Even we can tune the wording for each category by platform, let's not mess up with the color scheme or the ordering of the categories (e.g. "cached" memory categories should be after the "used" memory categories).
No again, because "cached" memory will have a totally different meaning across 3 different systems! On some OS it's reclaimable, on some others it's not. On some OS it's optional, on some others it's mandatory. On some it caches filesystem metadata only, on some others it caches buffers, on some others it's part of a unified cache, on some others it's ONE of the various caching mechanisms, on some others it's part of a memory zone... I really don't want to sound offensive, but you need to get more understanding on how differently the operating systems grasp memory management!
I'm sorry to differ but no it has not. Apart from "used" and "free", it has not.
I will even go further: even these 2 categories are not generalizable. Look at how Darwin computes "used" memory. It counts inactive pages, i.e. allocations that have been freed and are literally no longer used by any program, in the used set!
@PierreMarieBaty You're over-engineering things. And there's one problem with your approach that, color scheme designers (theme designers) are losing the capability of customizing colors for the memory fields. That's why I said earlier this was a bad idea.
Nope. The DYNAMIC_xxxx colors are themable - and already themed. Look in CRT.c, it's already done. Even on black/white displays, they fallback to different ASCII characters. I tested it on all the OSes on which I could compile.
I bet you didn't have the idea on where the ASCII characters drawn on the bar meter are defined.
Unlike GPU meter where items are by driver (or, in the user's perspective, GPU vendor), memory meter has common categories that are best colored consistently across platforms.
I'm sorry to differ but no it has not. Apart from "used" and "free", it has not. I can only advise you to have a dive in it yourself: install these operating systems in a VM, look at what "top" displays, look at their ways to extract memory information, so as to understand how they do thing. I have the strong feeling that you are under the illusion that every OS must be doing it in a roughly similar way than what Windows and Linux do. Nothing could be more wrong...
You said "used" and "free". I take that as your admission that there are common categories. Yes you can make subcategories under "used", e.g. kernel pages vs. process pages, and some pages are even shared across processed. But objectively speaking they are all "used" categories.
Even we can tune the wording for each category by platform, let's not mess up with the color scheme or the ordering of the categories (e.g. "cached" memory categories should be after the "used" memory categories).
No again, because "cached" memory will have a totally different meaning across 3 different systems! On some OS it's reclaimable, on some others it's not. On some OS it's optional, on some others it's mandatory. On some it caches filesystem metadata only, on some others it caches buffers, on some others it's part of a unified cache, on some others it's ONE of the various caching mechanisms, on some others it's part of a memory zone... I really don't want to sound offensive, but you need to get more understanding on how differently the operating systems grasp memory management!
The best thing we should do is to examine why the "show cached memory" option was introduced in the first place.
From the htop FAQ:
The memory meter in htop says a low number, such as 9%, when top shows something like 90%! (Or: the MEM% number is low, but the bar looks almost full. What's going on?)
The number showed by the memory meter is the total memory used by processes. The additional available memory is used by the Linux kernel for buffering and disk cache, so in total almost the entire memory is in use by the kernel.
Buffers and Cache memory in the Linux kernel are reclaimable and not actually "unavailable" for reallocating to processes. It was a clever mechanism for Linux to prefetch data and put those memory space to use or otherwise it would be waste. Since then we had bug reports requesting those two fields be hidden in Graph mode. So the option wasn't actually about "cache", but "cache memory that is reclaimable on demand".
With this purpose in mind, "cache" memory in operating systems that is not reclaimable shouldn't be covered under the toggle option. Note that I don't care what kind of "cache" the kernels are going to use for, what matters is whether the memory is reclaimable or not. That is, when a new process starts, whether the kernel can flush those cache pages immediately and reallocate them to processes that need them.
Buffers and Cache memory in the Linux kernel are reclaimable and not actually "unavailable" for reallocating to processes. It was a clever mechanism for Linux to prefetch data and put those memory space to use or otherwise it would be waste. Since then we had bug reports requesting those two fields be hidden in Graph mode. So the option wasn't actually about "cache", but "cache memory that is reclaimable on demand".
With this purpose in mind, "cache" memory in operating systems that is not reclaimable shouldn't be covered under the toggle option. Note that I don't care what kind of "cache" the kernels are going to use for, what matters is whether the memory is reclaimable or not. That is, when a new process starts, whether the kernel can flush those cache pages immediately and reallocate them to processes that need them.
That's exactly what my new implementation does. The "countsAsCache" member of the MemoryClass struct serves this purpose. And about why some of these caches are displayed before active pages and not regrouped together, that is because sometimes (read: often), these caches are subsets of other OS-specific memory classes, and it makes more sense to display them together. The platform-specific memory collection code merges the relevant categories together when the showCachedMemory option is toggled off.
I said earlier that my changes fixed other issues. Here's one. Some other times, the number of cached pages obtained through a sysctl or a VMM stats struct is in fact spanned over several categories. In that case, it's not relevant to display these pages at all in the graph, because they don't belong to the same "view" of the physical memory, and there's no acceptably lightweight way to obtain this information. The memory bar should display well-defined categories of physical pages, not mix these with sets that span over multiple categories when there's no way to know where they came from. Doing so like it was done could in theory make memory bars go beyond 100%.
I bet you didn't have the idea on where the ASCII characters drawn on the bar meter are defined.
Well, ncurses?
Anyway. Should I bother completing that work? If it's going to be rejected, just say so and I'd better stop here.
Buffers and Cache memory in the Linux kernel are reclaimable and not actually "unavailable" for reallocating to processes. It was a clever mechanism for Linux to prefetch data and put those memory space to use or otherwise it would be waste. Since then we had bug reports requesting those two fields be hidden in Graph mode. So the option wasn't actually about "cache", but "cache memory that is reclaimable on demand". With this purpose in mind, "cache" memory in operating systems that is not reclaimable shouldn't be covered under the toggle option. Note that I don't care what kind of "cache" the kernels are going to use for, what matters is whether the memory is reclaimable or not. That is, when a new process starts, whether the kernel can flush those cache pages immediately and reallocate them to processes that need them.
That's exactly what my new implementation does. The "countsAsCache" member of the MemoryClass struct serves this purpose. And about why some of these caches are displayed before active pages and not regrouped together, that is because sometimes (read: often), these caches are subsets of other OS-specific memory classes, and it makes more sense to display them together. The platform-specific memory collection code merges the relevant categories together when the showCachedMemory option is toggled off.
I believe countsAsUsed and countsAsCache should be designed to be mutually exclusive. That is, not both true at the same time.
And note the FAQ again. The "used" memory number never adds the reclaimable cache memory. htop's behavior already differs from Linux's top in this regard. Trying the make the memory number display close to the operating system's top would not be part of htop's goal.
I bet you didn't have the idea on where the ASCII characters drawn on the bar meter are defined.
Well, ncurses?
Look at the Help screen (F1) to see the character set.
I believe
countsAsUsedandcountsAsCacheshould be designed to be mutually exclusive. That is, not both true at the same time.
That's what I thought first, and I tried hard to stick to that dogma, but it was just impossible. The way the official tools of various OSes count "used" memory is simply too different. Again, some OSes officially count these reclaimable caches as "used" memory.
If htop's goal is to coerce everybody to interpret memory information "à la Linux", then I find that stance pretty disappointing and I wouldn't call this a "properly ported" tool. The #1 use of htop is to monitor resource pressure over time. If your server gets OOM at some point and a catastrophe happens while htop says it has plenty of RAM left, what's the point of the tool?
I believe
countsAsUsedandcountsAsCacheshould be designed to be mutually exclusive. That is, not both true at the same time.That's what I thought first, and I tried hard to stick to that dogma, but it was just impossible. The way the official tools of various OSes count "used" memory is simply too different. Again, some OSes officially count these reclaimable caches as "used" memory.
If htop's goal is to coerce everybody to interpret memory information "à la Linux", then I find that stance pretty disappointing and I wouldn't call this a "properly ported" tool.
No. It wasn't about Linux. It was simply the fact that matching the display of the operating system's top is never part of htop's goal!
If an OS counts some "buffer" memory as parts of the "used" memory, then for htop's purpose the numbers have to be split. There's no grouping of memory numbers intentionally because the numbers have to fill up a bar or graph. If we said "Used: 10.0G" that includes (i.e. overlaps with) "Buffers: 100M" that would be difficult to draw on the bar. For this particular reason no overlapping would be allowed. Each memory category has to be separate.