hwloc
hwloc copied to clipboard
open-mpi
report available memory
From Daniel Gruber:
Do you plan to support an easy way to get the use memory per NUMA region like "numactl --hardware" does on Linux? Do you see any way to accomplish that?
Shouldn't be hard on Linux. Already done on Windows (see #134 )
By the way, we actually already report the free hugepages instead of the total ones on Linux.
Would need a new field in the memory structure (or a topology flag saying that it replaces total memory).
On Solaris, we can use LGRP_MEM_SZ_FREE instead of LGRP_MEM_SZ_INSTALLED.
Imported from trac issue 60. Created by bgoglin on 2011-12-16T05:36:08, last modified: 2012-05-08T15:59:49
Trac comment by sthibaul on 2011-12-17 15:43:42:
(or a topology flag saying that it replaces total memory).
I'm strongly against such hack when it can be avoided :)
Would need a new field in the memory structure
This is way preferrable.
We patched the sources (patch is for 1.11-branch) to support MemFree as well, by extending the memory structure. Note, this patch is Linux-only (and should probably be considered very hacky):
Any updates on getting this into the codebase? Could try to update the patch for master and maybe open a PR
Your patch breaks the ABI by inserting a field in the middle of the memory structure. We could have applied it before releasing 2.0 at the end of 2017, but your comment came a couple months too later unfortunately. Now we would have to put the new attribute at the end of structure (which is now hwloc_numanode_attr_s instead of hwloc_obj_memory_s in master).
But there's also a question of whether exposing this information statically is a good idea. The available memory will likely decrease between when the topology is loaded (early, with possibly lots of free memory) and when the applications looks at it (later, with possibly much less available since the application and runtimes started working). However hwloc currently has no way to update such information dynamically. A query interface that returns the currently available free memory might be better (but it needs to handle each individual NUMA nodes, or return an array, or?).
Yes you are right. I just now realized that we currently re-initialize hwloc every time we want to read this value, thus getting up to date values. When using hwloc in a saner way, these values would indeed quickly be wrong (out of date).
A query-interface would indeed be the way to go. As you said, either by individual node or in an array.
On a side note: I was about to propose enhancing this even further (not only MemFree, but also Buffers and Cache). But I just now realized that the /sys/devices/system/nodes-interface might be lacking:
/sys/devices/system/node/*/meminfo contains MemTotal, MemFree and MemUsed
/proc/meminfo contains MemTotal, MemFree, MemAvailable, Buffers, Cached => So there is no MemUsed, but "Buffers" and "Cached" instead.
/sys/devices/ seems to simply calculate MemTotal - MemFree = MemUsed, which does not take Buffers into account.
So the parsed MemFree from my patch is not really correct either, since caches could also be used (if the OS drops them). But I have found no way to get this kind of information about individual NUMA-nodes (only /proc/meminfo). Do you?
For the record, there's a proposal to expose available memory through the memory attributes, because those may look more dynamic (we can update them on application query).