VirtualMemoryStat.UsedPercent misleading

[mwmahlberg]

For example, on on OS X, here is some sample output

Total: 4294967296 Free: 16121856 Used: 4278845440 Utilization: 83

Whereas it should be 99%. The reason for this is that instead of using only Total, and Used, Total is reduced by Available, which includes Cached.

So either it should be explicitly documented that the utilization is reduced by the potentially available Cached memory or – and this is what I suggest – the utilization should show the current state of the memory consumption as follows:

ret.UsedPercent = float64(ret.Used) * (float64(100)/float64(ret.Total))

Please let me know which road to take, I'll be happy to submit an according pull request.

[upccup]

@mwmahlberg i think is will be like

MemUsed := v.Used - v.Buffers - v.Cached

[mwmahlberg]

@upccup Which still would be misleading. The notion of used should be what actually is used at a given point in time. Reducing by buffers and cached actually shows how much would be available, right?

Available := Total - Used + Buffers + Cached

Unless I am getting something wrong, in which case I'd love to be enlightened. ;)

[upccup]

@mwmahlberg You are right. But this is a UNIX system's memory management policies. Only when in use will be released before the memory program. I do not know if I say that will not make you more confused

[mwmahlberg]

@upccup I investigated this a bit further. The thing is that "available" is just an estimate wether enough memory is available for starting an application, taking into account potentially freeable memory. So, as a value, there is nothing wrong with Available. As a base value for RAM utilization, this still seems wrong to me. At least it should be noted.

[walles]

@mwmahlberg, whether to count caches as used or not depends on the use case. What does the requester want to know?

If they want to know how much memory is left before the system starts swapping, caches should be counted as free / available.

If they want to know what percentage of their RAM chips are actually used for storing bits, caches should be counted as used.

I'd argue most people want the former.

Either way, I agree that having documentation for what Used and UsedPercent actually means would be good: https://github.com/shirou/gopsutil/blob/master/mem/mem.go#L10

[walles]

I've started work to clarify this in a branch: https://github.com/shirou/gopsutil/compare/master...walles:walles/vmtest?expand=1

At the time of writing this comment, the struct has been updated with documentation and I've written tests for the (now) documented behavior.

I haven't updated the implementations on any platform though, so it's not ready to be a PR yet.

[sparrc]

I don't agree with changing the current implementation. Used memory is a documented linux kernel term: https://www.centos.org/docs/5/html/5.2/Deployment_Guide/s2-proc-meminfo.html in that it is the opposite of MemFree, which is exactly how gopsutil calculates it:

MemFree — The amount of physical RAM, in kilobytes, left unused by the system.

[walles]

@sparrc, do you have a reference for where Used is documented as a linux kernel term?

[sparrc]

Used is not explicitly documented but MemFree is the "unused" space, which to me means that the used space should just be the opposite of that.

[sparrc]

which at least for Linux, is how it is currently calculated: https://github.com/shirou/gopsutil/blob/master/mem/mem_linux.go#L54

[mwmahlberg]

Would be interesting to dig how free et al calulcate free vs used RAM. I am bound to at least three projects atm, but can investigate after that. @walles As per available RAM, I agree with you. That's what one wants to know: "Can I run something that needs X bytes?" As per RAM utilization however, for example for warning levels, the physical memory utilized seems a lot more reasonable.

[walles]

@mwmahlberg, free on Linux does used = total - free - buffers - cache.

I suggest we:

• use kernel specific terms (Buffers, Cached, Wired etc) for kernel specific values.
• use other terminology for human consumable values (Total, Available, Used)

Can you describe a scenario where you have actually wanted to warn on how the kernel decided to use Available memory? This sounds like an edge case to me, how would you feel about doing Total - Free (both well documented and available on all platforms) to get this value on any platform?

[rogercoll]

Old issue but I was doing some tests on the used value and I not am sure if its formula is the most suitable, or if it should be relied when setting memory alerts.

Small function:

func gops(memory mem.VirtualMemoryStat) {
	fmt.Println(memory.Total, " Total memory bytes")
	fmt.Println(memory.Available, " Available memory bytes")
	fmt.Println(memory.Used, " Used memory bytes")
	fmt.Println(memory.Total-memory.Available, " Total - Available memory bytes (used v2)")
}

//Output (Rocky linux)
8346812416  Total memory bytes
7641399296  Available memory bytes
323620864   Used memory bytes
705413120   Total - Available memory bytes (not available memory)

I was expecting the latest two values to be very similar, but instead one is less than half the other. The not available memory is much higher than the used one. If some alerting systems relies on used to trigger high usages of memory, its alerts might never trigger.

Would it make sense to move the Used calculation to a more "sensitive" one?

I made some memory load testing, this are the results (~8% of difference):

cc @walles @mwmahlberg

[mwmahlberg]

@rogercoll As said, cached memory needs to be taken into consideration. However, utilization should denote the percentage of physical memory used, imho.