OS provider should support various binding modes

[bratpiorka]

The operating system provider should support different binding modes. All these combinations should be tested and documented (animations like in memkind blog?) so that users can easily choose between them. There should be information in the documentation if a particular mode is not supported by a particular operating system.

• [x] UMF_NUMA_MODE_DEFAULT - The default binding mode. The actual binding policy is system-specific. OS provider should not call mbind() in this mode. If this mode is specified, the nodemask must be NULL and the maxnode must be 0.
• [x] UMF_NUMA_MODE_BIND - Restricts memory allocation to nodes specified in nodemask. Allocations might come from any of the allowed nodes. The nodemask must specify at least one node and the maxnode should be set to the highest ID of the node in the nodemask. If all nodes specified by nodemask are out of memory and there is no space left in the swap, the allocation returns NULL and the native error is set to UMF_OS_RESULT_ERROR_ALLOC_FAILED.
• [x] UMF_NUMA_MODE_INTERLEAVE - This mode interleaves memory allocations across the set of nodes specified in nodemask at a page-level granularity. The nodemask must specify at least one node and the maxnode should be set to the highest ID of the node in the nodemask. If all nodes specified by nodemask are out of memory and there is no space left in the swap, the allocation returns NULL and the native error is set to UMF_OS_RESULT_ERROR_ALLOC_FAILED
• [x] UMF_NUMA_MODE_PREFERRED - the nodemask must specify the single, preferred node for allocations. If allocation cannot be fulfilled and there is no space left in any node specified by the nodemask and there is no space left on the swap, the memory will be allocated from other nodes. If all available nodes in the system are out of memory and there is no space left in the swap, the allocation call returns NULL, and the native error is set to UMF_OS_RESULT_ERROR_ALLOC_FAILED
• [ ] UMF_NUMA_MODE_LOCAL - the memory is allocated on the CPU node that triggered the allocation. If this mode is specified, nodemask must be NULL and maxnode must be 0. If this node is out of memory and there is no space left in the swap, the memory will be allocated from other nodes

[bratpiorka]

@DamianDuy please look at this list - the multiple - NUMA tests should follow this

[igchor]

As far as I know, I think the only mode support on Windows would be UMF_NUMA_MODE_PREFERRED (see https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualallocexnuma).

Also, perhaps we should just introduce a generic UMF_RESULT_ERROR_OUT_OF_PROVIDER_MEMORY and use it instead of the native `UMF_OS_RESULT_ERROR_ALLOC_FAILED? I think it will be needed for all providers anyway.

[lukaszstolarczuk]

As Igor stated (here) we should test falling back to other nodes and failed allocations. Go to the link for more details.

[lukaszstolarczuk]

one other thing here is that older hwloc (e.g. on 2.4.1 with kernel 5.14.0) does not support PREFERRED_MANY mode (I've seen this mode is for sure in kernel from ver. 5.15). So we either enforce newer hwloc or we describe differences in our docs (preferred vs preferred_many).