cpuallocator: implement package prioritization

[marquiz]

Try allocate all cpus from a single package, if possible. For each allocation request calculate priority order between cpu packages, based on requested number of cpus and the cpu priority preference.

[kad]

@marquiz Package is not enough. we need to check whole hierarchy: cluster->die->package

[marquiz]

@marquiz Package is not enough. we need to check whole hierarchy: cluster->die->package

I know but I would leave that as a future improvement. We can discuss that in #191.

This PR is supposed to fix the greatest shortcoming in the current cpuallocator.

[askervin]

This change breaks a couple of e2e tests. I'm verifying if the new allocation order makes sense in these cases. For instance policies.test-suite/podpools/n4c16/test01-basic-placement

(Thanks for very good debug prints. ;) )

[askervin]

@marquiz, suggesting changes/fixes.

Make sure a.topology.cpuPriorities is non-empty. It seems like it would be empty on any platform without this:

diff --git a/pkg/cpuallocator/allocator.go b/pkg/cpuallocator/allocator.go
index 4f9b74e0..daf71b4a 100644
--- a/pkg/cpuallocator/allocator.go
+++ b/pkg/cpuallocator/allocator.go
@@ -527,6 +527,7 @@ func (c *topologyCache) discoverCPUPriorities(sys sysfs.System) {
                        prio[p] = prio[p].Union(cset)
                }
        }
+       c.cpuPriorities = prio
 }

If the platform doesn't have cpufreq data, do you think all CPUs could be considered of Normal priority? Like adding following to discoverSstCPUPriority() (near // Finally, determine priority of each CPU):

if len(eppList) == 0 {
        if cpus, ok := epps[sysfs.EPPUnknown]; ok {
                prios[PriorityNormal] = cpus
        }
}

(Thanks to @klihub for spotting this!)