petabridge
Build more complete hardware profile in NBench reports
Ran into a fun issue on DotNetty https://github.com/Azure/DotNetty/pull/95 where it looked like a change in the underlying hardware we're running NBench on affected the benchmark significantly with no explanation, thanks to our use of auto-scaling build agents on Windows Azure.
We really need to expand the hardware profile captured by SystemInfo and include much more detail about:
- The clock speeds of all CPUs, and model information if possible
- Total amount of available memory and any other details we can get about it
- BIOS information, data about the underlying hyper visor if there is one
- And more that I'm probably forgetting.
That way it'll be easier to tell if it was a code change or a hardware change that resulted in a divergence in a metric, and not cause the developers to panic when it's the latter.
If you're open to a platform-specific option, WMI may yield enough info. An example from PowerShell: Get-WmiObject -Class Win32_Processor | fl or Get-WmiObject -Class Win32_Processor | Get-Member Some of the available data:
- CurrentClockSpeed
- L2CacheSize
- L2CacheSpeed
- L3CacheSize
- L3CacheSpeed
- NumberOfCores
- NumberOfEnabledCore
- NumberOfLogicalProcessors
- Stepping, etc.
We can get WMI data via Microsoft.Management.Infrastructure, if you're open to that...
@antondelsink I'm open to platform-specific tooling here, although here's the catch: we have to make it switchable underneath so the core NBench library can appear to be platform-agnostic. This is an absolute must-have for supporting .NET Core in the future.
So here's what I propose: we create a provider model for SysInfo which will lazily load one of three possible SysInfo concrete implementations:
- The generalized, "safe" default one we have now;
- A Windows-specific WMI provider; and
- A to-be-announced Linux / Posix-specific provider.
Provider 1 is included in the default NBench library, and it's always the fallback in the event that implementations 2 or 3 fail to load or aren't available.
Implementations 2 and 3 will be implemented in external DLLs, so NBench.SysInfo.Windows and NBench.SysInfo.Linux perhaps.
At runtime, we'll detect the OS and Assembly.Load the appropriate implementation and then that that SysInfo object as the data provider going forward. If any part of that operation fails, fall back to Provider 1 and continue.
We'll also need to expand the fields on the SysInfo object to include this additional data too. Maybe it's better to just have a dictionary of properties we can populate rather than a bunch of static fields.
What do you think @antondelsink ?
Assembly.Load would work... also consider a Conditional on the reference to the library NBench.SysInfo.Windows so the decision is made at time of build, perhaps with the default build providing only the "safe" default SysInfo.
I'll put together a naïve NBench.SysInfo.Windows.WMI class in a new project under the same solution, but @Aaronontheweb tell me more about how you would like to load it. Perhaps from the current SysInfo.CreateInstance?
@Aaronontheweb: take a look at #109; I've also added:
- Get-WmiObject -Class Win32_PhysicalMemory
- Get-WmiObject -Class Win32_BIOS