docker-tools
docker-tools copied to clipboard
dotnet
Automate testing on internal Windows builds
New builds of Windows are produced on a regular basis. We should setup automation to respond to the availability of a new Windows image by building and testing our Windows-based images for .NET Core and .NET Fx.
This is referring to the servicing builds for existing releases of Windows.
This isn't trivial to do for .NET FX because we would need to modify the Dockerfiles to install the LCU.
@mthalman - Can describe the benefit this would provide to us? Can you also provide a strawman plan of the necessary changes needed to support building/testing these images?
@MichaelSimons - Can you find any past issues that this type of testing would have caught?
The one time I can think of that this would have caught an issue would have been Dec 2018 servicing in the .NET Fx repo. There was a regression in Windows Server Core where applying the LCU failed after installing FoD (Feature-on-Demand) packages. The Windows team thought they had this scenario covered in their tests but there were subtle differences in how they were testing. A fix wasn't made available until the next patch cycle. The two questions I have are.
- Did windows address their testing to cover this scenario? A broader question at hand is how well are they testing their base images.
- Even if we caught this issue before the release, would it have held the release or would the issue have been fixed?
This kind of testing would have also helped to identify the 2B container on 1B host issue that recently occurred. Identifying issues like this early on would have a huge impact, hopefully preventing breaks like this from ever being publicly published and reaching customers.
At this point in time, we've setup a process to do manual pre-release testing of .NET Framework images using internally available patched updates for existing versions of Windows as well as pre-releases of new Windows versions.
For .NET Core, there is also a testing process to do manual testing of pre-release Windows versions.
The remaining work is to automate the process of testing .NET Core images on internally available patched updates for existing versions of Windows. So this issue will remain open for that work.
This work is required in order to support the full implementation of internal .NET builds, specifically for COSMIC.