Create a smoke test for installers

[smlambert]

Would like to design/create a smoke test consisting of several basic checks to be run or triggered at the end of the installer jobs to verify the installers are sane.

Verify each set of installers has some testing in place and documented:

• [ ] Linux installers - x64 linux installers are tested, need https://github.com/adoptium/installer/issues/375
• [ ] Windows installers - https://github.com/adoptium/installer/issues/559
• [ ] Mac installers

Related to https://github.com/AdoptOpenJDK/TSC/issues/151 and https://github.com/AdoptOpenJDK/openjdk-tests/issues/1421

[douph1]

Did we speak about automated/scripted smoke test here ? or just a manual check-list ?

[smlambert]

Automated tests based on TestContainers.

There is a revamped way of building and testing installers based on the what @aahlenst has in https://github.com/aahlenst/adoptium-packages-linux.

The intention is to (eventually) shift to this new approach for building Linux packages. We will move that repository into the adoptium github organization, and add a Jenkinsfile to trigger build and test of these linux packages from Jenkins build pipelines. You can see that in this repo there are tests for the installers (based on TestContainers). Those tests would be definitely sufficient to address this issue.

[douph1]

Ok, I was speeking about the MSI installer :) for wich we have no test beside the Wix smoke.exe

Smoke | Runs validation checks on .msi or .msm files.

I don't know well testcontainers but it seems not usable to validate MSI on Windows container

https://www.testcontainers.org/supported_docker_environment/windows/ Windows Container on Windows (WCOW) WCOW is currently not supported, since Testcontainers uses auxiliary Linux containers for certain tasks and Docker for >Windows does not support hybrid engine mode at the time of writing.

And as MSI install depends on a Windows service "Windows Installer" I think it will probably not be a good candidate for containerization

PS: we don't have Jenkinsfile either now for the msi installer

[aahlenst]

There was some experimentation to make TestContainers work with that thing that's built into Windows Server 2016 and newer. Theoretically, it should work if you disable the auxiliary containers (we have to do the same on Linux because not every arcane architecture is currently being supported by TestContainers). I'd be surprised if it's not possible to install an MSI on a Windows container, but considering that the container support in Windows is a strange beast 🤷

There are two possible alternatives: Using docker commands directly (was briefly considered for Linux; so basically issuing shell commands with docker exec against the running container and evaluating the output) or using Vagrant with VMs (very slow because you have to start a new VM for every test to have a reliable environment, especially registry).

The biggest challenge is going to be macOS because macOS doesn't like to be automated or virtualized. I see two workable solutions: Use Vagrant + Parallels Desktop Pro (the edition that supports linked clones for fast startup) or try https://veertu.com/anka-build/.

I wanted to commission some projects to get this done for all platforms. But there's zero interest. I find that topic interesting, but it needs organizational support that doesn't exist either. So I gave up.

[smlambert]

FYI @sophia-guo

[aahlenst]

Boilerplate to build upon for a test with a container on Windows:

$container = docker run --rm -d -t mcr.microsoft.com/windows/servercore:ltsc2019
docker cp "C:\Users\Administrator\Downloads\OpenJDK11U-jdk_x64_windows_hotspot_11.0.11_9.msi" "${container}:C:\Users\Administrator\OpenJDK11U-jdk_x64_windows_hotspot_11.0.11_9.msi"
docker exec -it $container powershell -command "Start-Process C:\Windows\System32\msiexec.exe -ArgumentList @('/i C:\Users\Administrator\OpenJDK11U-jdk_x64_windows_hotspot_11.0.11_9.msi', 'INSTALLLEVEL=1') -Wait"
docker exec -it $container powershell -command "java -version"
docker kill $container

Those 5 lines copy a local MSI into a container (for example, generated by the build process), install it (you have to wait for the installation to complete, therefore Start-Process instead of invoking msiexec directly), run java -version and remove the container afterwards. That follows what the tests for the Linux packages do, too. Instead of running java -version, you can do other things (like querying the registry) and inspect the output to stdout. I'd write a Java class that wraps the Docker commands with ProcessBuilder and forget about TestContainers (possibly for Linux, too). Because:

• We don't need most of the functionality that TestContainers brings to the table (like connecting to MySQL, for example) or having fancy wait commands.
• By having to disable all auxiliary containers, we're not better off than if we run everything ourselves. If the JVM gets killed, a container might stay around. TestContainers fixes this by having an auxiliary container (Ryuk) that removes stale containers. If we disable Ryuk, we're back on square one.
• TestContainers has some problems I discovered while developing tests, for example weird wrapping of stdout or not being able to copy moderately sized files without huge amounts of RAM. docker cp is significantly more efficient. This could be fixed, but seems like a lot of work.
• TestContainers brings still some dependencies on native code AFAIK. Considering the state of the Windows on ARM ecosystem, I wouldn't hold my breath to wait for everything to get ready. This is also the reason I'd use Java instead of my usual msbuild/C#/VS Studio combo when working with MSIs because there's no VS Studio yet for Windows on ARM as far as I know and testing needs to happen on the target platform.
• Being not dependant on TestContainers on Linux might be an advantage, too, especially considering the recent push for podman on some distributions where it's hard to get Docker to work (podman is still no drop-in replacement for Docker when it comes to TestContainers).

One thing I noticed while working with Docker on Windows: It seems slow (might be caused by that machine I used that was running on OpenStack). Using Vagrant with a performant virtual machine manager might be as fast. There's even a Hyper-V plug-in for Vagrant. The big advantage of using Vagrant: We'd have a UI. This would allow to examine the graphical installer which might be of interest. But there's currently no Vagrant for Windows/ARM. Compiling one might be doable, however.