Please add support for Hyper-V platform

[mlacko64]

What I'd like: Please add support for Hyper-V platform. Currently when I start VM Bottlerocket (as part of EKS-Anywhere) it freezes in very early boot stage. For example Ubuntu VM for EKS Anywhere works fine, Thank you.

Any alternatives you've considered:

[abhinavmpandey08]

Hi @mlacko64, EKS Anywhere maintainer here. Just curios, what EKS-A provider are you using with Hyper-V since EKS-A doesn't natively support Hyper-V? Also, if this is a feature request for EKS-A specifically, mind opening this issue at https://github.com/aws/eks-anywhere?

[mlacko64]

Hello @abhinavmpandey08 , I am using --provider tinkerbell bare metal installation. If I choose osFamily: bottlerocket installation will fail, because seems Bottlerocket does not include some Hyper-V tools or drivers to work on it. But it works fine with Ubuntu.

I can of course open a request to EKS-A repo, but I was thinking about running baremetal installation on Hyper-V (so without any special features provided by Hyper-V), that's why I opened a request here. I think, if Bottlerocket itself does include support for Hyper-V platform (drivers or something like that), it could then run as a baremetal.

I can run that way for example baremetal installation of OpenShift using their CoreOS image, which is like Bottlerocket optimized for container run. Yes, it is "just" baremetal (no Hyper-V features), but it is still great to test deployment, do development etc. And real production can be then deployed on real HW.

[mlacko64]

I have opened also request to EKS-A , see here. Still, if even baremetal deployment with Bottlerocket will be possible, it will be great way to test baremetal deployment.

[mlacko64]

Boot on Hyper-V freezes here, seems because it is not able to mount root.

Similarly same error produces VMWare workstation , when VM has SCSI disk attached. When I set disk to SATA , it works.

I am using this image (from EKS-Anywhere baremetal): OS Image: Bottlerocket OS 1.14.0 (metal-k8s-1.27)

So it seems some driver for virtual SCSI is maybe missing, which quite makes sense, if baremetal should be strictly limited to physical HW (at least that was reply from EKS-A).

[stmcginnis]

Thanks for posting the request @mlacko64. Hopefully it is just a matter of including another driver for virtual SCSI, but that may require some testing to validate if that's the only missing piece.

[stockholmux]

@mlacko64 ~There is a meetup next Wednesday (Aug 30). A few folks on the team were discussing this enhancement request.~ Would you be able to join the meetup and chat about your use case? It'd be great to learn more.

Looks like we missed you @mlacko64. Let us know if you can attend on Sept 13.

[mlacko64]

Hello, I was on vaciation, I missed your comment. Anyway, it seems it had bit late timing in my timezone.

Still, to understand what I am trying to achieve:

• we plan (deployment in progress) run production EKS-Anywhere clusters on physical servers (migrate from different kubernetes solution)
• I am currently testing settings and application on dev cluster which is running on VMWare (it is baremetal running on VMWare, not native VMWare, so I have same environment as will be in production)
• why I wanted Hyper-V support - my VMWare cluster is far from ideal, but have plenty of available Hyper-V resources to run dev cluster there
• as my goal is to emulate baremetal, I was not seeking native Hyper-V features support in EKS-Anywhere (like vSphere has for example)
• I talked already with EKS-Anywhere support, but they want to limit baremetal to physical HW only, so no support for Hyper-V or any other virtualization features on baremetal

Perhaps just recompiling Bottlerocket with Hyper-V kernel options , same like Ubuntu has for example, could be a way how to achieve it, but I did not tested it yet.

[stockholmux]

@mlacko64 Thanks for the information! Door is still open if you want to chat about this on Sept 13.

[chrismade]

As this case is not closed yet ... I would like to add an observation which might be relevant - I experienced the same problem that dm0 is waiting forever to mount root (or even worse directly reboot the system which makes troubleshooting much more difficult) when the underlaying disk is a standard /dev/sdX e.g. /dev/sda type - it seems to me that the drivers for /dev/sdX are somehow included in the kernel but these are no good for booting / mounting the root filesystem via dm-xx - if you change in KVM to a virtio disk /dev/vdX then it boots just fine and I guess this is exactly what happens every day in AWS EKS - I propose to rename this request to "allow BootlerocketOS dm-verity to boot from standard /dev/sdX devices as well" which would widely expand the HW range on which BootlerocketOS can actually boot from baremetal to Hyper-V massively and deliver on the EKS anywhere promise.

I also tried to add this support in the kernel myself - or to find a kernel boot-parameter which does something like mapping /dev/vda to /dev/sda to overcome this issue but I'm not smart enough to get any working solution

Repro instructions: just take any motherboard with a SATA disk and copy an uncompress bootlerocket image onto the disk - or dd this onto a USB Stick - an then go into the boot manager and boot from this device you had copied the image - you will see it booting until dm-verity is trying to mount the root filesystem - it is either waiting for ever or rebooting which is the situation described here

[chrismade]

sorry for another post - I checked again the screens provided by @mlacko64 on Aug 7, 2023 - it seems to be evident that it is a device-mapper issue - as said in previous post drivers for /dev/sdX ad /dev/vdX are both in the kernel but the device mapper config seem to set in a way that the needed verity device 0:252 can only be made from /dev/vdX devices - not from /dev/sdX devices - this should be easily fixable by changing the device-mapper config - or udev rules if used