commando-vm
commando-vm copied to clipboard
mandiant
Feature Request: add support for KVM hypervisor
Describe the bug and expected behavior
During installation, after the first reboot the system will not boot anymore (black screen screen).
Upon force-resetting the VM multiple times it finally boots into Windows recovery but no recovery methods so far have worked. I think the bootloader is toast.
To Reproduce Steps to reproduce the behavior: (not sure if this will reproduce the issue; this is what I did)
- Install libvirt and virt-manager
- Create a new Windows VM.
- Add a dummy virtio network and a dummy virtio disk (for installing drivers)
- Install Windows 10 Pro on the VM
- Install the virtio drivers from Redhat
- Download install.ps1
set-executionpolicy unrestricted -f.\install.ps1
Screenshots It's just a black screen after first reboot.
Version
- GuestOS: Windows 10 Pro x64
- HostOS: ArchLinux (up to date)
Additional context
I have also tried to deactivate the installation of Hyper-V by commenting out Install-HyperV in the file commando-vm/commandovm.win10.preconfig.fireeye/tools/Default.preset with no success.
EDIT: Upon retrying and paying close attention to the installation, my attempt to deactivate the Hyper-V installation failed, it is still installing it. How can I deactivate that?
The virtualization is done via libvirt, which uses qemu kvm.
I would like to not install Hyper-V in order to verify whether or not this is part of the problem. So I downloaded the entire repository and ran the install.ps1 with a custom profile and modified the Default.preset. But it seems to download the template and not use my locally modified version. How do I make it use my local version?
UPDATE: I managed to hack my way around the limitations and can now confirm that the installation of HyperV via the default win10.preconfig package does break the machine. However I was unable to successfully install commando completely due to my lack of knowledge (and time to investigate) the install script.
It would be nice if the default packages detected being run on a KVM based VM and would then skip the installation of HyperV.
One example to make the detection work would be checking the "System Manufacturer" in the System Information, which is set to QEMU in my case. Although an environment variable set by the user prior to installation would probably be easier and more robust.
Very interesting. I can help you work through the rest of the install issues but we have not tested/tried install using RedHat as the base OS or with virt manager so I will need more information on what you are seeing. I can attempt to place logic in there to attempt detection of the virtual environment but it will be difficult. It also may not be super reliable, I tried some of this when attempting Docker installs but it was not fruitful.
We have plans to release multiple profiles for the VM, a lite version, one with no nested virtualization, and a full version. Perhaps when we do that we can look closer at this issue.
By now I managed to install it with some more hacking of the install script.
As for what I see... nothing. :see_no_evil: During installation one of the last things I see is "Installing Hyper-V", then it automatically reboots into a 100% unresponsive black screen. One CPU is at 100% but nothing happens.
Reverting my previous guess: I think the bootloader is still alive, after all it does boot me into recovery mode after a few forced reboots. I think that installing Hyper-V inside a KVM guest just activates some drivers or some part of the OS that doesn't work inside the VM.
Btw. I didn't test this but while deactivating Hyper-V installation I also deactivated any Docker related stuff preemptively, as I don't need it and it might cause similar issues.
For reproducing you don't necessarily need RedHat. You just need a Linux distro that has packages for libvirt/virt-manager and qemu (which is most I believe; unless you are fine with compiling it yourself). My Host OS is ArchLinux but I know for certain that CentOS and Ubuntu have packages for it.
Here is the ubuntu libvirt wiki page. virt-manager is optional, as it is just a management GUI for libvirt. You can also use virsh, which is CLI based.
I've attempted to install this twice, both times - on first reboot "updates" reach 93-94 % and the VM freezes. Stop/Start - boot into windows - the installer resumes - full of errors. After reboot again - VM has constant 50% CPU utilization on all cores (observed in PVE), totally unresponsive.
Host: Proxmox 6.1 Guest: Windows 10 1909 x64, 8G ram, 8 "host" cores (48 cores available)
root@proxmox:~# kvm -version QEMU emulator version 4.1.1 (pve-qemu-kvm_4.1.1) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
Why?
Just looking for similarities between our systems. I use KVM too, which I assume to be part of the problem. See above.
Something that would help you guys in the mean time, you can edit the install script and remove the preconfig package (currently on line 360 and line 383 ).
The preconfig is a copy of the win10-initial-setup-script with edits such as disabling defender, firewall, llmnr, installing hyper-v, etc.
After removing this from the install script, I would recommend downloading it and running it manually before running the Commando install again. Certain things like Defender need to be disabled before install otherwise a lot of the tools will be cleaned off of the system. You can use our Default.preset file as a reference for the configuration changes we make. You will need to edit the Default.preset file that comes with the repo (do not re-use our Default.preset file with the current repo as a lot of the module names have changed), as the preset file and install script we use is currently quite a few commits behind (will be updated very shortly).
Sorry if that is unclear. Basically, make sure Defender is totally disabled before running the commando install. You can use the setup script to make lots of changes as well.
You'll also probably want to follow the instructions for a custom install and make sure to remove all of the docker packages and the kali package.
Well hopefully this will get tested and committed... I'll wait till that happens.
I have hit this multiple times before as well.
What I found is that once you hit the black screen, hit ctrl+alt+delete or ctrl+shift+esc to open task manager. File > Run new task > explorer
As soon as you hit enter the desktop loads and the rest of the install continues. Hopefully this helps root cause or at least help those hitting this as well.
How many reboots does the installer do? I've disabled line 360 and line 383 as suggested, in order to make it work on PVE (kvm), however, it's been like .. I stopped counting at 20 reboots now..
Is that normal ?!
Additionally, even though hyper-v was disabled, docker and wsl still got installed which resulted in huge cpu usage...
Looks like it finally ended.. From the looks of it - Visual Studio 2017 didn't get installed, weird?
C'mon guys, when is this gonna get a good KVM support? "Hacking" through the install script doesn't get the job done, it seems some stuff just doesn't get installed.
Someone please create a version without Hyper-V, Docker, Kali & WSL.... Or make the installers NOT DOWNLOAD stuff, instead just using the stuff that is already in the git repo where we can edit it ourselves..
@Onepamopa I will be distributing more custom profiles this month, probably in a few weeks. In the mean time if you want to try installing on KVM you should disable the preconfig packages, which used to be on lines 360 and 383. Now they are on line 454 and 490 (and will likely change again after the next update). You want to disable this package because this is where Hyper-V is enabled within the machine, which is what was believed to be causing issues. Make sure that if you disable this package, that you fully disable defender before continuing install as that is something that is also normally handled with that preconfig package.
The reason docker and wsl are getting installed is because they were likely not removed from the install package, because you likely did not do a custom install. What you can do for this is follow instructions on the ReadMe for a custom install, and in the profile.json file, remove lines 101, 112, 125, and 206; and then run the install as described: .\install.ps1 -profile_file .\profile.json
You can also check our 2nd blog for more detailed info on custom installations
If you want more direct help you can always join the Bloodhound Gang Slack and message me in the #commando-vm channel.
@day1player By the way, Docker/WSL may also cause some issues, at least that's what happened when I installed CVM the way you suggested. As soon as docker (or something immediately before/after it) was installed, CPU usage jumped to 60% constant, and remained the same after the next reboot.
I had to manually remove docker and disable "Containers" and "Virtual Machine Platform" from Turn windows features on/off.
That solved the high cpu usage, though, at the end I don't think everything got installed properly, the desktop background was also not set to the CVD's image.
This occurred on both 1909 and 2004 (I'm currently running 2004), and installed some stuff manually using choco.
@Onepamopa Thats good to know, thank you for posting. I will be trying to make things easier for KVM users with the next update but likely wont be able to add full support until Q3 or Q4 because all of my lab equipment is unfortunately in the office and I am currently stuck working from home until further notice :-(
As for the desktop not changing thats indicative of the config package failing, you should be able to fix that by running cinst -y commandovm.win10.config.fireeye as admin.
Addressed in #147, default install will no longer enable hyperv or wsl, or install Kali or Docker.
Issue will stay open until I can confirm this works..
We are closing this issue because this should no longer be a problem with the latest release of Commando VM; packages that are to be installed can be selected in the interactive GUI installer.
