Support installation over loop devices

[davidcassany]

Within the current elemental it should really simple to run an installation from directory, docker image or channel to a loop device. This eventually could allow as to build disk images from any linux host just by calling elemental install <directory|docker|channel> /dev/loop0, or taking it even further with something like elemental builddisk mydisk.<raw|qcow|vmdk>. This simple step could eventually free us from having to use Packer, no need to build ISOs for building disks.

Note this could also significantly speed up our CI: -> no need to build iso to build disks, both could happen in parallel. -> No need to spawn a VM to create a disk, faster procedure

What is missing to make it possible:

• ~~[ ] Custom docker|directory|channel images on recovery installation. Two options: either use the same image for everything or set extra flags for recovery image (aka --recovery-docker-image)~~ This is related to #1145
• [ ] New command that does the following:
  1. Create raw disk image (truncate -s <size> mydisk.raw)
  2. Load it as loop device (losetup -f --show mydisk.raw)
  3. Install elemental (elemental install /dev/loop0)
  4. Convert disk image to specified format (qemu-img convert mydisk.raw )
• [ ] Make our runner more roboust, check commands exist before executing and log some human readable error. If we enable to run installer on any host we should consider the host might not have all tools available (parted, qemu-img, grub, etc.)
• [ ] Running as root checks, this commands require root privileges, so similar to the above, there should be an early check for that if aim to use it to build disk images in any host.

[kkaempf]

Sounds awfully similar to https://github.com/rancher-sandbox/os2/issues/6 🤔 😉

[mudler]

Within the current elemental it should really simple to run an installation from directory, docker image or channel to a loop device. This eventually could allow as to build disk images from any linux host just by calling elemental install <directory|docker|channel> /dev/loop0, or taking it even further with something like elemental builddisk mydisk.<raw|qcow|vmdk>. This simple step could eventually free us from having to use Packer, no need to build ISOs for building disks. We are damn close of being capable to do that.

Note this could also significantly speed up our CI: -> no need to build iso to build disks, both could happen in parallel. -> No need to spawn a VM to create a disk, faster procedure

What is missing to make it possible:

* [ ]  Custom docker|directory|channel images on recovery installation. Two options: either use the same image for everything or set extra flags for recovery image (aka `--recovery-docker-image`).

* [ ]  New command that does the following:
  
  1. Create raw disk image (`truncate -s <size> mydisk.raw`)
  2. Load it as loop device (`losetup -f --show mydisk.raw`)
  3. Install elemental (`elemental install /dev/loop0`)
  4. Convert disk image to specified format (`qemu-img convert mydisk.raw `)

* [ ]  Make our runner more roboust, check commands exist before executing and log some human readable error. If we enable to run installer on _any_ host we should consider the host might not have all tools available (`parted`, `qemu-img`, `grub`, etc.)

* [ ]  Running as root checks, this commands require root privileges, so similar to the above, there should be an early check for that if aim to use it to build disk images in any host.

Note only the first item requires some refactoring, all the rest are low hanging fruits.

This sounds really good to me, I think it's also in the #108 scope as part of the build aspect.

This would alleviates CI to need to wait for each other, however we should still run some smoke tests for ISOs, packer etc to guarantee that we have still support for that. (For example, by testing produced VMs, we don't know if ISOs are functional, and vice-versa)

[mudler]

Sounds awfully similar to rancher-sandbox/os2#6 thinking wink

note that https://github.com/rancher-sandbox/os2/issues/6 can be already achieved with the current cOS featureset, it is mostly about tying it the os2 configuration

[davidcassany]

This would alleviates CI to need to wait for each other, however we should still run some smoke tests for ISOs, packer etc to guarantee that we have still support for that. (For example, by testing produced VMs, we don't know if ISOs are functional, and vice-versa)

Yes, in fact I was thinking of keeping the current installer tests which are based on ISOs and boot the resulting disk. However would completely drop the disk image creation using Packer form the CI and simple call, in parallel to the ISO creation, the disk creation, basically as a substitute of vbox-nonsquashfs-green and its squashfs variant. With this we could reduce one step the CI chain and feature wise keep everything equally tested.

Then regarding Packer support we could add a new test for it, even IMHO we could completely drop it, but keeping it around should not be a big deal.

[davidcassany]

Just to update:

$ qemu-img create -f raw disk.img 30G
$ sudo losetup -f --show disk.img
/dev/loop0
$ sudo elemental install --docker-image quay.io/costoolkit/releases-green:cos-system-0.8.2-6 /dev/loop0
$ sudo losetup -d /dev/loop0

Is already possible and functional

[davidcassany]

While some part of this card are still valid, like adding toolchain checks in elemental-cli the whole story about building disks is outdated and about to be superseded by rancher/elemental-cli#422