Consider building custom ISOs to distribute initial workstation installs

[emkll]

We currently use the Qubes Builder[1] to build custom templates for Qubes. The Qubes Builder also provides tooling to build ISOs. We should also consider using it to create custom Qubes ISOs, pre-configured and containing our base templates, to make the initial install for production contexts much simpler. This would allow the SecureDrop workstation to resemble more like an appliance.

Given the complexity of the underlying process/code, and the relative frequency at which we will be building, we should ensure we have some form of automated testing in place to ensure the ISOs produced are valid (see https://github.com/freedomofpress/securedrop-workstation/issues/16)

The ISO can include :

• A base dom0 configuration
• The SecureDrop base templates
• SecureDrop Workstation AppVMs

The configuration would then be applied by an admin after installing the ISO

This may require hardware to build final ISOs, and some automated testing to QA/test the artifact (https://github.com/freedomofpress/securedrop-workstation/issues/524)

We could also add the securedrop-workstation-dom0-config package into the qubes-contrib repo, to make it easier or admins to bootstrap the workstation.

*** Updated 20201217 per engineering meeting discussion ***

[1] : https://github.com/QubesOS/qubes-builder

[conorsch]

We discussed this in a tech meeting today. Let's also consider getting the securedrop-workstation-dom0-config RPM package into the Qubes Contrib repositories. If we go that route, then first-time installs would involve far fewer manual steps to get up and running. Most importantly, Contrib status for the RPM doesn't close the door on a custom ISO, in fact it'd likely simplify the custom ISO logic, too.

Frankly, I'm skeptical about the maintenance burden of a custom ISO for the Workstation, but willing to pursue Contrib status and re-evaluate post-pilot.

[zenmonkeykstop]

@deeplow See previous discussion - I think it makes sense to revisit, especially if the improved Qubes CI process can help with verification of ISO builds.

[deeplow]

This is very much in line with what we talked. I'm sure it had been discussed somewhere :slightly_smiling_face:. Thanks for finding it!

[deeplow]

I've been exploring this option a bit. We'd need work with the Qubes Builder V2, which is not trivial at all and the documentation is a bit lacking at the moment. But I'll to break down what I see as the potential advantages and disadvantages of this approach.

The main advantage of having the SDW as an ISO is that it make bootstrapping easier.

Steps that the user would be able to skip:

• fetching the SDW gpg key
• creating the repo
• installing the workstation
• upgrading fedora versions

How complex would this be?

An ISO could be a relatively simple task (under certain assumptions)

Approach 1) Simple ISO with repos, key and workstation pre-installed.

This is the same as building the official Qubes ISO, but with a different kickstart file. These seem relatively simple and all that we're doing is specifying the repo, key and default packages (of which the workstation would be one). Basically our own version of this file this.

Approach 2) Workstation pre-installed + salt deployment on initial setup

When we install Qubes and then reboot, it shows us a menu where we can customize which templates we want. Then it creates all the VMs. This is running saltstates under the hood. Theoretically we could ship the workstation's saltstates and install it then, without the user needing do any further configuration.

But if wanted to add some customization at this point we can do so with InitialSetup.

The advantage here is that all of the VM setup part would be done during the installation so the user woudn't have to start it on first boot. But the time-difference should be minimal.

Disadvantages / Challenges

• The need to build and sign an ISO at least as (in)frequently as the Qubes team does it.
• Need to get familiar and support the Qubes Builder (V2) - which has some rough edges and lacks documentation (although it's not really a fork because the differences from the official ISO would be minimal).
• Boot security challenges, if we either go the SecureBoot or the Heads way we'd need our signing key somehow fused into the roots of trust (or signed by a root of trust), as I currently understand. So this aspect requires some exploration. (Update: for secureboot this should be fine since only the dom0 kernel needs to be signed and we could use the Qubes-provided one)

Alternative approaches

Having the SecureDrop keys / bootstrapping repo in Qubes-contrib or even the keys in Qubes itself #945

[deeplow]

I have attempted to build a Qubes ISO myself and ran into a few challenges. Here are some of my learnings and some open questions. Hopefully they're useful if we pick this up again.

Qubes Builder (v2) Pitfals

• when cloning the repo, make sure to clone the submodules
• the intructions mention to install mkmetalink without any other refrence. The way to do this is to actually do the following line and install qubes-infrastructure-mirrors
• qubes-infrastructure-mirrors
• obtaining the GPG singing keys in a trusted manner is not as easy as the previous builder
• one also needs to copy the RPC services and not just the policy when setting up the Qubes executor
• Qubes executor dvm needs restorecon installed or else it fails
• qubes-builder-dvm's template root partition has to be increased from 10GB to 20GB (not just the private partition of the DVM)
• the package linux-pvgrub2 fails to build on the release4.2 branch. You need to change its branch on the config file

  -  - linux-pvgrub2
  +  - linux-pvgrub2:
  +      branch: main
  

• the qubes-gui-agent (debian12) failed to build and it also failed on the main branch, so I didn't know where to proceed.

Ultimately I was able to produce an ISO but when installing it got stuck when installing qubes-artwork. I don't know if this was because I didn't build the package well. There were so many issues in package building that it's understandable if somehow I generated a corrupted ISO. More exploration is needed (and upstream help).

I have the error logs for most of this stuff, should it be needed.

Open Questions

1. It is just running ./qb install init-cache all to build the ISO or do we need to build the packages and templates first
2. Is there some setting I can toggle to pull in most components from the repos instead of building locally? (I think builderv1 had something like this)

Next Steps

1. Produce a valid and installable vanilla Qubes 4.2 ISO
2. Customize kickstart file to add the SDW repos and packages (and include the GPG key)
3. Edit FirstInstall to SDW setup salt states on first install.

[deeplow]

Update per feedback from Marek:

• Boot security challenges, if we either go the SecureBoot or the Heads way we'd need our signing key somehow fused into the roots of trust (or signed by a root of trust), as I currently understand. So this aspect requires some exploration. (Update: for secureboot this should be fine since only the dom0 kernel needs to be signed and we could use the Qubes-provided one)

[tlaurion]

@deeplow check qusal recipe for qubes-builder.

[ben-grande]

@deeplow check qusal recipe for qubes-builder.

qusal: qubes-builder.