Automate building base boxes

[dezull]

I noticed that base boxes for different providers are not always updated together (example). And we currently use ESXi to host SecGen.

I would like to try to automate building base boxes, would this be something you are interested in?

Currently, this is how "Windows 2008 Virtualbox puppet 64" is built using Packer. By scripting the process, it will help address several issues such as:

• Outdated base boxes (eg: due to APT repository URL changed, this had happened before)
• Ensure base boxes for different providers are always similar
• If anything went wrong, you can inspect the script and easier for contributors to fix
• Some might want to build their own base boxes instead of using the ones on Vagrant cloud

[cliffe]

Hi @dezull,

I agree it's a little bit messy at the moment. We primarily maintain our oVirt base boxes as needed, as these are what we use day-to-day. Every so often we simply apply software updates to the bases.

We maintain the public VirtualBox bases, but since we don't actively make use of them we are unlikely to notice if something needs updating unless someone raises a GitHub issue (and yes, it's probably time for us to update all the bases). Now that Debian Wheezy is archived it's unlikely much will change, but Kali does change fairly rapidly in comparison. Often it's just a matter of running apt-get update; apt-get upgrade; and sometimes importing any replaced repository public keys. If the public bases need updating it is possible to do that locally to the base that Vagrant downloads to your bases folder, then run SecGen again.

I'm not sure how much building the bases from Packer will simplify things -- I am actually not sure that Packer even supports oVirt for example. Our university network means we are behind a proxy, the settings of which lives in all of our oVirt base boxes. Also the base needs to include the VM guest tools for whichever platform is targetted.

It would be nice to have documentation/instructions for how each base was installed (for example, a couple of commands to install the required software).

As far as I am aware you are the only person maintaining the ESXi bases, so feel free to make a pull request with new Vagrant urls when new versions of the bases are made.

Hope that helps. Thanks for your interest in the project.

[cliffe]

So it turns out it might be possible to build oVirt bases using Packer: https://github.com/ganto/packer-builder-ovirt Also, #162 is related.

I would be happy to provide advice and review your efforts.

Could you suggest how we might make some elements such as the guest tools and network settings specific to the provider and/or local requirements?

[cliffe]

Also, it would be nice if we could set the password for the vagrant user, when we build the bases -- as this is something that we want unique to our own environment.

[dezull]

Thanks @cliffe for your interest. I'm not really well versed in Packer, but I'll make some research and try a few things first. I will get back to you in a few days.

But here are some thoughts, since we don't use oVirt, it might not be feasible for us to build & test the bases. But steps common to all providers can be shared (again, windows 2008 base is the example), and contributors may verify and send fix specific to certain providers.

[cliffe]

If you got say VirtualBox and ESXi working, that would be a great start. Good luck!

[dezull]

What's the minimum requirements for the bases? guest tools and puppet?

[cliffe]

Hi @dezull

Please see README-Creating-Bases.md for a short list of some packages that need to be included in bases.

[jeremytourville]

I have a basic Ansible playbook that can be used to create Vagrant Boxes. Right now it can handle various apt based systems (Debian, Ubuntu, etc) I am presently using a combination of Virtual Box and oVirt to test the boxes.

[jeremytourville]

It would be nice to have documentation/instructions for how each base was installed (for example, a couple of commands to install the required software).

I have documented my Vagrant Base Box build and it is mostly complete. That is why I asked the questions in #162 I just had a few small gaps to fill in and wanted to ensure accuracy as best as I could. :)

Also, Marcus Young has done an excellent job of providing a script that will get yum based servers ready to use Vagrant. You should check his script https://github.com/myoung34/vagrant-ovirt4/blob/master/tools/prepare_redhat_for_box.sh Many of the packages has has listed are not required but it does give you a very good sense of what needs to be done to create a base box.

[cliffe]

Oh cool! @jeremytourville

I look forward to seeing what you have come up with. It would be nice to include it in the main SecGen repo. Is it possible to parameterise the vagrant password and proxy settings for the generated bases?

[jeremytourville]

I look forward to seeing what you have come up with. It would be nice to include it in the main SecGen repo

That is my intent. I'd be glad to upload to the repo when I have it finished and a few of your contributors have reviewed it. A colleague and I are working on making the playbook so that it can handle both yum and apt based distros regardless of version.

My Ansible playbook takes a minimal install VM and adds all the things needed to make it work with Vagrant. As part of the process I set up a new user "vagrant" and set the password. I assume that is what you mean by parameterize.

It is possible to ask the user for input and you could set the proxy settings.

https://www.mydailytutorials.com/user-inputs-with-ansible-prompt-with-examples/

We could ask the user if they have a proxy. If yes, ask for the input values.

[cliffe]

Thanks again. The draft you sent me via email looks great.

By parameterised I mean using variables in the playbook to set the vagrant password and proxy IP address.

I don't have any experience with ansible but this looks relevant: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#id35

We set a proxy in a bunch of places:

/etc/apt/apt.conf

Acquire::http::Proxy "http://172.33.0.51:3128";

/etc/security/pam_env.conf

<<<SNIP...>>>
HTTP_PROXY      DEFAULT="172.33.0.51:3128"

/etc/systemd/system/docker.service.d/http-proxy.conf (even though docker isn't installed by default, we set the proxy in case it is used)

[Service]
Environment="HTTP_PROXY=http://172.33.0.51:3128/"

[jeremytourville]

Hi @cliffe Yes, I can setup the playbook with parameters. I'll work on that in the next version. See my latest comment in #167. I want to confirm that the scenario built correctly before I claim success and spend some more time on the playbook. It helps to have a "known good" to start with. After doing the updates I think this is the first time I ever got a successful build of the scenario in oVirt. I can also provide documentation on how to build a control node for Ansible (the control node runs the playbook) and how to run the playbook on a template machine so that vagrant gets installed.

[dezull]

@cliffe I'm currently testing with packer scripts forked from Bento project: https://github.com/dezull/bento/compare/master..bento-for-secgen?diff=split

I managed to build the base boxes for virtualbox and ESXi, specifically for debian_wheezy_server & debian_stretch_server.

I'm not familiar with ansible but I wonder if @jeremytourville and I are working on a similar thing?

[jeremytourville]

@dezull I am working on building a "base box" for oVirt. SecGen uses a template in oVirt instead of downloading a base box. My Ansible playbook takes a minimal install install VM and adds all the software, services and accounts needed to make it work properly with Vagrant. The concepts are achieved in slightly different ways but in the end you have a VM built that can further be provisioned within SecGen using puppet. I hope this helps clarify.

[dezull]

@jeremytourville Correct me if I'm wrong, this is done outside of Secgen and specifically written for oVirt?

[jeremytourville]

@dezull That is correct. I am not aware of any base boxes that can be downloaded for import into oVirt as a template. When using SecGen with oVirt it requires that you have a pre-built template available for use.

[dezull]

@cliffe I forked bento and keep only templates related to SecGen in this branch https://github.com/dezull/bento/tree/bento-for-secgen-trim. There are currently templates for 6 base boxes: debian 7 (& kde variant), debian 9 (& kde variant), kali light, kali light msf. I have tested with both ESXi and Virtualbox.

This is still WIP but do you think I should proceed with SecGen integration?

[cliffe]

Hi @dezull amd @jeremytourville -- just closing this issue as I'm cleaning up the SecGen issue tracker. Thanks for your interest in the project.

We are still relying on manual updates to bases, and looking at moving to a newer Debian base soon. Anyway, have either of you been using automated techniques recently?