Virtualbox Dynamic Inventory Nested Group Assignments

[lyrandy]

Summary

My usage of Ansible takes advantage of Ansible's nested groups for variable management. I make heavy use of Vagrant + VirtualBox to speed up local development and testing of Ansible code.

The way the dynamic inventory assigns groups currently does not match the VirtualBox documentation. According to the documentation, you can assign a VM with multiple groups by delimiting it with a comma , character, and nest groups using the slash / character.

Ultimately, the objective is to be able to harmonize the way groups are set up with VirtualBox and with Ansible. It appears the concept of nested (or parent-child) groups match, so it should align.

See: https://www.virtualbox.org/manual/UserManual.html#gui-vmgroups and https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html#how-variables-are-merged

From VirtualBox Docs:

# Create multiple groups. For example:
VBoxManage modifyvm "vm01" --groups "/TestGroup,/TestGroup2"

#Create nested groups, having a group hierarchy. For example:
VBoxManage modifyvm "vm01" --groups "/TestGroup/TestGroup2"

The way the dynamic inventory parses the groups considers the / character as the delimiter for group assignment. As a consequence, we are unable to use nested groups in Ansible. If a user does not care about nested group, and assign multiple groups by simply using the / character (e.g. /TestGroup/TestGroup2), the VirtualBox UI will show a deeply nested list of groups before displaying the actual VM. See vm-1 in the screenshot for an example.

The example I provide shows 2 issues resulting from how groups are parsed:

vm-1 is not set up with nested groups, but is set up to participate in 3 top-level groups. vm-2 groups have a trailing comma (except for the last one).

I did not spot mentions about how groups are automatically inferred in the official docs: https://docs.ansible.com/ansible/latest/collections/community/general/virtualbox_inventory.html

Issue Type

Bug Report

Component Name

virtualbox inventory

Ansible Version

ansible [core 2.17.0]
  config file = None
  configured module search path = ['/home/rly/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rly/Documents/dev/sandbox/.venv/lib/python3.12/site-packages/ansible
  ansible collection location = /home/rly/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/rly/Documents/dev/sandbox/.venv/bin/ansible
  python version = 3.12.3 (main, Apr 23 2024, 09:16:07) [GCC 13.2.1 20240417] (/home/rly/Documents/dev/sandbox/.venv/bin/python)
  jinja version = 3.1.4
  libyaml = True

Community.general Version

# /home/rly/Documents/dev/sandbox/.venv/lib/python3.12/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 9.0.1

Configuration

CONFIG_FILE() = None

OS / Environment

My Host OS: 6.9.1-arch1-1 Vagrant: 2.4.1 VirtualBox: 7.0.18 r162988 VirtualBox VMs: bento/rockylinux-9 Vagrant images

Steps to Reproduce

I made a sample Vagrantfile to help illustrate. Notice that I'm purposefully trying to make the hierarchy of groups for vm-1 to be the reverse from lexicographical ordering.

Ensure your VirtualBox host has a host-only network named vboxnet0 with 192.168.56.1/24 set up. The Vagrantfile will also copy your SSH key into the VMs for convenience when using with Ansible.

Vagrant.configure("2") do |config|
  DEFAULT_VAGRANT_BOX = "bento/rockylinux-9"
  
  config.vm.define "vm-1" do |config|
    hostname = "vm-1"
    config.vm.box = DEFAULT_VAGRANT_BOX
    config.vm.hostname = hostname
    config.vm.network "private_network", ip: "192.168.56.10", name: "vboxnet0"

    config.vm.provider "virtualbox" do |vb|
      vb.name = hostname
      vb.customize ["modifyvm", :id, "--groups", "/zgroup1/ygroup2/xgroup3"]
    end

    config.vm.provision "shell", run: "always" do |s|
      ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
      s.inline = <<-SHELL
        echo "SSH Public key will be added to /home/vagrant/.ssh/authorized_keys"
        echo '#{ssh_pub_key}' >> /home/vagrant/.ssh/authorized_keys
      SHELL
    end
  end

  config.vm.define "vm-2" do |config|
    hostname = "vm-2"
    config.vm.box = DEFAULT_VAGRANT_BOX
    config.vm.hostname = hostname
    config.vm.network "private_network", ip: "192.168.56.11", name: "vboxnet0"
    config.vm.provider "virtualbox" do |vb|
      vb.name = hostname
      vb.customize ["modifyvm", :id, "--groups", "/othergroup1,/othergroup2,/othergroup3"]
    end

    config.vm.provision "shell", run: "always" do |s|
      ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
      s.inline = <<-SHELL
        echo "SSH Public key will be added to /home/vagrant/.ssh/authorized_keys"
        echo '#{ssh_pub_key}' >> /home/vagrant/.ssh/authorized_keys
      SHELL
    end
  end
end

And my simple inventory file

# virtualbox.yml
plugin: community.general.virtualbox

network_info_path: /VirtualBox/GuestInfo/Net/1/V4/IP 
running_only: true

compose:
  ansible_user: "'vagrant'"

And some sample group vars

grep -R my_custom_variable group_vars/ | sort

group_vars/all.yml:my_custom_variable: This is in "all" group
group_vars/xgroup3.yml:my_custom_variable: This is in "xgroup3" group
group_vars/ygroup2.yml:my_custom_variable: This is in "ygroup2" group
group_vars/zgroup1.yml:my_custom_variable: This is in "zgroup1" group

vagrant up
ansible -i virtualbox.yml -m debug -a "var=group" all
ansible -i virtualbox.yml -m debug -a "var=group_names" all
ansible -i virtualbox.yml -m debug -a "var=my_custom_variable" all

Expected Results

I expected to see vm-1 have its variable resolved properly, in consideration for nested groups when using /zgroup1/ygroup2/xgroup3, and for vm-2 to be part of separate groups when using /othergroup1,/othergroup2,/othergroup3

For vm-1, if we were to consider this in a static inventory, it might look like this:

all:
  vars:
    my_custom_variable: This is in "all" group
  children:
    zgroup1:
      vars:
        my_custom_variable: This is in "zgroup1" group
      children:
        ygroup2:
          vars:
            my_custom_variable: This is in "ygroup2" group
          children:
            xgroup3:
              vars:
                my_custom_variable: This is in "xgroup3" group
              hosts:
                vm-1:
                  ansible_host: 192.168.56.10
                  ansible_user: vagrant

and

ansible -i static.yml -m debug -a "var=my_custom_variable" all
vm-1 | SUCCESS => {
    "my_custom_variable": "This is in \"xgroup3\" group"
}

The same would be true with the dynamic inventory.

ansible -i virtualbox.yml -m debug -a "var=my_custom_variable" vm-1
vm-1 | SUCCESS => {
    "my_custom_variable": "This is in \"xgroup3\" group"
}

For vm-2, it should be part of the othergroup1, othergroup2, othergroup3 groups.

ansible -i virtualbox.yml -m debug -a "var=group_names" vm-2
vm-2 | SUCCESS => {
    "group_names": [
        "othergroup1",
        "othergroup2",
        "othergroup3"
    ]
}

Actual Results

ansible -i virtualbox.yml -m debug -a "var=group_names" all

vm-1 | SUCCESS => {
    "group_names": [
        "xgroup3",
        "ygroup2",
        "zgroup1"
    ]
}
vm-2 | SUCCESS => {
    "group_names": [
        "othergroup1,",
        "othergroup2,",
        "othergroup3"
    ]
}

In this case, vm-1 appears to be correctly assigned, but watch out!

ansible -i virtualbox.yml -m debug -a "var=my_custom_variable" all
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
vm-1 | SUCCESS => {
    "my_custom_variable": "This is in \"zgroup1\" group"
}
vm-2 | SUCCESS => {
    "my_custom_variable": "This is in \"all\" group"
}

The zgroup1 group vars wins because it's ordered last, lexicographically. This shows they're considered to be "on the same level".

Also, vm-2 groups have a trailing , character which is incorrect. This is due to the code simply splitting on the / character, so the , remains.

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/inventory/virtualbox.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[lyrandy]

I believe this area can be improved to accommodate.

https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/virtualbox.py#L180-L187

While doing some more digging, I also found https://github.com/ansible/ansible/issues/17100#issuecomment-504467928 which mentions that there aren't really a hierarchy of groups, since everything eventually renders down to hosts and vars. I think it's still helpful to conceptualize the group hierarchy to rationalize how variables will eventually "settle", so to speak.

[felixfontein]

As mentioned in the PR: this is not a bug, but a feature request.

[lyrandy]

Thank you for having a look at the PR and providing feedback. I'll address the CI and then make the recommended changes. I'm not too certain how to change the label from bug to feature, do I need to open a new issue?

For my own understanding, how are undocumented behaviours treated when it comes to making modifications to them? In my case, I did not notice any mention of how it integrates with VirtualBox groups, just the usual groups and keyed_groups keywords I've seen in basically all other dynamic inventory plugins. Is the approach to treat changes to the behaviour as a new feature until it is formally documented, after which it becomes more obvious to triage?

Also, breaking changes understandably must have sufficient communication and lead time for consumers to prepare for the change. How does it work to schedule the release of the breaking change? I only know that it will usually be included in major version updates, but I am curious about the other logistics before the release (like from when a PR is raised, how is it communicated, etc)

I am new to the scene so I appreciate your help and guidance regarding processes and workflows :)

[felixfontein]

@lyrandy sorry, I forgot to reply to this comment...

For my own understanding, how are undocumented behaviours treated when it comes to making modifications to them? In my case, I did not notice any mention of how it integrates with VirtualBox groups, just the usual groups and keyed_groups keywords I've seen in basically all other dynamic inventory plugins. Is the approach to treat changes to the behaviour as a new feature until it is formally documented, after which it becomes more obvious to triage?

There is no clear way how to treat changing undocumented behavior. (Even for documented behavior it's hard to figure out whether the documentation is faulty or the behavior. There are cases where it seems clear, but there's always https://xkcd.com/1172/ ... and some less humorous examples, like when the documentation claims something for four years, but the behavior was always different in these four years, and fixing the behavior could break paybooks of users. What's best to do in that case is generally not clear.)

I guess I generally weight existing behavior as more important than expectations of folks who didn't use it before, especially if it has been that behavior for a long time. (If it's a feature that was just added in the previous minor release, it tends to break less users than something that has been this way for many years.) But it's always a judgement call...

Also, breaking changes understandably must have sufficient communication and lead time for consumers to prepare for the change. How does it work to schedule the release of the breaking change? I only know that it will usually be included in major version updates, but I am curious about the other logistics before the release (like from when a PR is raised, how is it communicated, etc)

Generally to make a breaking change (except in security relevant cases, where breaking changes can be acceptable as bugfixes) the current behavior needs to be deprecated for at least 1-2 major release cycles (it generally should be at least a year, and we have a new major release every 6 months; for behaviors that have been there for many years and that don't look like a bug, even longer deprecation cycles make sense), and should be communicated to the user with module.deprecate() (modules) or display.deprecated() (plugins), preferably only in cases where the behavior change actually changes something for the user.

Also preferably it should be possible to somehow select the wanted behavior, so that the user can decide now which behavior they want (and get rid of the deprecation warning - the deprecation warning should only be shown if the user didn't pick the behavior, but relied on the default value for that configuration). Then effectively the default value for the configuration that selects the behavior gets deprecated.

[lyrandy]

No worries Felix! Thanks for taking the time circle back to this to help me understand and to provide guidance! You're amazing!