fboender
Move Ansible hosts storage to file system
Currently, ansible-cmdb is storing parsed Ansible hosts and variables in dictionary variable in memory. While this works well for tens and hundreds of hosts, it does not scale for larger inventories with thousands hosts or more. The script would run for over 1 hour and finally crash with memory exception for me. Generating cmdb inventory for 10.000+ hosts is basically impossible.
This change moves hosts management to a separate AnsibleHosts class where it implements Python dictionary so that the data is saved to a temporary directory on disk rather than your RAM thus enabling linear processing of hosts where the free space on disk is your only limit.
This only changes the back-end of the hosts dictionary so that it implements functionality of the regular Python dictionary providing standard methods, such as:
hosts = AnsibleHosts() # create new object
hosts[key] = value # set new value
item = hosts.get(key) # get value by key
len = len(hosts) # get number of items
items = hosts.items() # iterate-able items
key in hosts # check if key exists
etc.
This way it does not affect other parts of code which will utilize the dictionary as usual. Unit tests attached to verify the required functionality as well.
