Duplicate IP assignment

[rarenerd]

When deploying multiple VMs in a network with a small amount of IP addresses available, there is a chance that an address gets assigned twice.

The current implementation of acquiring guest IP addresses looks up a list of all available addresses (using a database query) and uses java.util.Random to randomly pick an entry from that list. https://github.com/MissionCriticalCloud/cosmic/blob/b14fab06bb06b9b58fd426254adfef3b5039463e/cosmic-core/server/src/main/java/com/cloud/network/IpAddressManagerImpl.java#L1185 After this a validation process follows, the NIC object gets populated with various fields and it gets persisted in the database. https://github.com/MissionCriticalCloud/cosmic/blob/a5c63749ee6862f7eaec71257d4e593d3bf2078a/cosmic-core/engine/orchestration/src/main/java/com/cloud/engine/orchestration/NetworkOrchestrator.java#L2085 This can take a second or 2 to complete.

With tools like terraform it has become very easy to deploy multiple VMs at the same time, sometimes within milliseconds of each other. So within this window we've seen that the same IP address gets picked for different requests, and it will always happen when there is only a single address left in the pool.