cloudstack
cloudstack copied to clipboard
Dedication of public IP ranges broken after upgrade to 4.17.0.1
ISSUE TYPE
- Bug Report
COMPONENT NAME
API, UI, SSVM, CPVM
CLOUDSTACK VERSION
4.17.0.1
CONFIGURATION
1 Zone, Advanced networking, 3 domains
OS / ENVIRONMENT
All running Ubuntu 20.04.5 Database mysql 8
SUMMARY
Existing configuration (working running CS 4.16.1)

Setting system.vm.public.reservation.mode.strictness
= true
After upgrade:
While deploying SSVM / CPVM, dedicated IP Ranges for use of system VMs aren't concidert for deployment. With system.vm.public.reservation.mode.strictness
enabled, a deployment of SSVM / CPVM fails "due to Insufficent address capacity".
With system.vm.public.reservation.mode.strictness
set to 'false' deployment is successfull while using IP ranges not marked as dedicated for system VMs.
Newly created IP ranges with dedication for systemVMs aren't stored or found in db table domain_vlan_map
. Also prior used ip ranges dedicated for systemVMs aren't referenced in domain_vlan_map
.
Currently tested with public IP addresses. Not tested with "strictness" for management IP addresses.
STEPS TO REPRODUCE
- Create IP range for public IPs with dedication for system VMs enabled
- Change setting
system.vm.public.reservation.mode.strictness
to true - Destroy existing CPVM / SSVM
EXPECTED RESULTS
- Table `domain_vlan_map` shows entry for newly created IP range (stored as new vlan) assigned to domain
- System VMs are successfully deployed using IPs from the dedicated IP range
ACTUAL RESULTS
- Table `domain_vlan_map` shows no entry
- Deployment of systemVMs fails due to insufficent address capacity
- Changeing setting `system.vm.public.reservation.mode.strictness` to "false"
--> deployment successfull using IP range of domain not dedicated for systemVMs
@Hudratronium I'm not able to reproduce this on 4.17 branch. I added a new public range and set system.vm.public.ip.reservation.mode.strictness
to true
. New system vms came up fine in the new range
(local) 🐓 > list zones id=8395094c-305b-46cd-878a-b1dcc01c37d0
{
"count": 1,
"zone": [
{
"allocationstate": "Enabled",
"dhcpprovider": "VirtualRouter",
"dns1": "10.147.28.6",
"guestcidraddress": "10.1.1.0/24",
"hasannotations": false,
"id": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"internaldns1": "10.147.28.6",
"localstorageenabled": false,
"name": "Sandbox-simulator",
"networktype": "Advanced",
"securitygroupsenabled": false,
"tags": [],
"zonetoken": "bfd27e96-3ba0-3df3-b387-f8ff690c63c5"
}
]
}
(local) 🐗 > list systemvms zoneid=8395094c-305b-46cd-878a-b1dcc01c37d0
{
"count": 2,
"systemvm": [
{
"activeviewersessions": 0,
"created": "2022-09-05T12:36:52+0530",
"dns1": "10.147.28.6",
"gateway": "192.168.2.1",
"hasannotations": false,
"hostid": "fc35ad4a-bdc8-496a-94e9-46d9d228526a",
"hostname": "SimulatedAgent.9d862bb4-6179-4a90-ab8a-f91b23fac374",
"hypervisor": "Simulator",
"id": "52ff1944-20d5-4954-8cfd-dcd0ed96b46f",
"isdynamicallyscalable": false,
"linklocalip": "169.254.196.124",
"linklocalmacaddress": "0e:00:a9:fe:c4:7c",
"linklocalnetmask": "255.255.0.0",
"name": "v-9-VM",
"podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
"podname": "POD0",
"privateip": "172.16.15.65",
"privatemacaddress": "1e:00:54:00:00:40",
"privatenetmask": "255.255.255.0",
"publicip": "192.168.2.2",
"publicmacaddress": "1e:00:91:00:00:c8",
"publicnetmask": "255.255.255.0",
"state": "Running",
"systemvmtype": "consoleproxy",
"templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
"templatename": "SystemVM Template (simulator)",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"zonename": "Sandbox-simulator"
},
{
"agentstate": "Up",
"created": "2022-09-05T12:36:53+0530",
"dns1": "10.147.28.6",
"gateway": "192.168.2.1",
"hasannotations": false,
"hostid": "bdefc6c3-4a43-4d6a-9570-0fc9a6ac6d14",
"hostname": "SimulatedAgent.d1b801a8-36b8-47e7-bfd7-710402b63a99",
"hypervisor": "Simulator",
"id": "eb726c05-b467-48b7-b42e-8a6c0a9bca42",
"isdynamicallyscalable": false,
"linklocalip": "169.254.181.229",
"linklocalmacaddress": "0e:00:a9:fe:b5:e5",
"linklocalnetmask": "255.255.0.0",
"name": "s-10-VM",
"podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
"podname": "POD0",
"privateip": "172.16.15.20",
"privatemacaddress": "1e:00:75:00:00:13",
"privatenetmask": "255.255.255.0",
"publicip": "192.168.2.3",
"publicmacaddress": "1e:00:16:00:00:c9",
"publicnetmask": "255.255.255.0",
"state": "Running",
"systemvmtype": "secondarystoragevm",
"templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
"templatename": "SystemVM Template (simulator)",
"version": "4.17.1.0-SNAPSHOT",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"zonename": "Sandbox-simulator"
}
]
}
(local) 🐰 > list vlanipranges zoneid=8395094c-305b-46cd-878a-b1dcc01c37d0
{
"count": 1,
"vlaniprange": [
{
"account": "system",
"cidr": "192.168.2.0/24",
"domain": "ROOT",
"domainid": "fe476c7c-2cdc-11ed-b1e6-645d8651f45a",
"endip": "192.168.2.200",
"forsystemvms": false,
"forvirtualnetwork": true,
"gateway": "192.168.2.1",
"id": "10b19120-cfd8-441c-81c8-e156c09c38e4",
"netmask": "255.255.255.0",
"networkid": "234ef426-bb2f-445e-83d9-5363fc6bfb38",
"physicalnetworkid": "a44e9a10-6682-42e1-81b3-ffabfdfc30f9",
"startip": "192.168.2.2",
"vlan": "vlan://50",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0"
}
]
}
(local) 🐈 > list configurations name=system.vm.public.ip.reservation.mode.strictness
{
"configuration": [
{
"category": "Advanced",
"description": "If enabled, the use of System VMs public IP reservation is strict, preferred if not.",
"isdynamic": true,
"name": "system.vm.public.ip.reservation.mode.strictness",
"value": "false"
}
],
"count": 1
}
(local) 🐬 > create vlaniprange gateway=192.168.3.1 netmask=255.255.255.0 startip=192.168.3.2 endip=192.168.3.200 zoneId=8395094c-305b-46cd-878a-b1dcc01c37d0 vlan=51 forsystemvms=true forvirtualnetwork=true
{
"vlan": {
"account": "system",
"cidr": "192.168.3.0/24",
"domain": "ROOT",
"domainid": "fe476c7c-2cdc-11ed-b1e6-645d8651f45a",
"endip": "192.168.3.200",
"forsystemvms": true,
"forvirtualnetwork": true,
"gateway": "192.168.3.1",
"id": "edc376e0-cf2d-4ab7-aaac-0b7b14e2a931",
"netmask": "255.255.255.0",
"networkid": "234ef426-bb2f-445e-83d9-5363fc6bfb38",
"physicalnetworkid": "a44e9a10-6682-42e1-81b3-ffabfdfc30f9",
"startip": "192.168.3.2",
"vlan": "vlan://51",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0"
}
}
(local) 🦌 > update configuration name=system.vm.public.ip.reservation.mode.strictness value=true
{
"configuration": {
"category": "Advanced",
"description": "If enabled, the use of System VMs public IP reservation is strict, preferred if not.",
"isdynamic": true,
"name": "system.vm.public.ip.reservation.mode.strictness",
"value": "true"
}
}
(local) 🐃 > destroy systemvm id=52ff1944-20d5-4954-8cfd-dcd0ed96b46f
{
"systemvm": {
"created": "2022-09-05T12:36:52+0530",
"dns1": "10.147.28.6",
"hasannotations": false,
"hostid": "fc35ad4a-bdc8-496a-94e9-46d9d228526a",
"hostname": "SimulatedAgent.9d862bb4-6179-4a90-ab8a-f91b23fac374",
"hypervisor": "Simulator",
"id": "52ff1944-20d5-4954-8cfd-dcd0ed96b46f",
"isdynamicallyscalable": false,
"name": "v-9-VM",
"podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
"podname": "POD0",
"state": "Running",
"systemvmtype": "consoleproxy",
"templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
"templatename": "SystemVM Template (simulator)",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"zonename": "Sandbox-simulator"
}
}
(local) 🌵 > destroy systemvm id=eb726c05-b467-48b7-b42e-8a6c0a9bca42
{
"systemvm": {
"created": "2022-09-05T12:36:53+0530",
"dns1": "10.147.28.6",
"hasannotations": false,
"hostid": "bdefc6c3-4a43-4d6a-9570-0fc9a6ac6d14",
"hostname": "SimulatedAgent.d1b801a8-36b8-47e7-bfd7-710402b63a99",
"hypervisor": "Simulator",
"id": "eb726c05-b467-48b7-b42e-8a6c0a9bca42",
"isdynamicallyscalable": false,
"name": "s-10-VM",
"podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
"podname": "POD0",
"state": "Running",
"systemvmtype": "secondarystoragevm",
"templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
"templatename": "SystemVM Template (simulator)",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"zonename": "Sandbox-simulator"
}
}
(local) 🐂 > list systemvms zoneid=8395094c-305b-46cd-878a-b1dcc01c37d0
{
"count": 2,
"systemvm": [
{
"activeviewersessions": 0,
"created": "2022-09-05T12:39:23+0530",
"dns1": "10.147.28.6",
"gateway": "192.168.3.1",
"hasannotations": false,
"hostid": "f0bcf07c-ab72-4137-a413-449684ae309c",
"hostname": "SimulatedAgent.f83bd68a-5642-49bc-aac2-dd8a4785aedc",
"hypervisor": "Simulator",
"id": "01a9646f-17af-430d-aab4-a07b4a9eec35",
"isdynamicallyscalable": false,
"linklocalip": "169.254.170.181",
"linklocalmacaddress": "0e:00:a9:fe:aa:b5",
"linklocalnetmask": "255.255.0.0",
"name": "v-11-VM",
"podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
"podname": "POD0",
"privateip": "172.16.15.145",
"privatemacaddress": "1e:00:60:00:00:90",
"privatenetmask": "255.255.255.0",
"publicip": "192.168.3.200",
"publicmacaddress": "1e:00:40:00:03:f7",
"publicnetmask": "255.255.255.0",
"state": "Running",
"systemvmtype": "consoleproxy",
"templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
"templatename": "SystemVM Template (simulator)",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"zonename": "Sandbox-simulator"
},
{
"agentstate": "Up",
"created": "2022-09-05T12:39:53+0530",
"dns1": "10.147.28.6",
"gateway": "192.168.3.1",
"hasannotations": false,
"hostid": "bdefc6c3-4a43-4d6a-9570-0fc9a6ac6d14",
"hostname": "SimulatedAgent.d1b801a8-36b8-47e7-bfd7-710402b63a99",
"hypervisor": "Simulator",
"id": "02c6d3dd-13ae-44d4-bf90-0d20e9647525",
"isdynamicallyscalable": false,
"linklocalip": "169.254.102.29",
"linklocalmacaddress": "0e:00:a9:fe:66:1d",
"linklocalnetmask": "255.255.0.0",
"name": "s-12-VM",
"podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
"podname": "POD0",
"privateip": "172.16.15.138",
"privatemacaddress": "1e:00:1f:00:00:89",
"privatenetmask": "255.255.255.0",
"publicip": "192.168.3.199",
"publicmacaddress": "1e:00:fd:00:03:f6",
"publicnetmask": "255.255.255.0",
"state": "Running",
"systemvmtype": "secondarystoragevm",
"templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
"templatename": "SystemVM Template (simulator)",
"version": "4.17.1.0-SNAPSHOT",
"zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
"zonename": "Sandbox-simulator"
}
]
}
@shwstppr Here the whole log with comments on database actions from my system. Out of interest: new created CS4.17 setup or upgraded from 4.16 / 4.15 / .... ?
Cann you maybe take a look into your db and lookig if you can find in domain_vlan_map
a entry for referencing the vlan which is enabled for use with systemVMs and the domain you are actually using ?
###################################################################### Nic allocation
2022-08-31 18:44:21,203 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nics for VM instance {id: "86", name: "v-86-VM", uuid: "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}
2022-08-31 18:44:21,203 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-79, consoleproxy-1,
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by: -Transaction.execute:39-NetworkOrchestrator.allocate:778-VirtualMachineManagerImpl$1.doInTransactionWithoutResult:478-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-VirtualMachineManagerImpl.allocate:469-VirtualMachineManagerImpl.allocate:528-ConsoleProxyManagerImpl.createProxyInstance:700-ConsoleProxyManagerImpl.startNew:572-ConsoleProxyManagerImpl.allocCapacity:809-ConsoleProxyManagerImpl.expandPool:1537
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been started.
2022-08-31 18:44:21,204 TRACE [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) allocating networks for v-86-VM(template c20799e4-a990-449a-aae6-4b67374c7aff); 3 networks
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Connection] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Creating a DB connection with txn: for 0: dbconn623913035. Stack: -TransactionLegacy.prepareStatement:466-TransactionLegacy.prepareAutoCloseStatement:459-GenericDaoBase.searchIncludingRemoved:396-GenericDaoBase.searchIncludingRemoved:359-GenericDaoBase.search:348-GenericDaoBase.search:1319-TemplateDeployAsIsDetailsDaoImpl.listDetailsByTemplateIdMatchingPrefix:64-TemplateDeployAsIsDetailsDaoImpl.listNetworkRequirementsByTemplateId:69-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT template_deploy_as_is_details.id, template_deploy_as_is_details.template_id, template_deploy_as_is_details.name, template_deploy_as_is_details.value FROM template_deploy_as_is_details WHERE template_deploy_as_is_details.template_id = ? AND template_deploy_as_is_details.name LIKE ?
#### template_deploy_as_is_details = empty --> no results returned
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT template_deploy_as_is_details.id, template_deploy_as_is_details.template_id, template_deploy_as_is_details.name, template_deploy_as_is_details.value FROM template_deploy_as_is_details WHERE template_deploy_as_is_details.template_id = 213 AND template_deploy_as_is_details.name LIKE x'6E6574776F726B2D25'
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Connection] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: Not closing DB connection because we're still in a transaction.
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = ? AND networks.removed IS NULL
2022-08-31 18:44:21,209 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nic for vm VM instance {id: "86", name: "v-86-VM", uuid: "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"} in network Network {"id": 200, "name": "null", "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with requested profile NicProfile {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = ? AND data_center.removed IS NULL
### Allocating public nic
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = 200 AND networks.removed IS NULL
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account.id, account.account_name, account.type, account.role_id, account.domain_id, account.state, account.created, account.removed, account.cleanup_needed, account.network_domain, account.uuid, account.default_zone_id, account.default FROM account WHERE account.id = ? AND account.removed IS NULL
### Getting Zone Information
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = 1 AND data_center.removed IS NULL
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by: -Transaction.execute:39-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823-NetworkOrchestrator$3.doInTransactionWithoutResult:797-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been started.
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = ? AND data_center.removed IS NULL
### Account Information "system"
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account.id, account.account_name, account.type, account.role_id, account.domain_id, account.state, account.created, account.removed, account.cleanup_needed, account.network_domain, account.uuid, account.default_zone_id, account.default FROM account WHERE account.id = 1 AND account.removed IS NULL
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = ? AND networks.removed IS NULL
### Getting Zone Information
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = 1 AND data_center.removed IS NULL
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account_vlan_map.id, account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM account_vlan_map WHERE account_vlan_map.account_id = ?
### Getting network information
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = null AND networks.removed IS NULL
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map WHERE domain_vlan_map.domain_id = ?
#### Account_vlan_map - no entry for account "System" nor for Account "Admin"
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_vlan_map.id, account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM account_vlan_map WHERE account_vlan_map.account_id = 1
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = ? AND vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )
### Checking for availeable VLANS for domain --> VLAN for system VM not shown / or in the result
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map WHERE domain_vlan_map.domain_id = 1
### Checking for availeable VLANS wich are NOT assigned to ANY Account NOR to ANY Domain --> Result is the intended vlan with expected IP Range
2022-08-31 18:44:21,214 TRACE [c.c.u.d.GenericDaoBase] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?
### Getting further information on the vlan --> correct choice of expected VLAN + IP-Range
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = ? AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN (?,?,?,?) AND user_ip_address.forsystemvms = ? AND user_ip_address.removed IS NULL AND (vlan.vlan_type = ? ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT 0, 1 FOR UPDATE
### Check of setting system.vm.public.ip.reservation.mode.strictness --> value "true"
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = x'73797374656D2E766D2E7075626C69632E69702E7265736572766174696F6E2E6D6F64652E7374726963746E657373'
2022-08-31 18:44:21,215 TRACE [c.c.u.d.GenericDaoBase] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT 0, 1 FOR UPDATE
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account_details.id, account_details.account_id, account_details.name, account_details.value FROM account_details WHERE account_details.account_id = ? AND account_details.name = ? ORDER BY RAND() LIMIT 1
### Trying to fetch IP-Address: No value / ip address returned - Domain "Root" shall provide ip addresses; thats where the vlanIDs are comming from --> user_ip_address.vlan_db_id IN (3,10,13,14)
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT 0, 1 FOR UPDATE
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ? ORDER BY RAND() LIMIT 1
### Trying to get information from the table account_detail for the account "system". Table is empty so no result back
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_details.id, account_details.account_id, account_details.name, account_details.value FROM account_details WHERE account_details.account_id = 1 AND account_details.name = x'7573652E73797374656D2E7075626C69632E697073' ORDER BY RAND() LIMIT 1
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?
### Check for Configuration enable.account.settings.for.domain --> current value = false
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = x'656E61626C652E6163636F756E742E73657474696E67732E666F722E646F6D61696E' ORDER BY RAND() LIMIT 1
2022-08-31 18:44:21,217 WARN [c.c.n.IpAddressManagerImpl] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to get ip address in zone id=1, vlanId id=[3, 10, 13, 14]
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = ? AND data_center.removed IS NULL
### Check for setting use.system.public.ips --> current value = false
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = x'7573652E73797374656D2E7075626C69632E697073'
### getting information for the current data_center / zone
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = 1 AND data_center.removed IS NULL
2022-08-31 18:44:21,219 DEBUG [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Rolling back the transaction: Time = 17 Name = consoleproxy-1; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Lock] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) SQL com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT 0, 1 FOR UPDATE took 4
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Connection] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing DB connection: dbconn623913035
2022-08-31 18:44:21,220 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to allocate proxy [Console VM instance {id: "86", name: "v-86-VM", uuid: "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}] on zone [1] due to [Insufficient address capacity].
com.cloud.exception.InsufficientAddressCapacityException: Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=1
@Hudratronium mine was a new env. Maybe will check with upgrade
@shwstppr
at least it occurs to me that in my current setup the missing entry in the cloud.domain_vlan_map
seems to be the crucial thing missing.
I can create a new dedicated range for the domain but it isn't represented in the cloud.domain_vlan_map
table. Which seems to be crucial in the overall "search" for a fitting IP-Range. I don't know why this isn't updated / the existing configuration doesn't work out anymore.
Maybe some more infos regarding our global conf:
use.system.public.ips
= false
system.vm.public.reservation.mode.strictness
= true
From my reading of the trace-log above the process ist:
- Check if the account has dedicated vlans --> result is "none"
- Check if the domain has dedicated vlans --> results are all vlan for a domain, but missing the one for use with systemVMs as it isn't present in
cloud.domain_vlan_map
- Check if there are any vlans NOT assigned to a account or domain --> result it the correct VLAN (guess that would be the check if there is a "systempool") --> not choosen as
use.system.public.ips
= false - Looking up setting
system.vm.public.reservation.mode.strictness
--> if true only domain related VLANs shall be checked /user_ip_address.forsystemvms
setting is irrelevant - lookup IPs - depenending on assigned vlans of the domain - and check for IPs which are not assigned and
user_ip_address.forsystemvms = 1
--> fails as the "correct" vlan isn't in the results of the former querys as it is missing in db table
@Hudratronium can you please share logs/API call for adding the new ip range. Also, the cmk output of list vlanipranges
I don't think there should be an entry in cloud.domain_vlan_map
when the range is reserved for system vms. Entry in this table is added only when the following code conditions are met for the range
} else if (domain != null && !forSystemVms) {
// This VLAN is domain-wide, so create a DomainVlanMapVO entry
final DomainVlanMapVO domainVlanMapVO = new DomainVlanMapVO(domain.getId(), vlan.getId());
_domainVlanMapDao.persist(domainVlanMapVO);
} else if (podId != null) {
// This VLAN is pod-wide, so create a PodVlanMapVO entry
final PodVlanMapVO podVlanMapVO = new PodVlanMapVO(podId, vlan.getId());
podVlanMapDao.persist(podVlanMapVO);
}
So the range is dedicated to a domain only when it is not marked for systemvms. Error could be due to the fact that the range is not marked for systemvms but assigned to a domain which prevents listing its IPs from user.cloud_ip_address when CPVM is created.
@shwstppr
I don't think there should be an entry in cloud.domain_vlan_map when the range is reserved for system vms.
Takeing a look into your shared code i would think the same.
But the point is, that this is not the thing we want to achieve imho with the configuration - like displayed in the GUI:
We want a 'IP Range (vlan)' which is assigned to the domain 'Root' and dedicated for systemVMs. And simply asking where is the information stored, that this vlan shall be used for systemVM only? I didn't found a table regarding 'vlans' which contained this information. I only stumbled apon the entries for user_cloud_ip_address.forsystemvm = 1
.
But this information, looking into the sql queries and there results, will not be fetched during the process. The finding and choosing of vlans to check for IPs is done before the lookup if dedicated IPs are availeable in the gathered set of vlan's.
Creating a vlan (IP range) without the entry to either domain_vlan_map
or account_vlan_map
might be something like a "systempool" for the Zone which usage is controlled via the setting use.system.public.ips
.
Although the nameing is a bit odd, as 'system' is a "Account" itself. And there is no reference for a vlan with usage for systemVMs in the db in account_vlan_map
- Even if the GUI would make you think there is one:

This "zone" vlan is checked via the allocator sql-statements:
join search statement is com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.**data_center_id = 1** AND vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )
And is availeable as a result - however it is not used for allocating ip addresses as, i guess, the current settings avoid it.
Back to your questions regarding logs / outputs:
list vlanipranges
+---------+---------------+--------+--------------------------------------+-------------+--------------+-------------------+------------+--------------------------------------+-----------------+--------------------------------------+--------------------------------------+-------------+-------------+--------------------------------------+
| ACCOUNT | CIDR | DOMAIN | DOMAINID | ENDIP | FORSYSTEMVMS | FORVIRTUALNETWORK | GATEWAY | ID | NETMASK | NETWORKID | PHYSICALNETWORKID | STARTIP | VLAN | ZONEID |
+---------+---------------+--------+--------------------------------------+-------------+--------------+-------------------+------------+--------------------------------------+-----------------+--------------------------------------+--------------------------------------+-------------+-------------+--------------------------------------+
| system | 10.129.0.0/25 | ROOT | 0676360f-1d87-11ec-83ce-525400c9c662 | 10.129.0.19 | true | true | 10.129.0.1 | adc15744-9b35-4bea-81f0-1fdbd5b544f3 | 255.255.255.128 | 1b0bf965-a391-478d-833d-3e37e4a15eb8 | dec372e0-2fa3-436b-9265-8caba82f0620 | 10.129.0.10 | vlan://1000 | 48e2e928-3300-43b5-8e3a-d9b358022502 |
| | 10.129.0.0/25 | ROOT | 0676360f-1d87-11ec-83ce-525400c9c662 | 10.129.0.29 | false | true | 10.129.0.1 | c52e9946-a8db-4d66-819f-ee11dfefbb37 | 255.255.255.128 | 1b0bf965-a391-478d-833d-3e37e4a15eb8 | dec372e0-2fa3-436b-9265-8caba82f0620 | 10.129.0.20 | vlan://1000 | 48e2e928-3300-43b5-8e3a-d9b358022502 |
| xxxxxx | 10.129.0.0/25 | xxxxxx | 8cbb6645-0f0f-4c25-a24d-101d1426bb8f | 10.129.0.49 | false | true | 10.129.0.1 | 1031a85e-35d9-4c81-aee2-6d43ed1185ae | 255.255.255.128 | 1b0bf965-a391-478d-833d-3e37e4a15eb8 | dec372e0-2fa3-436b-9265-8caba82f0620 | 10.129.0.30 | vlan://1000 | 48e2e928-3300-43b5-8e3a-d9b358022502 |
+---------+---------------+--------+--------------------------------------+-------------+--------------+-------------------+------------+--------------------------------------+-----------------+--------------------------------------+--------------------------------------+-------------+-------------+--------------------------------------+
This output leads to the question: Where is the reference stored, that the Account 'system' has a mapped vlan?
Currently i would think the only way to get this information is to look into user_cloud_ip_address.forsystemvm = 1
. Or there is a table i haven't found yet where this information can be stored...Where ever it is, performing the SQL Queries from the log on my db, these 'value' isn't represented at all.
logs/API call for adding the new ip range
2022-09-05 13:15:12,138 DEBUG [c.c.a.ApiServlet] (qtp961160488-2321:ctx-a8d73839) (logid:e4b1e254) ===START=== 192.126.0.168 -- GET gateway=10.129.0.1&netmask=255.255.255.128&startip=10.129.0.50&endip=10.129.0.59&zoneId=48e2e928-3300-43b5-8e3a-d9b358022502&vlan=1000&forsystemvms=true&forvirtualnetwork=true&command=createVlanIpRange&response=json
2022-09-05 13:15:12,150 DEBUG [c.c.a.ApiServer] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) CIDRs from which account 'Acct[60be55c5-1d8a-11ec-83ce-525400c9c662-admin] -- Account {"id": 2, "name": "admin", "uuid": "60be55c5-1d8a-11ec-83ce-525400c9c662"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-09-05 13:15:12,164 DEBUG [c.c.c.ConfigurationManagerImpl] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) Acquiring lock for committing vlan
2022-09-05 13:15:12,167 DEBUG [c.c.c.ConfigurationManagerImpl] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) Access granted to Acct[60be55c5-1d8a-11ec-83ce-525400c9c662-admin] -- Account {"id": 2, "name": "admin", "uuid": "60be55c5-1d8a-11ec-83ce-525400c9c662"} to zone:1 by AffinityGroupAccessChecker
2022-09-05 13:15:12,173 DEBUG [c.c.c.ConfigurationManagerImpl] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) Saving vlan range Vlan[vlan://1000|10.129.0.1|255.255.255.128|null|null|10.129.0.50-10.129.0.59|null|200]
2022-09-05 13:15:12,208 DEBUG [c.c.a.ApiServlet] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) ===END=== 192.126.0.168 -- GET gateway=10.129.0.1&netmask=255.255.255.128&startip=10.129.0.50&endip=10.129.0.59&zoneId=48e2e928-3300-43b5-8e3a-d9b358022502&vlan=1000&forsystemvms=true&forvirtualnetwork=true&command=createVlanIpRange&response=json
While working on this i stumbled upon the strictness setting for the Management IPs... and i guess one might find the same thing there.
I don't think this is critical/blocker. I tried UI. When For system VMs
is selected in the UI it correctly marks the IPs from the range for system VMs even when domain is selected (also with API, https://github.com/apache/cloudstack/issues/6698#issuecomment-1236621729). Traditionally, I don't see any linkage between domain and system VMs. It would be better to discuss and make changes for it if needed in the next release. Moving to the next milestone
@shwstppr
Just to be shure:
And with system.vm.public.reservation.mode.strictness
= "true" and use.system.public.ips
= "false" the deployment worked for you?
Have you more then one Domain configured in your setting?
As i said i am just curious as this configuration has been working until i upgraded twards 4.17.0.1. Might be that something went wrong during the upgrade of the Management-Server. However from the logs of the SQL - statements everything is working as one would expected from a logical point.
When For system VMs is selected in the UI it correctly marks the IPs from the range for system VMs even when domain is selected (also with API, https://github.com/apache/cloudstack/issues/6698#issuecomment-1236621729).
It does indeed - but that doesn't help when during the allocationprocess the group of vlans gathered for a domain isn't containing the vlan-id, which is used to reference to the IP range (IP - addresses) dedicated to systemVMs. From the GUI point of view literally nothing has changed after the upgrade and everything is shown correctly.
Maybe you could share a log of the deploying sequence with "trace"-logging so that i can look up to it if they differ ?
Traditionally, I don't see any linkage between domain and system VMs.
At least from the overall architecture in my understanding this is going to be relevant if one is working with mulitple zones as well as "Dedicated Zones" for different Domains / Accounts - depending of course on what one wants to achieve.
@nvazquez just mentioning you here as you asked me to raise the issue and i don't know if i informed you about this on the mailinglist
Thanks @Hudratronium - I will book some time to try replicating it for an upgraded environment
@nvazquez Did you found some time to look into this? As we are planning our schedule for the next CS Upgrades, i just wanted to be shure to provide valid information if needed.
@nvazquez Just asking if you might see any chance to take a look into this topic? As we are planning the next upgrade, i won't be able to give informations for this CS Version after the end of the month.
@shwstppr @nvazquez this is critical for 4.17.2.0, it is reproducible?
@rohityadavcloud I could not reproduce it myself. Maybe @nvazquez can tell better
@Hudratronium is this still an issue? Are new dedications possible?
@DaanHoogland As we haven't upgraded to a newer release - no, no update on this. Still the same behaviour as well as the above mentionend SQL statements.
@Hudratronium it is normal you cannot find related records in domain_vlan_map or account_vlan_map.
can you share the result of the following mysql query ?
select public_ip_address,state,forsystemvms,user_ip_address.removed from user_ip_address join vlan on user_ip_address.vlan_db_id = vlan.id and vlan.uuid='adc15744-9b35-4bea-81f0-1fdbd5b544f3';
hi @weizhouapache
here the output of the sql-statement you requested:
public_ip_address | state | forsystemvms | removed |
---|---|---|---|
10.129.0.10 | Free | 1 | |
10.129.0.11 | Free | 1 | |
10.129.0.12 | Free | 1 | |
10.129.0.13 | Free | 1 | |
10.129.0.14 | Free | 1 | |
10.129.0.15 | Free | 1 | |
10.129.0.16 | Free | 1 | |
10.129.0.17 | Free | 1 | |
10.129.0.18 | Free | 1 | |
10.129.0.19 | Free | 1 |
Which are the IPs intended to use for systemVM's public usage
looks good @Hudratronium
What about the following query? (copied from your log) If it returns empty, try without some of the conditions
SELECT user_ip_address.* FROM user_ip_address INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
@weizhouapache Tried it and results are as to be expected:
SELECT user_ip_address.* FROM user_ip_address INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
Return empty - no results
Removing the 'user_ip_address.vlan_db_id IN (3,10,13,14)' condition does the trick and the intended IP-Range is given back as a result.
@Hudratronium so it looks the intended Ips (10.129.0.10-10.129.0.19) donot belong to the vlans (3,10,13,14).
the vlans are returned by
SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL );
if the intended vlan is not dedicated to an account (check account_vlan_map table), the intended vlan might be dedicated to domain. please check the domain_vlan_map.
The Query returns all vlans which are exclusiv for the usage of systemvms - numbers '9'we and '17' and are shown in the gui as 'domain = root' and 'account = system'. Like the screenshot in the post above https://github.com/apache/cloudstack/issues/6698#issuecomment-1237019304 These VLANs are not present - neither in 'account_vlan_map' nor 'domain_vlan_map'. And as you wrote above - that is to be expected.
However - as written beforehand (https://github.com/apache/cloudstack/issues/6698#issuecomment-1237019304) - That's where i find the concept or nameing a bit confusing. Looking at the sql from my instance there is currently no way that this is going to work as intended and the nameing in the gui itself is to some degree misleading.
The Query returns all vlans which are exclusiv for the usage of systemvms - numbers '9'we and '17' and are shown in the gui as 'domain = root' and 'account = system'. Like the screenshot in the post above https://github.com/apache/cloudstack/issues/6698#issuecomment-1237019304 These VLANs are not present - neither in 'account_vlan_map' nor 'domain_vlan_map'. And as you wrote above - that is to be expected.
However - as written beforehand (#6698 (comment)) - That's where i find the concept or nameing a bit confusing. Looking at the sql from my instance there is currently no way that this is going to work as intended and the nameing in the gui itself is to some degree misleading.
thanks @Hudratronium , for all the information.
I can reproduce the issue if (1) use.system.public.ips = true (2) system.vm.public.ip.reservation.mode.strictness = true (3) there is a dedicated IP range for ROOT domain (4) there is a dedicated IP range for SystemVMs
Your comment in https://github.com/apache/cloudstack/issues/6698#issuecomment-1236740037 is correct. We need to add a record in account_vlan_map (for ROOT/system) or change the logic of picking up a public ip address. I will look into it.
ping @shwstppr @weizhouapache @DaanHoogland
@rohityadavcloud PR #7144 has been added to 4.18.1.0 milestone
I can reproduce the issue if (1) use.system.public.ips = true (2) system.vm.public.ip.reservation.mode.strictness = true (3) there is a dedicated IP range for ROOT domain (4) there is a dedicated IP range for SystemVMs
Your comment in https://github.com/apache/cloudstack/issues/6698#issuecomment-1236740037 is correct. We need to add a record in account_vlan_map (for ROOT/system) or change the logic of picking up a public ip address. I will look into it.
@DaanHoogland I am able to reproduce the issue.
notes
- use.system.public.ips = false
- restart management server after global configuration changes (fixed by https://github.com/apache/cloudstack/pull/7909)
fixed by #7144