cloud-config provided in OpenStack vendor data is ignored

[frct1]

Bug report

I'm developing open source project https://github.com/ib-systems/openstack-consul-dynamic-vendordata that will help to easily provide vendor data to OpenStack instances using Consul KV. This is useful when operator works with a lot of independent geographically distributed clouds but store instances vendor data in distributed Consul KV used by billing software for example that unifying managing all of regions in the same interface.

Current cloud-init behavior expect cloud-init key to present at vendor_data2.json to run provided #cloud-config (docs link). But providing #cloud-config as string is not possible in current vendor_data2.json DynamicJSON provider due to OpenStack Nova component expect JSON encoded response from vendor_data2.json provider. Log line when i tried to return string contains #cloud-config from vendor data service.

Error from dynamic vendordata service cloud-init at http://10.10.10.10:8000/ocdv: Expecting value: line 1 column 1 (char 0): json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 

raised at this line

I've created blueprint with possible workaround at Nova tracker but not sure that it will be accepted. Anyway this have to be solved after years of misconcerns.

Steps to reproduce the problem

Configure nova-api to request Dynamicvendor_data2.json

[api]
vendordata_providers=DynamicJSON
vendordata_dynamic_targets=testing@http://10.10.10.10:8888

Following nova specs vendor_data2,json will be returned in format as

{
  "testing": {
    "cloud-init": "#cloud-config\nhostname: ocdvworks",
    "region":"eu-1"
  }
}

Current dynamic vendordata on Consul KV is store in next format:

{
    "cloud-init": "#cloud-config\nhostname: ocdvworks", 
    "region":"eu-1" // ignore region here, this is used internally, will be ignored in next samples
}

Following OpenStack specs there is could be a lot of different as called vendor data providers that could be used for licensing, software metadata and so on.

it works, but...

I've changed vendordata_dynamic_targets to vendordata_dynamic_targets=cloud-init@http://10.10.10.10:8888 in order to have cloud-init key in vendor_data2.json and then monkey-patched nova-api's metadata code inside docker container at this line to return plain text. So now i have vendor_data2.json looks like

{
    "cloud-init": "#cloud-config\nhostname: ocdvworks" // returned as plain text from Consul KV that contain just #cloud-config, not json
}

then as result cloud-init runs provided #cloud-config absolutely correct with no issues at all.

I would like to propose possible solution for cloud-init to merge all of #cloud-config provided from all of providers and then continue basic behavior (override with user-data if provided or go ahead in processing pipeline). In case only one key is presented in vendor_data2.json like:

{
  "testing": {
    "cloud-init": "#cloud-config\nhostname: ocdvworks"
  }
}

I guess this is most used case for vendors) simply extract #cloud-config from cloud-init key (if provided).

Environment details

• Cloud-init version: 21.3, 24.1.3 tried, no success
• Operating System Distribution: Ubuntu
• Cloud provider, platform or installer type: OpenStack Zed (vanilla)

cloud-init logs

None. Only log seen when JSON provided within cloud-init key in vendor_data2.json with "wrong format message".

refs

Openstack's vendor_data2.json is not handled openstack: read the dynamic metadata group vendor_data2.json

[holmanb]

Current cloud-init behavior expect cloud-init key to present at vendor_data2.json to run provided #cloud-config (docs link). But providing #cloud-config as string is not possible in current vendor_data2.json DynamicJSON provider due to OpenStack Nova component expect JSON encoded response from vendor_data2.json provider. Log line when i tried to return string contains #cloud-config from vendor data service.

Error from dynamic vendordata service cloud-init at http://10.10.10.10:8000/ocdv: Expecting value: line 1 column 1 (char 0): json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

raised at this line

I've created blueprint with possible workaround at Nova tracker but not sure that it will be accepted. Anyway this have to be solved after years of misconcerns.

This sounds like a bug in a Nova feature to me (or maybe even just a Nova feature request). I'm not sure what cloud-init is supposed to do about this.

[holmanb]

Unless I misunderstand and cloud-init is actually unable to use the cloud-init key in vendor_data2.json that is provided by Nova, I'm inclined to close this as "won't fix", since cloud-init appears to be working as intended.

@frct1 Am I missing something?

[frct1]

This sounds like a bug in a Nova feature to me (or maybe even just a Nova feature request). I'm not sure what cloud-init is supposed to do about this.

Same thing. Since cloud-init works fine if cloud-init key in vendor_data2.json is presented as string it works as expected but there is discrepancy between Nova docs and cloud-init logic 🤷‍♂️ I've already created proposal to Nova to let vendor_data2.json JSON object contain cloud-init key as plain string.

[SeanMooney]

reviewing this form a nova perspective the cloud-init integration is not compatible with nova implemation and this looks to be a bug in cloud-init

nova's implementation of vendor data expect the content of sopenstack/2016-10-06/vendor_data2.json

to be a single json object containing a key per dynamic vendor data service with the content of the respocne for that backend being a valid json respocne

sopenstack/2016-10-06/vendor_data2.json

in this case we have two vender data backedn

vendordata_dynamic_targets=[ 'testing@http://127.0.0.1:123', 'hamster@http://127.0.0.1:123' ],

when we parst vendor_data2.json we expect to have two keys in the reultant object testing and hamster and each is a dict containing addition keys

that means we would expect the vendor data to look like

{ "testing": { "cloud-init": "#cloud-config\nhostname: ocdvworks", "region":"eu-1" } }

not

{ "cloud-init": "#cloud-config\nhostname: ocdvworks", "region":"eu-1" }

so https://cloudinit.readthedocs.io/en/latest/reference/datasources/openstack.html#vendor-data is not compatible with OpenStack and never has been as far as i can tell at a quick glance.

[SeanMooney]

just to clarify

nova provide the following format in the vendor data

{
"<provider 1>": {<provider 1 repsonce as nested dict>},
"<provider 2>": {<provider 2 repsonce as nested dict>}
}

i.e.

{
  "testing": {
    "cloud-init": "#cloud-config\nhostname: ocdvworks",
    "region":"eu-1"
  }
}

we require vendor data backends to provide a json response which we then embed in the vendor_data files keyed by the backend

so form my perspective there is a bug in at least the Cloudinit docs as the example provided in https://cloudinit.readthedocs.io/en/latest/reference/datasources/openstack.html#vendor-data

{"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}

is a valid reponce form a backend but its not a valid example fo the content of vendor_data2.json as produced by nova.

it would look like this

{
"my_backend":{"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}
}

supporting a different format that allowed adding content directly to the root object would be a file format change as consumer of the file would not be able to rely on the deserialised json being a dict of dicts

there may be one way to hack this to work today if you named the backend 'cloud-init' https://docs.openstack.org/nova/latest/configuration/config.html#api.vendordata_dynamic_targets i.e.

[API] 
vendordata_dynamic_targets=['cloud-init:string@http://127.0.0.1:123']

and the backend provider returned just the cloud-init content so literally

"#cloud-config\npackage_upgrade: True\npackages:\n - htop"

then that might work a single string is trecnially a valid json value

"#cloud-config\npackage_upgrade: True\npackages:\n - htop"

then it might work even if that is technically not intended to work today

[frct1]

@SeanMooney suggested a possible way to hack this while explicit support on provider data type is not implemented.

from oslo_serialization import jsonutils
res = "#cloud-config\n..."
jsonutils.dumps(res)

Following this way cloud-config returned as a raw string and counts as valid JSON.