ansible-collections
gcp_compute_disk doesn't work with a registered gcp_compute_snapshot source
SUMMARY
when gcp_compute_snapshot is used to obtain a source snapshot and then given to gcp_compute_disk via source_snapshot arg, the disk creation fails with:
"msg": "GCP returned error: {'error': {'code': 400, 'message': "Invalid value for field 'resource.sourceSnapshot':
'********/disks/********'. Unexpected resource collection 'disks'.", 'errors': [{'message': "Invalid value for field 'resource.sourceSnapshot':'********/disks/********'. Unexpected resource collection 'disks'.", 'domain': 'global', 'reason': 'invalid'}]}}"
it seems to only work if supplied using selfLink dictionary instead of the registered variable. This contradicts the documentation:
The source snapshot used to create this disk. You can provide this as a partial or full URL to the resource. This field represents a link to a Snapshot resource in GCP. It can be specified in two ways. First, you can place a dictionary with key 'selfLink' and value of your resource's selfLink Alternatively, you can add
register: name-of-resourceto a gcp_compute_snapshot task and then set this source_snapshot field to "{{ name-of-resource }}"
The second way does not seem to work. Only the first method (providing selfLink) seemed to work. Bizarrely, it seems to obfuscate its own parameters in the output, I'm not sure if it's giving these obfuscated to the API? (also not sure why these would be set as no_log in the first place?):
The full traceback is: [236/54831]
File "/tmp/ansible_gcp_compute_disk_payload_xt__r__b/ansible_gcp_compute_disk_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 312, in raise_for_status
response.raise_for_status()
File "/home/scott/src/setup/venv/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
fatal: [localhost]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"auth_kind": "serviceaccount",
"description": null,
"disk_encryption_key": null,
"env_type": null,
"labels": null,
"licenses": null,
"name": "mydisk",
"physical_block_size_bytes": null,
"project": "myproject",
"scopes": [
"https://www.googleapis.com/auth/compute"
],
"service_account_contents": null,
"service_account_email": null,
"service_account_file": "mykey.json",
"size_gb": null,
"source_image": null,
"source_image_encryption_key": null,
"source_snapshot": {
"changed": true,
"creationTimestamp": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"failed": false,
"id": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"kind": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"labelFingerprint": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"lastAttachTimestamp": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"lastDetachTimestamp": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"name": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"physicalBlockSizeBytes": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"selfLink": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"sizeGb": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"status": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"type": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"users": [
"VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
],
"zone": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
},
"source_snapshot_encryption_key": null,
"state": "present",
"type": "pd-ssd",
"zone": "us-west2-b"
}
},
"msg": "GCP returned error: {'error': {'code': 400, 'message': \"Invalid value for field 'resource.sourceSnapshot': '********/disks/********'. Unexpected resource collection 'disks'.\", 'errors': [
{'message': \"Invalid value for field 'resource.sourceSnapshot': '********/disks/********'. Unexpected resource collection 'disks'.\", 'domain': 'global', 'reason': 'invalid'}]}}"
}
adding no_log: false to either or both of gcp_compute_snapshot or gcp_compute_disk did not seem to have any effect (still obfuscated params, still fails).
Note: strangely, if I run the playbook twice in a row with no changes, it will end up succeeding after the second pass: first run will succeed creating snapshot, but fail making a disk from it (due to aforementioned error). The second run already has an existing snapshot and passes it over (idempotently), registers the snapshot var, and passes to gcp_compute_disk, which that time works. If both tasks (snapshot and disk create) happen in the same ansible-playbook run, the snapshot succeeds, but disk create fails every time.
Also note, even when providing the snapshot via selfLink (so it finishes), the sourceSnapshot from the result is printed as VALUE_SPECIFIED_IN_NO_LOG_PARAMETER, and the module_args['source_snapshot'] field is shown as selfLink: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER.
ISSUE TYPE
- Bug Report
COMPONENT NAME
gcp_compute_disk
ANSIBLE VERSION
ansible 2.10.7
config file = /home/scott/src/setup/.ansible.cfg
configured module search path = ['/home/scott/src/setup/lib']
ansible python module location = /home/scott/src/setup/venv/lib/python3.8/site-packages/ansible
executable location = /home/scott/src/setup/venv/bin/ansible
python version = 3.8.8 (tags/v3.8.8:024d8058b0, Mar 2 2021, 18:15:50) [GCC 9.3.0]
CONFIGURATION
stock
OS / ENVIRONMENT
linux
STEPS TO REPRODUCE
# ...
- name: snapshot_dev_disk
gcp_compute_snapshot:
name: '{{snapname}}'
source_disk:
name: '{{diskdev}}'
# ...
register: snapdisk
- name: create_disk_from_snapshot
gcp_compute_disk:
name: mydisk
type: '{{disktype}}'
source_snapshot: '{{snapdisk}}'
# ...
# ...
I got the same problem using other gcp modules - e.g. gcp_compute_subnetwork using the registered return from gcp_compute_network for network parameter. Code like this: ` - name: create network google.cloud.gcp_compute_network: ... auth_kind: serviceaccount service_account_contents: ... state: present register: new_network
google.cloud.gcp_compute_subnetwork:
...
network: "{{ new_network }}"
`
Did you use service_account_contents in your playbook, @smemsh? I'm asking, because i found a no_log=True in the gcp_utils.py (line ~286) for service_account_contents. So it seems that all values inside the service_account_contents is obfuscated, also within variables. If you set no_log=False in this file, everything works fine.
@Tobrek no, our playbooks use service_account_file, not service_account_contents. I see the line you're talking about which sets no_log=True in gcp_utils.py, but service_account_file does not have it. Maybe the default is True for some reason.
The other really interesting thing to me (already noted in the description) is that, if the play creates the snapshot and then the disk in same play run, the disk create fails, but if the same play is run again and the snapshot already exists, it will work. I bet just adding the task twice in a row in the playbook would work every time. Maybe it gets the data structure from the API and uses it as-is in already created case, but when it gets it from itself upon creation, the values are obfuscated.
That's interesting that you found this bug in other modules as well. There's clearly something going on. Too bad this module is basically unmaintained and no developer will ever see these bug reports. We can try to fix it in MagicModules ourselves with a PR, that appears to be only way to get any bugs fixed in this collection. Our shop has decided to use the libcloud community versions where we can, since the official Google collection does not appear to be maintained.
Came across this ticket as I was struggling with the same issue. I've verbosely printed out the output from registering a gcp_compute_snapshot task, and it turns out that somehow the output is the disk that's being snapshotted, rather than the snapshot itself. Could anyone verify this?
changed: [localhost] => {
"changed": true,
"creationTimestamp": "2023-01-30T23:21:57.487-08:00",
"description": "*****",
"id": "*****",
"invocation": {
"module_args": {
"auth_kind": "application",
"description": null,
"env_type": null,
"labels": null,
"name": "*****snapshot",
"project": "*****",
"scopes": [
"https://www.googleapis.com/auth/compute"
],
"service_account_contents": null,
"service_account_email": null,
"service_account_file": null,
"snapshot_encryption_key": null,
"source_disk": {
"name": "*****disk"
},
"source_disk_encryption_key": null,
"state": "present",
"storage_locations": null,
"zone": "us-central1-c"
}
},
"kind": "compute#disk",
"labelFingerprint": "*****",
"lastAttachTimestamp": "2023-01-30T23:22:09.760-08:00",
"name": "*****disk",
"physicalBlockSizeBytes": "4096",
"selfLink": "https://www.googleapis.com/compute/v1/projects/*****/zones/us-central1-c/disks/*****disk",
"sizeGb": "5000",
"status": "READY",
"type": "https://www.googleapis.com/compute/v1/projects/*****/zones/us-central1-c/diskTypes/pd-standard",
"users": [
"https://www.googleapis.com/compute/v1/projects/*****/zones/us-central1-c/instances/*****"
],
"zone": "https://www.googleapis.com/compute/v1/projects/*****/zones/us-central1-c"
}
Came across this ticket as I was struggling with the same issue. I've verbosely printed out the output from registering a gcp_compute_snapshot task, and it turns out that somehow the output is the disk that's being snapshotted, rather than the snapshot itself. Could anyone verify this?
Do you have the task to help with verification?
with regards to the original issue: it does sound like obfuscation is the issue. The way wiring in the resource directly works is by pulling out the value of the selfLink and using that.
I do see that gcp_compute_disk actually has source_snapshot as no_log=True. That seems like a mistake, I can get a PR out.
