amazon.aws
amazon.aws copied to clipboard
Published
20 hours ago •
ansible-collections
ansible-collections
s3_object in copy mode cannot change metadata
Summary
The documentation suggests that metadata can be set when mode=copy but this doesn't appear to be the case. I cannot get metadata to be changed on the copy whatever I do.
This appears to be because the module doesn't override (or provide the option to override) the MetadataDirective parameter, which defaults to COPY (search for it in the boto3 docs).
I first observed this trying to copy an object while modifying its metadata (not shown here, but the copy always has the same metadata as its source). I suspect the reason is that objects have system defined metadata called Content-Type (visible in the console), which in essence is copied due to the directive, but this is speculation on my part.
Additionally, please consider updating the examples which show metadata as a comma separated string while the documentation lists it as a dict (which works as shown below).
Issue Type
Bug Report
Component Name
s3_object
Ansible Version
$ ansible --version
ansible [core 2.16.3]
config file = /home/andrei/work/tools/andrei/playbooks/ansible.cfg
configured module search path = ['/home/andrei/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.11/site-packages/ansible
ansible collection location = /home/andrei/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.11.7 (main, Jan 29 2024, 16:03:57) [GCC 13.2.1 20230801] (/usr/bin/python)
jinja version = 3.1.3
libyaml = True
Collection Versions
$ ansible-galaxy collection list
# /home/andrei/.ansible/collections/ansible_collections
Collection Version
---------------------------------------- -------
amazon.aws 7.2.0
ansible.netcommon 4.1.0
ansible.posix 1.5.1
ansible.utils 3.0.0
cloud.common 2.1.3
community.aws 7.1.0
community.crypto 2.10.0
community.docker 3.4.0
community.general 7.4.0
community.postgresql 3.3.0
community.routeros 2.5.0
kubernetes.core 2.4.0
# /usr/lib/python3.11/site-packages/ansible_collections
Collection Version
---------------------------------------- -------
amazon.aws 7.2.0
ansible.netcommon 5.3.0
ansible.posix 1.5.4
ansible.utils 2.12.0
ansible.windows 2.2.0
arista.eos 6.2.2
awx.awx 23.6.0
azure.azcollection 1.19.0
check_point.mgmt 5.2.2
chocolatey.chocolatey 1.5.1
cisco.aci 2.8.0
cisco.asa 4.0.3
cisco.dnac 6.10.2
cisco.intersight 2.0.7
cisco.ios 5.3.0
cisco.iosxr 6.1.1
cisco.ise 2.7.0
cisco.meraki 2.17.2
cisco.mso 2.5.0
cisco.nxos 5.3.0
cisco.ucs 1.10.0
cloud.common 2.1.4
cloudscale_ch.cloud 2.3.1
community.aws 7.1.0
community.azure 2.0.0
community.ciscosmb 1.0.7
community.crypto 2.17.1
community.digitalocean 1.26.0
community.dns 2.8.0
community.docker 3.7.0
community.general 8.3.0
community.grafana 1.7.0
community.hashi_vault 6.1.0
community.hrobot 1.9.0
community.library_inventory_filtering_v1 1.0.0
community.libvirt 1.3.0
community.mongodb 1.6.3
community.mysql 3.8.0
community.network 5.0.2
community.okd 2.3.0
community.postgresql 3.3.0
community.proxysql 1.5.1
community.rabbitmq 1.2.3
community.routeros 2.12.0
community.sap 2.0.0
community.sap_libs 1.4.2
community.sops 1.6.7
community.vmware 4.1.0
community.windows 2.1.0
community.zabbix 2.3.1
containers.podman 1.11.0
cyberark.conjur 1.2.2
cyberark.pas 1.0.25
dellemc.enterprise_sonic 2.4.0
dellemc.openmanage 8.7.0
dellemc.powerflex 2.1.0
dellemc.unity 1.7.1
f5networks.f5_modules 1.27.1
fortinet.fortimanager 2.3.1
fortinet.fortios 2.3.4
frr.frr 2.0.2
gluster.gluster 1.0.2
google.cloud 1.3.0
grafana.grafana 2.2.4
hetzner.hcloud 2.4.1
hpe.nimble 1.1.4
ibm.qradar 2.1.0
ibm.spectrum_virtualize 2.0.0
ibm.storage_virtualize 2.2.0
infinidat.infinibox 1.3.12
infoblox.nios_modules 1.6.1
inspur.ispim 2.2.0
inspur.sm 2.3.0
junipernetworks.junos 5.3.1
kubernetes.core 2.4.0
lowlydba.sqlserver 2.2.2
microsoft.ad 1.4.1
netapp.aws 21.7.1
netapp.azure 21.10.1
netapp.cloudmanager 21.22.1
netapp.elementsw 21.7.0
netapp.ontap 22.9.0
netapp.storagegrid 21.11.1
netapp.um_info 21.8.1
netapp_eseries.santricity 1.4.0
netbox.netbox 3.16.0
ngine_io.cloudstack 2.3.0
ngine_io.exoscale 1.1.0
openstack.cloud 2.2.0
openvswitch.openvswitch 2.1.1
ovirt.ovirt 3.2.0
purestorage.flasharray 1.26.0
purestorage.flashblade 1.15.0
purestorage.fusion 1.6.0
sensu.sensu_go 1.14.0
splunk.es 2.1.2
t_systems_mms.icinga_director 2.0.1
telekom_mms.icinga_director 1.35.0
theforeman.foreman 3.15.0
vmware.vmware_rest 2.3.1
vultr.cloud 1.12.1
vyos.vyos 4.1.0
wti.remote 1.0.5
AWS SDK versions
$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.34.34
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.34.34
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer
Configuration
$ ansible-config dump --only-changed
CONFIG_FILE() = /home/andrei/work/tools/andrei/playbooks/ansible.cfg
DEFAULT_HOST_LIST(/home/andrei/work/tools/andrei/playbooks/ansible.cfg) = ...
DEFAULT_ROLES_PATH(/home/andrei/work/tools/andrei/playbooks/ansible.cfg) = ...
EDITOR(env: EDITOR) = vim
INVENTORY_ENABLED(/home/andrei/work/tools/andrei/playbooks/ansible.cfg) = ['ini', 'constructed', 'script']
PAGER(env: PAGER) = less
OS / Environment
Arch Linux
Steps to Reproduce
- name: Test playbook
hosts: localhost
gather_facts: false
tasks:
- name: Create test bucket
amazon.aws.s3_bucket:
name: ansiblemetatest
state: present
- name: Create test object
amazon.aws.s3_object:
bucket: ansiblemetatest
object: nometa
mode: put
content: "some content"
- name: Copy and add metadata
amazon.aws.s3_object:
bucket: ansiblemetatest
object: "metacopy"
mode: copy
copy_src:
bucket: ansiblemetatest
object: nometa
metadata:
something: exists
version: "1.0.2"
- name: Create test object with metadata
amazon.aws.s3_object:
bucket: ansiblemetatest
object: withmeta
mode: put
content: "some content"
metadata:
something: exists
version: "1.0.2"
- name: Get object info
amazon.aws.s3_object_info:
bucket_name: ansiblemetatest
object_name: "{{ item }}"
loop:
- nometa
- metacopy
- withmeta
register: obj_info
- debug:
var: obj_info.results
Expected Results
metacopy should have its metadata set, like withmeta does.
Actual Results
PLAY [Test playbook] **********************************************************************************************************
TASK [Create test bucket] *****************************************************************************************************
changed: [localhost]
TASK [Create test object] *****************************************************************************************************
changed: [localhost]
TASK [Copy and add metadata] **************************************************************************************************
[WARNING]: PutObjectAcl operation : The bucket does not allow ACLs.
changed: [localhost]
TASK [Create test object with metadata] ***************************************************************************************
changed: [localhost]
TASK [Get object info] ********************************************************************************************************
ok: [localhost] => (item=nometa)
ok: [localhost] => (item=metacopy)
ok: [localhost] => (item=withmeta)
TASK [debug] ******************************************************************************************************************
ok: [localhost] => {
"obj_info.results": [
{
"ansible_loop_var": "item",
"changed": false,
"failed": false,
"invocation": {
"module_args": {
"access_key": null,
"aws_ca_bundle": null,
"aws_config": null,
"bucket_name": "ansiblemetatest",
"ceph": false,
"debug_botocore_endpoint_logs": false,
"dualstack": false,
"endpoint_url": null,
"object_details": null,
"object_name": "nometa",
"profile": "sandbox",
"region": null,
"secret_key": null,
"session_token": null,
"validate_certs": true
}
},
"item": "nometa",
"object_info": [
{
"object_data": {
"accept_ranges": "bytes",
"content_length": 12,
"content_type": "binary/octet-stream",
"e_tag": "\"9893532233caff98cd083a116b013c0b\"",
"last_modified": "2024-02-23T07:51:24+00:00",
"metadata": {},
"server_side_encryption": "AES256"
}
}
]
},
{
"ansible_loop_var": "item",
"changed": false,
"failed": false,
"invocation": {
"module_args": {
"access_key": null,
"aws_ca_bundle": null,
"aws_config": null,
"bucket_name": "ansiblemetatest",
"ceph": false,
"debug_botocore_endpoint_logs": false,
"dualstack": false,
"endpoint_url": null,
"object_details": null,
"object_name": "metacopy",
"profile": "sandbox",
"region": null,
"secret_key": null,
"session_token": null,
"validate_certs": true
}
},
"item": "metacopy",
"object_info": [
{
"object_data": {
"accept_ranges": "bytes",
"content_length": 12,
"content_type": "binary/octet-stream",
"e_tag": "\"9893532233caff98cd083a116b013c0b\"",
"last_modified": "2024-02-23T07:51:26+00:00",
"metadata": {},
"server_side_encryption": "AES256"
}
}
]
},
{
"ansible_loop_var": "item",
"changed": false,
"failed": false,
"invocation": {
"module_args": {
"access_key": null,
"aws_ca_bundle": null,
"aws_config": null,
"bucket_name": "ansiblemetatest",
"ceph": false,
"debug_botocore_endpoint_logs": false,
"dualstack": false,
"endpoint_url": null,
"object_details": null,
"object_name": "withmeta",
"profile": "sandbox",
"region": null,
"secret_key": null,
"session_token": null,
"validate_certs": true
}
},
"item": "withmeta",
"object_info": [
{
"object_data": {
"accept_ranges": "bytes",
"content_length": 12,
"content_type": "binary/octet-stream",
"e_tag": "\"9893532233caff98cd083a116b013c0b\"",
"last_modified": "2024-02-23T07:51:28+00:00",
"metadata": {
"something": "exists",
"version": "1.0.2"
},
"server_side_encryption": "AES256"
}
}
]
}
]
}
PLAY RECAP ********************************************************************************************************************
localhost : ok=6 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Code of Conduct
- [X] I agree to follow the Ansible Code of Conduct
