community.aws
community.aws copied to clipboard
ansible-collections
cloudfront_distribution: Type error when validating distribution origins: "'list' object has no attribute 'get'"
SUMMARY
I get an unexpected type error, for what I assume to be the "origins" argument to the community.aws.cloudfront_distribution module: Type error when validating distribution origins: "'list' object has no attribute 'get'"
Posted on StackExchange DevOps: https://devops.stackexchange.com/questions/13639/unexpected-type-error-in-ansible-community-aws-cloudfront-distribution-origins
ISSUE TYPE
- Bug Report
COMPONENT NAME
cloudfront_distribution
ANSIBLE VERSION
ansible 2.10.7
config file = /home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg
configured module search path = ['/home/bryant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/bryant/.pyenv/versions/vst-ansible-deployment/lib/python3.9/site-packages/ansible
executable location = /home/bryant/.pyenv/versions/vst-ansible-deployment/bin/ansible
python version = 3.9.1 (default, Feb 16 2021, 14:25:16) [GCC 9.3.0]
CONFIGURATION
DEFAULT_CALLBACK_WHITELIST(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = ['ansible.posix.timer']
DEFAULT_HOST_LIST(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = ['/home/bryant/src/api_guys/ansible-deploy-vst/inventory']
DEFAULT_PRIVATE_KEY_FILE(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = /home/bryant/.ssh/ansible-control-node_rsa
DEFAULT_REMOTE_USER(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = ubuntu
DEFAULT_VAULT_PASSWORD_FILE(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = /home/bryant/src/api_guys/ansible-deploy-vst/.vault_passwords
HOST_KEY_CHECKING(/home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg) = False
OS / ENVIRONMENT
control node: Ubuntu 20.04 LTS as PopOs, running ansible in a pyenv virtual environment managed node: localhost (since using aws modules); routes to an IAM user in our aws account with sufficient attached policies.
STEPS TO REPRODUCE
I run a role, containing the task below, from my main playbook; boto profiles are configured in ~/.aws/credentials with correct keys, secrets and regions in ~/.aws.config.
From the main playbook site.yml:
- name: Perform AWS tasks for a full deployment
hosts: localhost
tags: aws_full_deployment
gather_facts: yes
vars:
boto_profile: "{{ vst_gov_boto_profile }}"
tasks:
- name: Conditionally run the AWS Full Deployment role
import_role:
name: aws_full_deployment
when: deployment_type == "full_deployment"
The task in the role presenting this issue:
- name: Create Cloudfront distribution for the interface and api reverse proxy
community.aws.cloudfront_distribution:
profile: "{{ vst_nongov_boto_profile }}"
state: present
enabled: yes
caller_reference: "{{ cloudfront_domain_alias }}"
alias: "{{ cloudfront_domain_alias }}"
aliases:
- "{{ cloudfront_domain_alias }}"
origins:
# reverse proxy for api
- id: "{{ base_domain }}"
domain_name: "{{ base_domain }}"
custom_origin_config:
http_port: 80
https_port: 443
origin_keepalive_timeout: 5
origin_protocol_policy: "https-only"
origin_read_timeout: 10
origin_ssl_protocols:
- "TLSv1.2"
# s3 bucket for interface
- id: "{{ s3_bucket_www_domain_name }}"
domain_name: "{{ s3_bucket_domain_name }}"
custom_origin_config:
http_port: 80
https_port: 443
origin_keepalive_timeout: 5
origin_protocol_policy: "http-only"
origin_read_timeout: 10
origin_ssl_protocols:
- "TLSv1.2"
default_cache_behavior:
# this contains as many settings as possible that are able to be
# replicated from the UI option caching policy
# "Managed-CachingOptimized"; some are guesses, which may need to be
# changed if we notice performance issues
target_origin_id: "{{ s3_bucket_domain_name }}"
forwarded_values:
allowed_methods:
- GET
- HEAD
items:
- GET
- HEAD
compress: yes
cookies:
forward: none
default_ttl: 86400
max_ttl: 31536000
min_ttl: 1
query_string: no
smooth_streaming: no
trusted_signers:
enabled: no
viewer_protocol_policy: "redirect-to-https"
cache_behaviors:
- path_pattern: "api/*"
target_origin_id: "{{ base_domain }}"
forwarded_values:
allowed_methods:
- GET
- HEAD
items:
- GET
- HEAD
- OPTIONS
- PUT
- POST
- PATCH
- DELETE
compress: no
cookies:
forward: all
default_ttl: 0
headers:
- Authorization
- Referer
- User-Agent
max_ttl: 0
min_ttl: 0
query_string: yes
query_string_cache_keys:
- '*'
smooth_streaming: no
trusted_signers:
enabled: no
viewer_protocol_policy: "redirect-to-https"
- path_pattern: "testapi*"
target_origin_id: "{{ base_domain }}"
forwarded_values:
allowed_methods:
- GET
- HEAD
items:
- GET
- HEAD
compress: no
cookies:
forward: all
default_ttl: 0
headers:
- Authorization
- User-Agent
max_ttl: 0
min_ttl: 0
query_string: yes
query_string_cache_keys:
- '*'
smooth_streaming: no
trusted_signers:
enabled: no
viewer_protocol_policy: "redirect-to-https"
custom_error_responses:
- error_caching_min_ttl: 10
error_code: 403
response_code: 200
response_page_path: "/index.html"
- error_caching_min_ttl: 10
error_code: 404
response_code: 200
response_page_path: "/index.html"
viewer_certificate:
acm_certificate_arn: "{{ nongov_acm_cert_upload_result.certificate.arn }}"
cloudfront_default_certificate: no
minimum_protocol_version: "TLSv1.2_2019"
ssl_support_method: "sni-only"
http_version: http2
ipv6_enabled: no
tags:
Name: "{{ generic_name_tag }}"
wait: yes
default_root_object: "index.html"
price_class: "{{ cloudfront_price_class }}"
comment: "Cloudfront distribution for {{ base_interface_domain }} interface and API reverse proxy"
notify: primary_cloudfront_invalidation
tags: cloudfront
EXPECTED RESULTS
I expect a cloudfront distribution to be created; I've review my syntax on the "origins" argument many times.
ACTUAL RESULTS
I get the type error explained above. It seems that perhaps ansible is expecting "origins" to be a dictionary, not a list, as described in the documentation?
TASK [aws_full_deployment : Create Cloudfront distribution for the interface and api reverse proxy] ***
task path: /home/bryant/src/api_guys/ansible-deploy-vst/roles/aws_full_deployment/tasks/main.yml:501
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: bryant
<127.0.0.1> EXEC /bin/sh -c 'echo ~bryant && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/bryant/.ansible/tmp `"&& mkdir "` echo /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877 `" && echo ansible-tmp-1616796020.2343066-177081-217806658917877="` echo /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877 `" ) && sleep 0'
Using module file /home/bryant/.ansible/collections/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py
<127.0.0.1> PUT /home/bryant/.ansible/tmp/ansible-local-175967cbvvbt8u/tmpamrug82t TO /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/ /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/bryant/.pyenv/versions/vst-ansible-deployment/bin/python /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_community.aws.cloudfront_distribution_payload_vqd_hi9m/ansible_community.aws.cloudfront_distribution_payload.zip/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py", line 1665, in validate_origins
File "/tmp/ansible_community.aws.cloudfront_distribution_payload_vqd_hi9m/ansible_community.aws.cloudfront_distribution_payload.zip/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py", line 1729, in validate_origin
AttributeError: 'list' object has no attribute 'get'
fatal: [localhost]: FAILED! => {
"boto3_version": "1.16.25",
"botocore_version": "1.19.25",
"changed": false,
"invocation": {
"module_args": {
"alias": "v6-full-deployment.icloudcms.com",
"aliases": [
"v6-full-deployment.icloudcms.com"
],
"aws_access_key": null,
"aws_ca_bundle": null,
"aws_config": null,
"aws_secret_key": null,
"cache_behaviors": [
{
"forwarded_values": {
"allowed_methods": [
"GET",
"HEAD"
],
"compress": false,
"cookies": {
"forward": "all"
},
"default_ttl": 0,
"headers": [
"Authorization",
"Referer",
"User-Agent"
],
"items": [
"GET",
"HEAD",
"OPTIONS",
"PUT",
"POST",
"PATCH",
"DELETE"
],
"max_ttl": 0,
"min_ttl": 0,
"query_string": true,
"query_string_cache_keys": [
"*"
],
"smooth_streaming": false,
"trusted_signers": {
"enabled": false
},
"viewer_protocol_policy": "redirect-to-https"
},
"path_pattern": "api/*",
"target_origin_id": "api.v6-full-deployment.icloudcms.com"
},
{
"forwarded_values": {
"allowed_methods": [
"GET",
"HEAD"
],
"compress": false,
"cookies": {
"forward": "all"
},
"default_ttl": 0,
"headers": [
"Authorization",
"User-Agent"
],
"items": [
"GET",
"HEAD"
],
"max_ttl": 0,
"min_ttl": 0,
"query_string": true,
"query_string_cache_keys": [
"*"
],
"smooth_streaming": false,
"trusted_signers": {
"enabled": false
},
"viewer_protocol_policy": "redirect-to-https"
},
"path_pattern": "testapi*",
"target_origin_id": "api.v6-full-deployment.icloudcms.com"
}
],
"caller_reference": "v6-full-deployment.icloudcms.com",
"comment": "Cloudfront distribution for v6-full-deployment.icloudcms.com interface and API reverse proxy",
"custom_error_responses": [
{
"error_caching_min_ttl": 10,
"error_code": 403,
"response_code": 200,
"response_page_path": "/index.html"
},
{
"error_caching_min_ttl": 10,
"error_code": 404,
"response_code": 200,
"response_page_path": "/index.html"
}
],
"debug_botocore_endpoint_logs": false,
"default_cache_behavior": {
"forwarded_values": {
"allowed_methods": [
"GET",
"HEAD"
],
"compress": true,
"cookies": {
"forward": "none"
},
"default_ttl": 86400,
"items": [
"GET",
"HEAD"
],
"max_ttl": 31536000,
"min_ttl": 1,
"query_string": false,
"smooth_streaming": false,
"trusted_signers": {
"enabled": false
},
"viewer_protocol_policy": "redirect-to-https"
},
"target_origin_id": "v6-full-deployment-s3-bucket.s3-website-us-gov-west-1.amazonaws.com"
},
"default_origin_domain_name": null,
"default_origin_path": null,
"default_root_object": "index.html",
"distribution_id": null,
"e_tag": null,
"ec2_url": null,
"enabled": true,
"http_version": "http2",
"ipv6_enabled": false,
"logging": null,
"origins": [
{
"custom_headers": {
"quantity": 0
},
"custom_origin_config": {
"h_t_t_p_port": 80,
"h_t_t_p_s_port": 443,
"origin_keepalive_timeout": 5,
"origin_protocol_policy": "https-only",
"origin_read_timeout": 10,
"origin_ssl_protocols": [
"TLSv1.2"
]
},
"domain_name": "api.v6-full-deployment.icloudcms.com",
"id": "api.v6-full-deployment.icloudcms.com",
"origin_path": ""
},
{
"custom_origin_config": {
"http_port": 80,
"https_port": 443,
"origin_keepalive_timeout": 5,
"origin_protocol_policy": "http-only",
"origin_read_timeout": 10,
"origin_ssl_protocols": [
"TLSv1.2"
]
},
"domain_name": "v6-full-deployment-s3-bucket.s3-website-us-gov-west-1.amazonaws.com",
"id": "v6-full-deployment-s3-bucket-www.s3-website-us-gov-west-1.amazonaws.com"
}
],
"price_class": "PriceClass_100",
"profile": "vst_nongov",
"purge_aliases": false,
"purge_cache_behaviors": false,
"purge_custom_error_responses": false,
"purge_origins": false,
"purge_tags": false,
"region": null,
"restrictions": null,
"security_token": null,
"state": "present",
"tags": {
"Name": "v6-full-deployment"
},
"validate_certs": true,
"viewer_certificate": {
"acm_certificate_arn": "xxxxxxxxxxOBFUSCATED",
"cloudfront_default_certificate": false,
"minimum_protocol_version": "TLSv1.2_2019",
"ssl_support_method": "sni-only"
},
"wait": true,
"wait_timeout": 1800,
"web_acl_id": null
}
},
"msg": "Error validating distribution origins: 'list' object has no attribute 'get'"
}
Files identified in the description: None
If these files are inaccurate, please update the component name section of the description or use the !component bot command.
Files identified in the description:
If these files are inaccurate, please update the component name section of the description or use the !component bot command.
Hello, can someone take a look at this? If I'm not making a syntax error, it seems like a rather significant issue (a list object being asked to "get" implies that it's a dictionary). Thank you
https://github.com/flowerysong/ansible-community.aws/commit/d48b9b9639e17de9bd50e6d9ac339f294b5e6952 might fix this, but tests for this module are disabled in CI and I probably won't get around to testing it manually any time soon.
@Life5ign I've had a quick test of @flowerysong's initial change. Building upon that I've submitted #540.
Since I'm still seeing idempotency issues (one of which looks like it mangles existing origins) the PR isn't ready to merge yet.
You can try the change in #540 I believe there are still bugs (related to idempotency) but I'm not sure if they were there before the change or not.
Hi guys, I'm trying to learn how to test, doing my best, but I'm still clunky at it. Is there any chance you can work on this so that the fix is merged? thank you
Hello @tremble , In order to test this, I have forked community.aws (and general), cloned them, and put them in a virtual environment; do I follow the core ansible instructions for testing a PR, but for collections instead? https://docs.ansible.com/ansible/2.3/dev_guide/developing_test_pr.html
I'm also not clear, do I need a source checkout of ansible itself? Please excuse my inexperience, I just really want to get this bug fixed and help out. thank you,
Or, @tremble , do I clone community.aws as in the instructions for community.general (https://docs.ansible.com/ansible/latest/dev_guide/developing_collections_contributing.html), switch to a new branch in the clone and checkout the PR, modify COLLECTIONS_PATHS in my project's ansible configuration to point to and prefer this version of the collection, and begin testing, continuing to use the version of ansible I have in my project's virtual environment, of course making sure to test on a VM or container?
I do think there is a different issue here. Looking at the documentation, it does not make sense to have "compression" e.g. under forwarded_values. Is it possible that the docs are erroneous, and the fix here would be to unindent anything that is not "forwarded_values?
