community.aws New Module: Amazon MQ

SUMMARY

replicates https://github.com/ansible-collections/amazon.aws/pull/266 as suggested

This PR contains some basic support for Amazon MQ. It covers

Managing Amazon MQ brokers
Managing Amazon MQ configurations
Managing Amazon MQ (local) users

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

Amazon MQ

ADDITIONAL INFORMATION

We use those modules to manage our MQ Brokers. The context of usage is

brokers are created by different means (e.g. terraform) - but the module is capable of doing that as well
we've developed a custom role around those modules
to manage MQ user credentials we use another local extension of amazon.aws collection (will come as separate PR) that interfaces with AWS SecretsManager
that role uses the reboot broker feature (part of this PR) to implement a custom handler that reboots a broker whenever a configuration run sees any changes in configuration and/or users

The functionality of the added modules is illustrated in the added test suite (tests/integration/targets/mq) some of them still require a running MQ broker to be usable.

Missing functionality/limitations:

requires a recent version of boto3 library (older versions don't support all Amazon MQ features used here)
API results are returned "as is", i.e. there's no conversion from camel case yaml to snake yaml

mq_broker.py

no support for LDAP connection (external user management)
update configuration only supported through mq_broker_config.py
known to work only with EngineType=ACTIVEMQ (proper support for RABBITMQ still missing)

mq_broker_config.py

needs proper XML comparison between current and desired configuration (current one is too simplistic)
only tested with ActiveMQ configurations

Feb 21 '21 10:02 fotto

cc @jillr @s-hertel @tremble @wimnat click here for bot help

Feb 21 '21 10:02 ansibullbot

I'll work on the code quality (sanity checks) I'll rebase the PR after that. You can ignore it until then

Feb 22 '21 10:02 fotto

Couple of quick comments:

It would be better if mq_broker could create/delete a broker. This would then support creating the broker as part of the integration tests so they could be run in our CI environment.
"info" states/actions are generally discouraged. It would be better to have a dedicated mq_broker_info module.

Feb 22 '21 10:02 tremble

Couple of quick comments:

* It would be better if mq_broker could create/delete a broker.  This would then support creating the broker as part of the integration tests so they could be run in our CI environment.

* "info" states/actions are generally discouraged.  It would be better to have a dedicated mq_broker_info module.

Thank you for your hints. I'll take them into account

Feb 22 '21 11:02 fotto

current state:

logic/structure refactored as suggested
documentation added/fixed
code and doc passes sanity checks

not covered, yet:

the modules have some functional limitations (see PR description)
the current test harness (tests/integration/targets/mq) needs to be reworked to run as automatic test (is there a good example to learn from?)

How to proceed from here?

@tremble

Feb 28 '21 10:02 fotto

@jillr wrote an initial guide: https://www.ansible.com/blog/getting-started-with-aws-ansible-module-development

tests/integration/targets/iam_saml_federation/ includes a relatively simple example (but doesn't support check_mode) https://github.com/ansible-collections/amazon.aws/blob/main/tests/integration/targets/ec2_vpc_subnet/tasks/main.yml has some slightly more complex examples

Biggest issue is likely to be IAM policies.
https://github.com/mattclay/aws-terminator/pull/125 is the pieces for iam_saml_federation which doesn't return creation time (DBTerminator) https://github.com/mattclay/aws-terminator/pull/119 is the pieces for ec2_launch_template which does return a creation time (Terminator)

Feb 28 '21 15:02 tremble

@fotto this PR contains the following merge commits:

https://github.com/ansible-collections/community.aws/commit/afbe9fe78301e333a6e5e766c5b2337bfef5c825

Please rebase your branch to remove these commits.

click here for bot help

Mar 28 '21 12:03 ansibullbot

current state:

some functional deficiencies resolved
test suit completely refactored to be compliant with recommendations from https://www.ansible.com/blog/getting-started-with-aws-ansible-module-development

state of integrations tests

against my AWS Account I was able to run the new test suit with something like

sudo ansible-test integration mq --python-interpreter /usr/bin/python3 --local

but I had to load an additional policy covering the MQ related permissions (and some related ones required to run MQ operations)

So far the test suite is marked as "unsupported" since I haven't prepared a (policy) extension for https://github.com/mattclay/aws-terminator, yet.

Please advice how to proceed from here

@tremble @jillr

May 26 '21 15:05 fotto

Build succeeded.

ansible-test-sanity-docker-devel : SUCCESS in 15m 56s (non-voting)
ansible-test-sanity-docker-milestone : SUCCESS in 36m 03s
ansible-test-sanity-docker-stable-2.9 : SUCCESS in 13m 12s
ansible-test-sanity-docker-stable-2.10 : SUCCESS in 21m 16s
ansible-test-sanity-docker-stable-2.11 : SUCCESS in 14m 26s
ansible-test-sanity-docker-stable-2.12 : SUCCESS in 11m 47s
ansible-test-units-community-aws-python38 : SUCCESS in 10m 12s
build-ansible-collection : SUCCESS in 8m 04s
ansible-test-splitter : SUCCESS in 1m 53s
ansible-test-cloud-integration-aws-py38_0 : SUCCESS in 8m 40s
ansible-test-cloud-integration-aws-py38_1 : SKIPPED
ansible-test-cloud-integration-aws-py38_2 : SKIPPED
ansible-test-cloud-integration-aws-py38_3 : SKIPPED
ansible-test-cloud-integration-aws-py38_4 : SKIPPED
ansible-test-cloud-integration-aws-py38_5 : SKIPPED
ansible-galaxy-importer : FAILURE in 5m 48s (non-voting)

Jan 10 '22 18:01 softwarefactory-project-zuul[bot]

Docs Build 📝

Thank you for contribution!✨

This PR has been merged and your docs changes will be incorporated when they are next published.

Jul 08 '22 06:07 github-actions[bot]

cc @markuman click here for bot help

Jul 08 '22 06:07 ansibullbot

Build succeeded.

Jul 08 '22 06:07 softwarefactory-project-zuul[bot]

Sorry it's taken so long to come back to this review. I've rebased against the current main branch, and done some docs linting cleanup

Only minor thing left that I've noticed is that the return values are still mostly in CamelCase. The Ansible 'standard' is to use snake_case, and we're currently working on some documentation around this https://github.com/ansible-collections/amazon.aws/pull/865

We have a helper ansible.module_utils.common.dict_transformations.camel_dict_to_snake_dict which should do the transformation (we exclude transforming Tags since tags are freeform with case sensitive keys):

    return_struct = { 
        'changed': changed,
        'broker': camel_dict_to_snake_dict(broker_info, ignore_list=['Tags']),
        'configuration': {
            'id': c_response['Id'],
            'revision': c_response['LatestRevision']['Revision']
        }
    }

Jul 08 '22 06:07 tremble

Build failed.

Aug 06 '22 07:08 softwarefactory-project-zuul[bot]

@fotto this PR contains the following merge commits:

https://github.com/ansible-collections/community.aws/commit/44847016b92c7f33ca49794c0d66950f3de57554

Please rebase your branch to remove these commits.

click here for bot help

Aug 06 '22 07:08 ansibullbot

Build failed.

Aug 06 '22 09:08 softwarefactory-project-zuul[bot]

Build failed.

Aug 06 '22 10:08 softwarefactory-project-zuul[bot]

Build failed.

Aug 06 '22 10:08 softwarefactory-project-zuul[bot]

Build failed.

Aug 06 '22 17:08 softwarefactory-project-zuul[bot]

@tremble Finally i found some time to work on your recent suggestions:

I've applied snake yaml conversion to response structures and adjusted integration tests accordingly.

Unfortunately I broke my system where I used to run those ansible integration tests against my own cloud environment before I could do the final branch cleanup.

Hence I launched some makeshift system and put latest state of all files into a single commit. I hope it's still fine (you should see a clean state in the RP) and PR can be merged soon.

Aug 07 '22 06:08 fotto

Build failed.

Aug 07 '22 07:08 softwarefactory-project-zuul[bot]

Build failed.

Aug 07 '22 07:08 softwarefactory-project-zuul[bot]

Build succeeded.

Aug 07 '22 08:08 softwarefactory-project-zuul[bot]

Build succeeded. https://ansible.softwarefactory-project.io/zuul/buildset/761ffa6de16a42febb21fba11d8d1ba9

:heavy_check_mark: ansible-galaxy-importer SUCCESS in 4m 07s :heavy_check_mark: build-ansible-collection SUCCESS in 12m 43s :heavy_check_mark: ansible-test-sanity-docker-devel SUCCESS in 11m 52s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-milestone SUCCESS in 8m 41s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 12s :heavy_check_mark: ansible-test-sanity-docker-stable-2.13 SUCCESS in 11m 48s :heavy_check_mark: ansible-test-sanity-docker-stable-2.14 SUCCESS in 8m 37s :heavy_check_mark: ansible-test-units-amazon-aws-python36 SUCCESS in 5m 54s :heavy_check_mark: ansible-test-units-amazon-aws-python38 SUCCESS in 7m 37s :heavy_check_mark: ansible-test-units-amazon-aws-python39 SUCCESS in 5m 32s :heavy_check_mark: ansible-test-units-amazon-aws-python310 SUCCESS in 6m 37s :heavy_check_mark: ansible-test-changelog SUCCESS in 4m 28s :heavy_check_mark: ansible-test-splitter SUCCESS in 4m 51s :heavy_check_mark: integration-community.aws-1 SUCCESS in 7m 26s Skipped 21 jobs

Feb 24 '23 13:02 softwarefactory-project-zuul[bot]

Build failed. https://ansible.softwarefactory-project.io/zuul/buildset/1904839c2b2840e2928b7fcb118743bd

:x: ansible-galaxy-importer FAILURE in 3m 55s :heavy_check_mark: build-ansible-collection SUCCESS in 12m 19s :heavy_check_mark: ansible-test-sanity-docker-devel SUCCESS in 8m 40s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-milestone SUCCESS in 12m 21s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-stable-2.12 SUCCESS in 12m 41s :heavy_check_mark: ansible-test-sanity-docker-stable-2.13 SUCCESS in 12m 40s :heavy_check_mark: ansible-test-sanity-docker-stable-2.14 SUCCESS in 12m 14s :heavy_check_mark: ansible-test-units-amazon-aws-python36 SUCCESS in 7m 34s :heavy_check_mark: ansible-test-units-amazon-aws-python38 SUCCESS in 7m 34s :heavy_check_mark: ansible-test-units-amazon-aws-python39 SUCCESS in 7m 42s :heavy_check_mark: ansible-test-units-amazon-aws-python310 SUCCESS in 9m 06s :heavy_check_mark: ansible-test-changelog SUCCESS in 4m 13s :heavy_check_mark: ansible-test-splitter SUCCESS in 4m 51s :heavy_check_mark: integration-community.aws-1 SUCCESS in 8m 21s Skipped 21 jobs

Mar 17 '23 19:03 softwarefactory-project-zuul[bot]

recheck

Mar 19 '23 13:03 tremble

Build succeeded. https://ansible.softwarefactory-project.io/zuul/buildset/e9c1dc58cd4e410593d5dcc670d72690

:heavy_check_mark: ansible-galaxy-importer SUCCESS in 3m 31s :heavy_check_mark: build-ansible-collection SUCCESS in 12m 49s :heavy_check_mark: ansible-test-sanity-docker-devel SUCCESS in 12m 21s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-milestone SUCCESS in 11m 51s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 38s :heavy_check_mark: ansible-test-sanity-docker-stable-2.13 SUCCESS in 11m 36s :heavy_check_mark: ansible-test-sanity-docker-stable-2.14 SUCCESS in 10m 00s :heavy_check_mark: ansible-test-units-amazon-aws-python36 SUCCESS in 6m 19s :heavy_check_mark: ansible-test-units-amazon-aws-python38 SUCCESS in 7m 39s :heavy_check_mark: ansible-test-units-amazon-aws-python39 SUCCESS in 7m 20s :heavy_check_mark: ansible-test-units-amazon-aws-python310 SUCCESS in 5m 46s :heavy_check_mark: ansible-test-changelog SUCCESS in 4m 15s :heavy_check_mark: ansible-test-splitter SUCCESS in 4m 41s :heavy_check_mark: integration-community.aws-1 SUCCESS in 5m 30s Skipped 21 jobs

Mar 19 '23 14:03 softwarefactory-project-zuul[bot]

Build failed (gate pipeline). For information on how to proceed, see http://docs.openstack.org/infra/manual/developers.html#automated-testing

https://ansible.softwarefactory-project.io/zuul/buildset/a4eb61cecfd447f98cb93f25824bdd3c

:x: ansible-galaxy-importer FAILURE in 4m 04s :heavy_check_mark: build-ansible-collection SUCCESS in 12m 37s :heavy_check_mark: ansible-test-sanity-docker-devel SUCCESS in 10m 44s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-milestone SUCCESS in 10m 14s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-stable-2.12 SUCCESS in 11m 20s :heavy_check_mark: ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 10s :heavy_check_mark: ansible-test-sanity-docker-stable-2.14 SUCCESS in 10m 15s :heavy_check_mark: ansible-test-units-amazon-aws-python36 SUCCESS in 6m 08s :heavy_check_mark: ansible-test-units-amazon-aws-python38 SUCCESS in 6m 50s :heavy_check_mark: ansible-test-units-amazon-aws-python39 SUCCESS in 5m 56s :heavy_check_mark: ansible-test-units-amazon-aws-python310 SUCCESS in 6m 10s :heavy_check_mark: ansible-test-changelog SUCCESS in 4m 27s :heavy_check_mark: ansible-test-splitter SUCCESS in 4m 48s :heavy_check_mark: integration-community.aws-1 SUCCESS in 5m 47s Skipped 21 jobs

Mar 20 '23 09:03 softwarefactory-project-zuul[bot]

regate

Mar 20 '23 10:03 tremble

Build failed (gate pipeline). For information on how to proceed, see http://docs.openstack.org/infra/manual/developers.html#automated-testing

https://ansible.softwarefactory-project.io/zuul/buildset/2ae5723893d24c2a9358cc517f948711

:x: ansible-galaxy-importer FAILURE in 3m 53s :heavy_check_mark: build-ansible-collection SUCCESS in 12m 28s :heavy_check_mark: ansible-test-sanity-docker-devel SUCCESS in 10m 07s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-milestone SUCCESS in 10m 17s (non-voting) :heavy_check_mark: ansible-test-sanity-docker-stable-2.12 SUCCESS in 11m 03s :heavy_check_mark: ansible-test-sanity-docker-stable-2.13 SUCCESS in 11m 09s :heavy_check_mark: ansible-test-sanity-docker-stable-2.14 SUCCESS in 9m 57s :heavy_check_mark: ansible-test-units-amazon-aws-python36 SUCCESS in 6m 15s :heavy_check_mark: ansible-test-units-amazon-aws-python38 SUCCESS in 5m 57s :heavy_check_mark: ansible-test-units-amazon-aws-python39 SUCCESS in 5m 56s :heavy_check_mark: ansible-test-units-amazon-aws-python310 SUCCESS in 6m 07s :heavy_check_mark: ansible-test-changelog SUCCESS in 4m 28s :heavy_check_mark: ansible-test-splitter SUCCESS in 4m 57s :heavy_check_mark: integration-community.aws-1 SUCCESS in 5m 43s Skipped 21 jobs

Mar 20 '23 10:03 softwarefactory-project-zuul[bot]

community.aws community.aws copied to clipboard

New Module: Amazon MQ

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Docs Build 📝

community.aws
community.aws copied to clipboard