tools/c7n_mailer - support for gcp

[thisisshi]

Accidentally closed the previous PR and am unable to open, previous PR here: https://github.com/cloud-custodian/cloud-custodian/pull/7073, many thanks to @mikejgray

Copied from the original PR:

Piggybacking off the work from Brent Clements in PR #4018, this adds basic GCP support for c7n-mailer.

The key caveat is that there is not currently an option to deploy via Cloud Function - it must be run using c7n-mailer run. I wanted to get this out there for feedback before adding the function deployment component.

• Confirmed successful SMTP test with mailtrap.io
• Allow Azure KV and GCP Secret Manager for ldap_bind_password
• Allow mailer to continue if LDAP creds are wrong
• Update docs
• Add Slack delivery support, change order of execution
• Address imports for secured strings
• Update tests

Please let me know if I need to be considering anything else here, like additional doc locations, missing setup.py additions, etc.

[mikejgray]

Hi @thisisshi, I'm going to try to deploy this locally and make sure it still works as we expected on my end. The refactor is definitely a good thing. Thanks for carrying this forward.

[mikejgray]

Hey @thisisshi , I haven't gotten too far in yet, but I had made a change to ldap_bind_password's schema to allow for GCP Secret Manager that appears to have been reverted. This is a secure option for storing that since the default AWS KMS encryption isn't available and I tried to design a clear separation of concerns between clouds running c7n_mailer. See error here:

  File "/Users/mikegray/.pyenv/versions/3.9.1/lib/python3.9/site-packages/jsonschema/validators.py", line 1058, in validate
    raise error
jsonschema.exceptions.ValidationError: {'type': 'gcp.secretmanager', 'secret': 'projects/123456789/secrets/ldap_bind_password'} is not of type 'string'

Failed validating 'type' in schema['properties']['ldap_bind_password']:
    {'type': 'string'}

On instance['ldap_bind_password']:
    {'secret': 'projects/123456789/secrets/ldap_bind_password',
     'type': 'gcp.secretmanager'}

[mikejgray]

Turns out my local environment was having issues, a clean install fixed the problem. I'm able to confirm successful operation in GCP using both email and Datadog, which are my primary use cases, and if it's passing tests for Slack we should be good there too.

The only oddity I see is that it's sending one email for every filtered resource but each email has the resources batched together. I've noticed that in our AWS operations as well so I think it's more likely our configuration than anything added via this update.

Thank you @thisisshi !

[kapilt]

its a big pr, but mostly just wanted to note we should validate behavior wrt to resource batching, @mikejgray is noting a separate email per resource, where as it should be a single email for a given resource batch (execution context ~ (region, account) + delivery target address).

[mikejgray]

its a big pr, but mostly just wanted to note we should validate behavior wrt to resource batching, @mikejgray is noting a separate email per resource, where as it should be a single email for a given resource batch (execution context ~ (region, account) + delivery target address).

To be more clear, I'm seeing a separate email per resource, but it's batching all the resource batches in each email. Functionally, you get a duplicate email that contains all the batched resources per resource in the batch. Ex: 6 VMs match the filter, you get 6 emails that list all 6 VMs. I'm not positive that's due to any code changes though, could be a misconfiguration on my end.

[thisisshi]

I'm unable to reproduce that @mikejgray can you provide a example policy?

policies:
  - name: gcp-notify
    resource: gcp.pubsub-topic
    description: example policy
    actions:
      - type: notify
        template: default
        priority_header: '2'
        subject: Hello from C7N Mailer
        to:
          - resource-owner
        transport:
          type: pubsub
          topic: projects/sonny-sandbox/topics/test

queue_url: projects/sonny-sandbox/subscriptions/test-sub
from_address: [email protected]
# c7n-mailer currently requires a role be present, even if it's empty
role: ""
smtp_server: smtp.mailtrap.io
smtp_port: 25
smtp_ssl: true
smtp_username: $redacted
smtp_password:
  type: gcp.secretmanager
  secret: projects/$redacted/secrets/smtppassword
slack_token: "foo"
contact_tags:
  - owner
email_base_url: stacklet.io

[kapilt]

@thisisshi can you resolve the conflicts, given your testing on behavior, I'm good to go with the pr as is.

[thisisshi]

@kapilt resolved

[kapilt]

looks like packaging conflicts got reintroduced by the 0.9.18 release prep pr merge.