Check for related resource which is not directly referenced

[starkpl]

Context

In AWS, if I create AWS Lambda function without creating CloudWatch log group first (with a name based on Lambda's name), the Lambda will create it automatically. I want to enforce some rules on CloudWatch log groups, but if I forget to create a log group, the automatically created group will not meet my requirements.

For example:

resource "aws_lambda_function" "test_lambda" {
  function_name = var.lambda_function_name
  # ... other configuration ...
  depends_on = [aws_cloudwatch_log_group.example]
}

# If I omit this, the log group will be created automatically, but it won't have retention_in_days set to 90
resource "aws_cloudwatch_log_group" "example" {
  name              = "/aws/lambda/${var.lambda_function_name}"
  retention_in_days = 90
}

Goal

I want to be able to create a scenario, which enforces that for every Lambda function created with Terraform there is a matching log group created.

Problem

With AWS provider, the log groups are not directly referenced by Lambda functions (or vice versa). It's just a convention that the Lambda named my-lambda will put logs to log group named /aws/lambda/my-lambda. Because of that I cannot find a way to create a scenario which will enforce that "for every Lambda function created with Terraform there is a matching log group created".

Based on other issues I tried experimenting with something like that:

  Scenario: Lambda functions should have corresponding CloudWatch log group
    Given I have aws_lambda_function defined
    Given I have any resource defined
    Then its type must be aws_cloudwatch_log_group
    When its type is aws_cloudwatch_log_group
    Then it must have name
    Then its value must match the "/aws/lambda/(.*)" regex

But it's not even close to the solution:

• it enforces the regex to every log group, but I could actually have some log groups which are not related to the Lambda in my Terraform code
• it checks for the pattern with a wildcard, but actually instead of a wildcard there should be the exact name of the Lambda
• it does something like a Cartesian product, but what I need is 1-1 relation

Question

Is there currently any way in terraform-compliance to implement such scenario? If not, is it planned?

Possible approaches

I can currently think of several approaches for this:

1. Make it possible to lookup resources based on other resources, something like:

Given I have aws_lambda_function defined
Then there should be aws_cloudwatch_log_group defined
And its name must be "/aws/lambda/{aws_lambda_function.function_name}"

2. Maybe we could make use of depends_on feature of Terraform. Because the log group needs to be created before Lambda, the Lambda has a dependency on the log group. Maybe it could be "mounted" just like it's currently done with directly referenced resources (for example aws_security_group and aws_security_group_rules)? Then it would be as easy as:

Given I have aws_lambda_function defined
Then it must contain aws_cloudwatch_log_group

3. Similar to approach 2, but maybe instead of depends_on, terraform-compliance could have "mounting" feature extended for specific cases, such as AWS Lambda + Log group, so that if it finds a log group named after a Lambda function, it mounts the log group to the Lambda.

Thanks :)

[Kudbettin]

Hi @starkpl

If I'm understanding you correctly, you're looking for a way to test:

having resource A implies having resource B

You can accomplish that with the following:

Feature: feature for issue #310

	@noskip_at_line_6
	Scenario: solution without specific names
		Given I have aws_lambda_function defined
		Given I have aws_cloudwatch_log_group defined     # line 6


	@noskip_at_lines_13_14
	Scenario: solution with specific names
		Given I have aws_lambda_function defined
		When its function_name is my_lambda_function_name
		Given I have aws_cloudwatch_log_group defined     # line 13
		When its name is /aws/lambda/some_name              # line 14

If you have lots of lambda_function-cloudwatch_log_group pairs, you could write a scenario outline to capture all pairs' scenarios at once:

	@noskip_at_lines_26_27_28 
	Scenario Outline: solution with scenario outline
		Given I have aws_lambda_function defined
		When its function_name is <function_name>
		Given I have aws_cloudwatch_log_group defined
		When its name is <name>

	Examples:
		| function_name | name |
		| lambda_function_name_1 | /aws/lambda/hi |      # line 26
                | lambda_function_name_2 | /aws/lambda/hello |   # line 27
                | lambda_function_name_3 | /aws/lambda/bye |     # line 28

Noskip documentation, which will be improved :)

[starkpl]

Hi @Kudbettin, thanks for your response.

Yes, this is the case, but actually I'm trying to build more general rules, without using specific Lambda names in the scenario. I'm automatically running terraform-compliance in my pipelines and the goal is that me or other developer should not forget to add log group when adding Lambda.

So there are two problems with using specific Lambda names:

• We are using one set of terraform-compliance features for all our terraform stacks. Having specific Lambda names in common features will make them grow very long over time, and it's an additional overhead to make changes to our terraform-compliance repository every time we add a Lambda.
• Anyway the goal is to make it impossible to forget to add the log group. But if someone forgets to add it, they could as well forget to add their Lambda name to the terraform-compliance feature.

So basically I'm looking for a way to implement this in a generic way (one scenario for all existing and future Lambdas).

[starkpl]

Just to summarize - currently I'm using @Kudbettin's solution:

Feature: feature for issue #310

	@noskip_at_line_6
	Scenario: solution without specific names
		Given I have aws_lambda_function defined
		Given I have aws_cloudwatch_log_group defined     # line 6

It covers most of simple cases and is a good starting point. It does give some false-positives, for example:

• If there are two Lambdas and only one has a Log Group - it will still pass.
• If there is a Lambda and a Log Group, but Log Group is not related to the Lambda - it will still pass.

I assume currently it's all that can be done. So the question is if there are any plans to implement some more advanced possibilities to check for related resources that are not directly referenced (for example using one of the approaches I suggested in the issue, or otherwise).

[Kudbettin]

Hi @starkpl,

This is a really good issue, and you have a solid solution idea.

Unless proven otherwise, Terraform-Compliance does lack the support for testing what you described.

At the moment using {aws_lambda_function.function_name} would not mix well with current steps, as there could be a multitude of aws_lambda_function.function_name instances (I assume a lot in your case). We're definitely planning to add functionality/steps that can handle this situation in the future.

If you'd like, I can ping the issue once something goes through.

[starkpl]

@Kudbettin it would be great, thank you.