heredocs don't work in pipeline parsers

[dhduvall]

Terraform Version

$ terraform -v
Terraform v1.1.7
on darwin_arm64
+ provider registry.terraform.io/datadog/datadog v3.9.0

Affected Resource(s)

• datadog_logs_custom_pipeline

Terraform Configuration Files

resource "datadog_logs_custom_pipeline" "golan" {
    is_enabled = true
    name       = "Golang key=value style logging"

    filter {
        query = "service:(agent OR prometheus) image_name:(\"grafana/agent\" OR \"prom/prometheus\")"
    }

    processor {
        grok_parser {
            is_enabled = true
            name       = "Golang key/value"
            samples    = [
                "ts=2022-01-25T19:44:10.626987538Z caller=tailer.go:99 level=info component=logs logs_config=default component=tailer msg=\"position timer: exited\" path=/var/log/pods/qa_devices-background-jobs-27385663-bs677_e10ba094-2ab4-41b2-
9ffd-b46618c06b91/devices-background-jobs/0.log",
                "ts=2022-01-27T00:12:14.759138219Z caller=filetarget.go:159 level=info component=logs logs_config=default msg=\"filetarget: watcher closed, tailer stopped, positions saved\" path=\"/var/log/pods/*d9a89e52-6065-4805-843d-418a0dd96eea/devices-background-jobs/*.log\"",
                "ts=2022-01-27T00:19:11.517593648Z caller=filetargetmanager.go:275 level=info component=logs logs_config=default msg=\"Removing target\" key=\"{app=\\\"background-jobs\\\", container=\\\"devices-background-jobs\\\", controller_uid=\\\"4ad670d7-e7c3-4ae6-92cc-5b83d1d6f958\\\", fluentd=\\\"true\\\", job=\\\"dev/background-jobs\\\", job_name=\\\"devices-background-jobs-27387378\\\", namespace=\\\"dev\\\", pod=\\\"devices-background-jobs-27387378-5b9x5\\\", service=\\\"devices\\\"}\"",
            ]
            source     = "message"

            grok {
                match_rules = <<-EOT
                    # How do we handle the embedded JSON in the `key` key?
                    key_value %%{data::keyvalue("=", "/-:*")}
                EOT
                # match_rules = "# How do we handle the embedded JSON in the `key` key?\nkey_value %%{data::keyvalue(\"=\", \"/-:*\")}"

                support_rules = ""
            }
        }
    }
}

Debug Output

https://gist.github.com/dhduvall/75818726db2d42717ac9fb2593bf0e98

Expected Behavior

The plan should be empty of changes.

Actual Behavior

$ terraform plan
Terraform will perform the following actions:

  # datadog_logs_custom_pipeline.golang will be updated in-place
  ~ resource "datadog_logs_custom_pipeline" "golang" {
        id         = "LgBvjJvDTN2qZ7II_FjjCA"
        name       = "Golang key=value style logging"
        # (1 unchanged attribute hidden)


      ~ processor {

          ~ grok_parser {
                name       = "Golang key/value"
                # (3 unchanged attributes hidden)

              ~ grok {
                  ~ match_rules = <<-EOT
                        # How do we handle the embedded JSON in the `key` key?
                        key_value %{data::keyvalue("=", "/-:*")}
                    EOT
                }
            }
        }
        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Steps to Reproduce

The state file has the following:

                "grok_parser": [
                  {
                    "grok": [
                      {
                        "match_rules": "# How do we handle the embedded JSON in the `key` key?\nkey_value %{data::keyvalue(\"=\", \"/-:*\")}",
                        "support_rules": ""
                      }
                    ],
                    "is_enabled": true,
                    "name": "Golang key/value",
                    "samples": [
                      "ts=2022-01-25T19:44:10.626987538Z caller=tailer.go:99 level=info component=logs logs_config=default component=tailer msg=\"position timer: exited\" path=/var/log/pods/qa_devices-background-jobs-27385663-bs677_e10ba094-2ab4-41b2-9ffd-b46618c06b91/devices-background-jobs/0.log",
                      "ts=2022-01-27T00:12:14.759138219Z caller=filetarget.go:159 level=info component=logs logs_config=default msg=\"filetarget: watcher closed, tailer stopped, positions saved\" path=\"/var/log/pods/*d9a89e52-6065-4805-843d-418a0dd96eea/devices-background-jobs/*.log\"",
                      "ts=2022-01-27T00:19:11.517593648Z caller=filetargetmanager.go:275 level=info component=logs logs_config=default msg=\"Removing target\" key=\"{app=\\\"background-jobs\\\", container=\\\"devices-background-jobs\\\", controller_uid=\\\"4ad670d7-e7c3-4ae6-92cc-5b83d1d6f958\\\", fluentd=\\\"true\\\", job=\\\"dev/background-jobs\\\", job_name=\\\"devices-background-jobs-27387378\\\", namespace=\\\"dev\\\", pod=\\\"devices-background-jobs-27387378-5b9x5\\\", service=\\\"devices\\\"}\""
                    ],
                    "source": "message"
                  }
                ],

Then terraform plan with the previous .tf file. If I uncomment the single-line match_rules entry, it works as expected.

References

• 954351335a09626e4401ab45e48236f71a898f4f

It looks like the grok parser schema code for match_rules needs to use strings.TrimSpace() in its StateFunc, the way it was added in the above commit. I don't know if that would have any ill effects on certain valid values, though.

[ianhundere]

i wanna chime in and say that heredocs work, but if you try and update them, nothing occurs. so you have to remove / readd them in order to apply.