newrelic-kubernetes-operator icon indicating copy to clipboard operation
newrelic-kubernetes-operator copied to clipboard

Handle scenario where creating an alert policy w/ a condition causes error

Open sanderblue opened this issue 4 years ago • 0 comments

Description

If you have an alert policy configured with an "invalid" alert condition, the policy will be created, but an invisible, sneaky, silent error will occur under the hood for the invalid alert condition (such as missing a required condition config attribute).

Objective

At a minimum we probably want to surface an error/warning message. Welcome to suggestions for a better UX as well.

Steps To Reproduce

Steps to reproduce the behavior:

  1. run kubectl apply -f ./policy.yaml

    # policy.yaml
    
    apiVersion: nr.k8s.newrelic.com/v1
    kind: Policy
    metadata:
      name: my-policy
    spec:
      api_key: API_KEY
      name: "Alert Policy Created With k8s"
      region: "us"
    
    # Invalid NRQL Alert Condition spec (hypothetical scenario)
    # `nrql.since_value` is missing
    # `enabled` is missing 
    conditions:
      - spec:
          name: "NRQL Alert Condition Created With k8s"
          nrql:
            query: "SELECT average(duration) FROM Transaction WHERE appName = 'Dummy App'"
          terms:
            - threshold: "1"
              time_function: "all"
              duration: "4"
              priority: "critical"
              operator: "above"
    
  2. Check New Relic. You should see a new alert policy, but no alert conditions have been added due to the error occurring on the NRQL alert condition request.

Note: You can tail the logs to see the requests and errors using the following commands:

k describe nodes -n newrelic-kubernetes-operator-system | grep newrelic-kubernetes

# Then using the hash from the other command, tail the logs
kubectl logs -f -n newrelic-kubernetes-operator-system -c manager newrelic-kubernetes-operator-controller-manager-<HASH GOES HERE>

sanderblue avatar May 26 '20 23:05 sanderblue