vector

vector copied to clipboard

Reported in discord: https://discord.com/channels/742820443487993987/746070591097798688/780845681883217970

The user has a metric that appears to have had a tag that has an empty value which results in it being rejected by rusoto when transformed into a CloudWatch metric dimension:

Nov 24 17:19:21.361 ERROR sink{component_kind="sink" component_name=cloudwatch component_type=aws_cloudwatch_metrics}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Th
e parameter MetricData.member.15.Dimensions.member.1.Value is required.

I think we should consider filtering tags that have no value (perhaps with a warning log message).

Do you think this should be AWS CloudWatch specific? I can't think of a reason we would allow this at all.

Hmm, I'm not sure. I think this is an irregularity with CloudWatch as a key with an empty value still conveys information. Presumably there are other metrics storage engines would be fine with it, but we could survey.

I believe I've seen something similar and in my case a whole batch of logs seems to have been dropped from CloudWatch. The message was:

Feb 01 23:17:24 <hostname> vector[803]: Feb 01 23:17:24.073  INFO sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=5}: vector::sinks::aws_cloudwatch_logs: Sending events. events=35
Feb 01 23:17:24 <hostname> vector[803]: Feb 01 23:17:24.081 ERROR sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=5}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Put: 1 validation error detected: Value '' at 'logEvents.32.member.message' failed to satisfy constraint: Member must have length greater than or equal to 1
Feb 01 23:17:24 <hostname> vector[803]: Feb 01 23:17:24.081 ERROR sink{component_kind="sink" component_name=aws_cloudwatch_logs component_type=aws_cloudwatch_logs}:request{request_id=5}: vector::sinks::util::sink: Request failed. error=CloudwatchError::Put: 1 validation error detected: Value '' at 'logEvents.32.member.message' failed to satisfy constraint: Member must have length greater than or equal to 1

So from the message it seems the log was completely empty. Looking at the instance, there was indeed an empty log line (and I am using the text "codec" for the cloudwatch sink, which is probably what caused my issue). If at all possible, the whole batch should not be dropped if 1 of them fails validation.

It seems like at least the CloudWatch sink should filter out these tags. In addition I get similar errors with NaN values, which CloudWatch doesn't accept.

This is what I'm using for the moment to solve this problem:

  to_aws_metrics:
    type: remap
    inputs: [ 'metric_*' ]
    source: |
      .tags = filter(object!(.tags)) -> |k, v| {
          v != ""
      }