Allow > 20 tags for tag selection values

[wkneewalden]

The resource tagging API endpoint that we use to resolve tags to resource IDs only allows specifying 20 tag values at a time. If more than 20 tag values are specified in the configuration, split the list and make multiple requests.

Note in the readme that this causes additional requests to the resource tagging API so that users are not surprised when they get quota issues.

---

Original report:

Using tag_selections with aws_tag_select, the list of filtered tag values for a given tag has a maximum length (20). The limit appears to be based on the resourcegrouptagging api. Is it possible to allow regex filtering for a tag value, or at least greatly extend that limit?

[or-shachar]

@wkneewalden Can you please share the error message you get (possibly with the stacktrace) when you exceed max length?

Also - if possible - example for a configuration with > 20 length...

I don't use that property but I'd be happy to help 🙏

[amyDing629]

Hi, thanks for help. So the error message I got is "software.amazon.awssdk.services.resourcegroupstaggingapi.model.ResourceGroupsTaggingApiException: 1 validation error detected: value [value1, value2, ...., value100] at 'tagFilters.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 20 (Service: ResourceGroupsTaggingApi, Status Code: 400, Request ID: ..., Extended Request ID: ...).

[amyDing629]

A sample config file is like this: region: xxx metrics:

• aws_namespace: AWS/Lambda aws_metric_name: Throttles aws_dimension: [FunctionName] aws_statistics: [Sum] aws_tag_select:    tag_selections:      TagKey: [value1, value2, ..., value100]    resource_type_selections: lambda:function    resource_id_dimensions: FunctionName

[matthiasr]

Hmm, since this is an upstream API limitation, it seems that filtering client-side would be useful but potentially costly as you would have to request all metrics.

I am confused because the API docs say the limit is 256 items?

[matthiasr]

I am also not sure how the exporter could do this. The tag selection works by

1. Getting resource IDs from the resource tagging API
2. Filtering a metric dimension to only include these IDs

How would the exporter get a list of resource IDs that match the tag value regex, without implementing "get all resources" separately for every possible resource in AWS?

[wkneewalden]

Appears related to get_resources, each key can have a list of values up to 20: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/resourcegroupstaggingapi.html

Great point about regex. Since that would be costly to get all resources, is it possible to include some logic in CWE to detect when more than 20 entries are provided for a TagKey in tag_selections, and simply run the API multiple times in a loop grabbing 20 each time until the entire list is completed?

[matthiasr]

The exporter isn't using boto3, it is using the Java SDK, although AFAIK both are basically generated clients so they inherit the upstream limitations. This pointed me in the right direction though, the GetResources API action specifies the same limit.

Yes, we can make multiple requests, although we should note in the README that this causes additional requests to this API since there is a quota on those.

[wkneewalden]

Great to hear, that would work nicely for our application.

[matthiasr]

I updated the issue title and description