JMESPath with Pagination returns a List of None for SQS and CloudFront

[AMMullan]

Describe the bug

I've been using jmespath heavily with Boto3 (after finally learning about it.... clearly didn't RTFM) and it's been great but I notice that, when there are no SQS Queues or CloudFront distributions in the account/region it returns a list of None - presumably there may be more but I've tested against quite alot of services and that's all I've run into thus far.

Any other service returns an empty list, which is expected.

Expected Behavior

Should return an empty list (i.e. [])

Current Behavior

Returns a list of None (i.e. [None])

Reproduction Steps

import boto3

session = boto3.session.Session()
region = 'eu-central-1'

sqs = session.client('sqs', region_name=region)
queues = sqs.get_paginator('list_queues').paginate(
    PaginationConfig={'PageSize': 100}
).search('QueueUrls[]')
print(list(queues))  # Outputs [None]

cloudfront = session.client('cloudfront', region_name=region)
distros = cloudfront.get_paginator('list_distributions').paginate(
    PaginationConfig={'PageSize': 100}
).search('DistributionList.Items[]')
print(list(distros))  # Outputs [None]

ec2 = session.client('ec2', region_name=region)
volumes = ec2.get_paginator('describe_volumes').paginate(
    PaginationConfig={'PageSize': 100}
).search('Volumes[]')
print(list(volumes))  # Outputs []

SDK version used

boto3==1.28.10, botocore==1.31.10

Environment details (OS name and version, etc.)

Ubuntu 22.04, Python 3.10.6

[tim-finnigan]

Hi @AMMullan thanks for reporting this. For reference I'll share this documentation on filtering results with JMESPath: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html#filtering-results-with-jmespath

You can find model definitions for paginators and service APIs in Botocore, for example SQS:

• https://github.com/boto/botocore/blob/develop/botocore/data/sqs/2012-11-05/paginators-1.json
• https://github.com/boto/botocore/blob/develop/botocore/data/sqs/2012-11-05/service-2.json

Because service teams own the implementation of their APIs, there are sometimes inconsistencies that may result in unexpected behavior. The ListQueues API for example does not return a QueueUrls list when there are no queues to return. This came up before here in another issue: https://github.com/boto/boto3/issues/2811. (Although the Java SDK team decided to return an empty list in this case, SDK teams generally try to avoid diverging from the default API responses.) SDK teams also try to avoid implementing inconsistent behavior between the SDKs or implementing backwards incompatible changes. For those reasons I'm not sure anything can be done here, but this behavior is worth calling out and discussing.

[AMMullan]

Hey @tim-finnigan - yeah I've read that page, though not sure how this might relate? The examples I gave used the JMESPath.

So I didn't realise that the QueueUrls list just wasn't returned rather than an empty list being returned - that makes sense as to why None is being returned.

I'll just have to figure out if there's a way using JMESPath to set an empty list if QueueUrls[] isn't present - rather not have to have different solutions for different services because I'm sure there are others that act like this.

Thanks for getting back to me mate.

[AMMullan]

OK, so I found a solution using jmespath library:

queues = sqs.get_paginator('list_queues').paginate(
    PaginationConfig={
        'PageSize': 1000
    }
).build_full_result()
queue_urls = jmespath.compile('QueueUrls[]').search(queues) or []
print(queue_urls)

This seems to work. This will obviously use more memory though presumably because instead of using an iterator like the boto3 paginator/jmespath combo does, it's just pulling all records into the queues variable - so not necessarily an ideal thing to do as standard....?

[tim-finnigan]

Thanks for following up and sharing the workaround. I brought this issue up for discussion with the team and they don't plan on creating a customization for the SQS API to return an empty list. But we can track examples like SQS ListQueues and CloudFront ListDistributions, and also investigate if there is a better way to handle those API responses when filtering with JMESPath.