pagination token should be constructed from search results "sort" field rather than from the last item's field

[philvarner]

Practically, it's probably the same thing, but the ES documentation says to use the "sort" field values.

[samsipe]

@philvarner having an issue with using sort and pagination together. Not sure if this issue is the same but I am noticing the next token is missing on when sortby is used. I thought pystac-client had the bug, but it's clear what's happening when using postman. Should the fallback here be to use page on the client side, risking duplicate object being ingested while paging?

Easiest way to reproduce this is using stac-browser

This works, and you get a next token.

--header 'Content-Type: application/json' \
--data '{
    "limit": 1
}'

Response:

{
    "type": "FeatureCollection",
    "stac_version": "1.0.0",
    "stac_extensions": [],
    "context": {
        "limit": 1,
        "matched": 61610747,
        "returned": 1
    },
    "numberMatched": 61610747,
    "numberReturned": 1,
    "features": [
        {
            ...
        }
    ],
    "links": [
        {
            "rel": "next",
            "title": "Next page of Items",
            "method": "POST",
            "type": "application/geo+json",
            "href": "https://earth-search.aws.element84.com/v1/search",
            "merge": false,
            "body": {
                "limit": 1,
                "next": "2023-08-25T23:36:41.463517Z,S1A_IW_GRDH_1SDV_20230825T233627_20230825T233654_050038_060542,sentinel-1-grd"
            }
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://earth-search.aws.element84.com/v1"
        }
    ]
}

And this doesn’t…

curl --location 'https://earth-search.aws.element84.com/v1/search' \
--header 'Content-Type: application/json' \
--data '{
    "sortby": "-id",
    "limit": 1
}'

Response:

{
    "type": "FeatureCollection",
    "stac_version": "1.0.0",
    "stac_extensions": [],
    "context": {
        "limit": 1,
        "matched": 61610832,
        "returned": 1
    },
    "numberMatched": 61610832,
    "numberReturned": 1,
    "features": [
        {
           ...
        }
    ],
    "links": [
        {
            "rel": "next",
            "title": "Next page of Items",
            "method": "POST",
            "type": "application/geo+json",
            "href": "https://earth-search.aws.element84.com/v1/search",
            "merge": false,
            "body": {
                "sortby": [
                    {
                        "field": "id",
                        "direction": "desc"
                    }
                ],
                "limit": 1
            }
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://earth-search.aws.element84.com/v1"
        }
    ]
}

See nothing in the links.body

[philvarner]

links is an array -- you want to get the Link within the links with rel=next, and it's body field from that you're looking for.

[samsipe]

@philvarner thanks for the reply. I am not having an issue with finding the body, and I only submitted curl requests because stac-browser and pystac-client were failing, and I couldn't find out why.

In rel=next if the body doesn't have a next token it appears that the client just keeps getting the same first page again and again. If not next token then there should at least be a page=2 instead right?

Try this, it just hangs:

stac-client search https://earth-search.aws.element84.com/v1/ -c sentinel-2-l2a --sortby properties.updated

[philvarner]

Ah, I understand now, it does seem that the next parameter is missing. Can you file a separate issue about this?

[samsipe]

Of course!

Issue: https://github.com/stac-utils/stac-server/issues/589