nextToken coming back null when limit exceeded

[massivespace]

I had a pipeline that was behaving normal and returning a list of data points similar to this: { nextToken: String, data: [DataPoint] } type DataPoint { value: Float, timeStamp: String }

The request was a simple Query request to a DynamoDB table (index).

I needed to expand this to return another value, so I updated DataPoint to return instead: { value: Float, rawData: Float, timeStamp: String }

As such, I'm returning a lot more data. In my frontend app, I was only receiving data for the first 10 or so days of my query after this change. I verified that nextToken is always coming back null now. I opened up the Query tab in AppSync console in AWS, and ran the query manually, and the same thing happened. I didn't get back all of my data, but nextToken is null, and there is no other indication that I hit any limits. By updating my resolver request template to add a limit of 1000, nextToken began to report a value back, and things are correctly paginated into my frontend. Setting the limit to 2000, I received more data but with large holes in the results, as if some of them returned nextToken and some didn't. I suspect maybe there's something strange going on here and just wanted to report it.

The work around is to use a low value for the limit parameter in the Query request, but it seems somewhat arbitrary picking one that works and doesn't return holes in the data. How do I know if I'm near some edge of a limit, which isn't being reported to me. I would like to keep the limit parameter as high as possible so the frontend loads quickly. Any suggestions?