@vercel/blob: pagination with limit: N, N <1000

[gheorghemolnar]

I'm currently evaluating the @vercel/storage blob in my project and I came across this pottential issue while using list with pagination.

Context:

• I've set up the token
• a total of 6 files uploaded on my project storage
• using the list with limit of 2

Problem Description:

• i'm issuing the first request with no cursor

  • I get the first 2 results, with: 'cursor' populated with a string and 'hasMore': true

• on the 2nd request, I put the cursor and its previously received value, in query

  • I get the same first results il 'files', with 'cursor' populated with a different string and 'hasMore': true

The behaviour of the 2nd request is happening for the followings request: - cursor populated every time with a differrent string value - hasMore is polpulated with true, event if it should return hasMore: false (ex. on the 4th time call of the 'limit' function).

I can get all the results (6 of them), only if I don't specify the limit in the optional parameters of the limit function. Can you give me some insights on this ? Thank you !

[vvo]

Hey there @gheorghemolnar I just tested your setup and could not generate the same issues.

Here's my script:

import { list } from "@vercel/blob";

async function listFilesTwoByTwo() {
  let hasMore = true;
  let cursor;
  const limit = 2;

  while (hasMore) {
    const {
      blobs,
      cursor: newCursor,
      hasMore: newHasMore,
    } = await list({ limit, cursor });

    console.log(blobs.length, typeof newCursor, newHasMore);

    cursor = newCursor;
    hasMore = newHasMore;
  }
}

listFilesTwoByTwo();

And the result:

> BLOB_READ_WRITE_TOKEN="xxx" node test.js
2 string true # Good
2 string true
2 string true
2 string true
2 string true
2 string true
2 string true
2 string true
1 undefined false # Good

Can you confirm my script output is the one you would expect? And check why you're getting a different result, thanks!

[gheorghemolnar]

Hi @vvo , thank you for your quick response. Indeed, I've tested the example, and it's working properly.

But in my case, i'm not using it in one loop, that gets all the results.

import { list } from "@vercel/blob";

let cursorPrevious = ""

async function listFilesTwoByTwo(cursor = "") {
    const limit = 2;
    const {
      blobs,
      cursor: newCursor,
      hasMore: newHasMore,
    } = await list({ limit, cursor });

    return {
        cursor:newCursor,
        hasMore:newHasMore
    }
}


await listFilesTwoByTwo(cursorPrevious);            // here I get {cursor: 'new value 01', hasMore: true}
await listFilesTwoByTwo(cursorPrevious);            // here I get {cursor: 'new value 02', hasMore: true}
... 
4th times
....
await listFilesTwoByTwo(cursorPrevious);             // here I get {cursor: 'new value 0X', hasMore: true}     
                                                                    // I would expect hasMore: false, and maybe cursor: ""

Thank you again

[vvo]

@gheorghemolnar I believe you should be using:

// [...] previous code
await listFilesTwoByTwo(cursorPrevious);
await listFilesTwoByTwo(cursorPrevious);    
await listFilesTwoByTwo(cursorPrevious);    

otherwise since you're not awaiting the listFilesTwoByTwo then you could run into race conditions where cursorPrevious is not yet with a new value, hope this is the issue you're facing, let me know!

[gheorghemolnar]

Thank you for you reply. I've just forgot to add the await in my code example above, which is showing my problem; sorry for that.

In my real code, I'm using it. I've tested it with Insomnia;

• I retrieved manually the cursor upon first request,
• then supplied it in the query for the next request, and so on ...

Thank you

[vvo]

Hey @gheorghemolnar would you be able to create a small code sample that demonstrates the issue you're facing? So I can try to run it, thanks a lot!

[gheorghemolnar]

import { list } from "@vercel/blob";

// Context: A total of 4 records


const limit                = 2;
let globalCursor   = "";                             // the Cursor is stored globally

async function populateBlobs () {
 
 /// add 4 blobs   
  
}

async function listFilesTwoByTwo(prevCursor = "") {
    const {
      blobs,
      cursor: newCursor,
      hasMore: newHasMore,
    } = await list({  limit,  cursor: prevCursor  });

    globalCursor =  newCursor;

    return {
        cursor:      globalCursor,
        hasMore:  newHasMore
    }
}

// add 4 records
populateBlobs();


await listFilesTwoByTwo();                 // 1st call:  I get {cursor: '1st cursor value', hasMore: true}  2 / 4
await listFilesTwoByTwo(globalCursor);     // 2nd call:  I get {cursor: '2nd cursor value', hasMore: true} 4/4
await listFilesTwoByTwo(globalCursor);     // 3nd call:  I get {cursor: '3rd cursor value', hasMore: true} 6/4 ????
.....

/*

Obs: 
     The usage of globalCursor as global variable, is just to illustrate my case.

And here it keeps on going no matter how many times I call listFilesTwoByTwo(previousCursor), the hasMore is always true, as if it never detects that the end of iteration / end of collection.

Does the limit supplied by user is taken into account ?
If it is not specified, I get ALL results (4 of them) and hasMore: FALSE.


*/

Can you confirm me that this is the intended pattern of usage of the cursor ? Thank you !