[Bug] listPaginated not returning all results

[harrisonlinowes]

Is this a new bug? In other words: Is this an error, flaw, failure or fault? Please search Github issues and check our Community Forum to see if someone has already reported the bug you encountered.

If this is a request for help or troubleshooting code in your own Pinecone project, please join the Pinecone Community Forum.

• [X] I believe this is a new bug
• [X] I have searched the existing Github issues and Community Forum, and I could not find an existing post for this bug

Describe the bug When calling the listPaginated() function with a prefix id, I am not able to receive all the expected results. I was able to validate this by calling the analogous function in python which returned different results.

Typescript code:

const recordsWithPrefix = [];
const response: ListResponse = await index.listPaginated({
  prefix: prefix,
});
recordsWithPrefix.push(...extractIdFromResponse(response));
let nextPage = response.pagination?.next;

while (nextPage !== undefined) {
  const nextResponse: ListResponse = await index.listPaginated({
    prefix: prefix,
    paginationToken: nextPage,
  });
  recordsWithPrefix.push(...extractIdFromResponse(response));
  nextPage = nextResponse.pagination?.next;
}

Python Code:

index = pc.Index("my-index")
allRecordIds = []
for ids in index.list(prefix="my_prefix"):
  allRecordIds += ids

Error information The results from the two snippets of code above should be equivalent however they are not. The python code returns all expected record ids while the typescript code only returns a subset of ids.

Steps to reproduce the issue locally See code above.

Environment

• OS and version: Mac
• Node version: v20.15.0
• Typescript SDK version: 3.0.1

Additional context Add any other context about the problem here.

[anawishnoff]

Thanks for opening - the team is investigating this issue and will follow up

[max-tano]

+1 our team would really like this fixed too

[harrisonlinowes]

Hi, is there any update on this?

[aulorbe]

Hi, is there any update on this?

Speak of the devil! It turns out that this issue was caused by a bug on our core DB. And coincidentally that team literally just deployed the fix for it.

Will test via the TS client now and get back to you if all is clear.

(The bug was that pagination tokens were being non-deterministically returned from the core DB, so the differences you saw btwn TS and Python were actually a product of a race condition, not differences in the clients themselves.)

[aulorbe]

Okay! I've confirmed their fix works on my end. Will you try now on your end and let me know if you're still seeing the error 🤞 ?