airbyte
airbyte copied to clipboard
airbytehq
source-jira: pagination failure, Users stream only returns 50 records
Environment
- Source Connector and version: source-jira 0.2.20
Current Behavior
The Users stream only returns 50 records. This is probably related to next_page_token implementation as the default number of returned values on this endpoint is 50.
Expected Behavior
The Users stream should return all available users
Logs
Hey yeah I was able to reproduce this with our integration account
Hey guys, I've provided a temporary solution to the problem by adding a maxResults field in the jira source interface.
https://github.com/antonioneto-hotmart/airbyte/tree/source-jira-custom
Commit https://github.com/airbytehq/airbyte/commit/2211070dbcb835efadd35abb844c4417c09794ae
Hey @antonioneto-hotmart, do you have time to help us with a PR as a fix?
Hey Airbyte Team, just wanted to comment that the 50 record limitation also exists on Sprints [where it appears that the sprints connector returns a maximum of 50 records per board it iterates through]. It appears that it would be possible to paginate through API calls by adding startAt=[last starting place+ 50] until the body returns results with an "isLast"=true, but I'm personally unsure how this would be implemented in airbyte. Would be a great feature addition, as it's pretty easy to hit the 50 sprint limit.
A slight adjustment to my previous comment; After reviewing the source code in streams.py for the jira connector, it seems that fixing pagination for Users and Streams would require different approaches.
While neither class blocks contain references to the next_page_token variable, The Users API doesn't include the paging metadata in the response, and instead might need to be handled differently, such as the temporary solution described above. However, the Sprints API does return pagination metadata, so adding logic that references next_page_token in that class block may solve that pagination issue. Let me know if you'd prefer to make that a separate issue.