oso
oso copied to clipboard
Add Graphql Pagination with Giveth Data Contribution
Summary
This PR introduces full support for pagination in GraphQL data sources using both offset-based and cursor-based methods. It extends the GraphQLResourceConfig and dynamically handles paginated GraphQL queries across any compatible API. The implementation is generic and reusable, with minimal changes required to onboard new data sources.
Features Added
1. Extended GraphQLResourceConfig
- Added a new
paginationfield to support dynamic pagination configuration. - Compatible with both
offsetandcursorstyles.
2. Offset-Based Pagination Support
- Implements
skip+takelogic. - Configurable parameter names (
offset_param,limit_param,start_offset). - Automatically paginates through all data using defined chunk sizes.
3. Cursor-Based Pagination Support
- Supports
afterorbeforecursors and limits viafirst/last. - Walks the GraphQL response using user-defined paths (
path_to_data,path_to_cursor,path_to_next). - Designed to work with GraphQL Relay-style APIs and dynamic datasets.
4. Retry and Debounce Handling
- Includes retry mechanism with exponential backoff across pagination calls.
- Prevents accidental infinite loops by checking
hasNextPageand cursor equality.
Changes Made
- Rewrote the
_execute_querylogic insidegraphql_factory. - Added a helper
walk_path()function for nested cursor paths. - Ensured backward compatibility for non-paginated resources.
Integration Test: Giveth Data Contribution
Documentation: https://hackmd.io/@torchablazed/S1Iloae-ge
@Rashmi-278 is attempting to deploy a commit to the Network-Goods Team on Vercel.
A member of the Team first needs to authorize it.
Closing in favor of #4527, thank u :)