subgrounds icon indicating copy to clipboard operation
subgrounds copied to clipboard

Backfilling, checkpointing, improved pagination

Open 0xMochan opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. I want better tools when working with large subgrounds queries.

I often deal with queries that can take overnight, so more tools to help manage the lifecycle of those queries can help me ensure the success of those queries.

Describe the solution you'd like

  • Improved pagination strategies that helps me recover when a graph node spits in error out late in the pagination cycle
  • Being able to checkpoint or backfill queries so that when errors do occur, I'm able to recover w/o any data-loss

Describe alternatives you've considered Using _iter methods to help checkpoint data

  • Difficult and not-well documented (also should be fixed)
  • Generally would be easier to have subgrounds handle this

Additional context Opinionated so should result in multiple solutions

Implementation checklist

  • [ ] Task 1

0xMochan avatar May 26 '23 19:05 0xMochan