airflow-client-python icon indicating copy to clipboard operation
airflow-client-python copied to clipboard

Handle pagination for client code

Open fredthomsen opened this issue 3 years ago • 2 comments

My thoughts on API client libraries are that pagination should be handled by them or at least have the option to be handled by them and the client code does not have to concern itself with calling functions again to deal with pagination etc. Now realizing this is auto-generated code is this something of interest? If so, I can put together something more concrete.

fredthomsen avatar Jan 11 '22 03:01 fredthomsen

I think maybe that's somethign that can be added in the tool we use for generation of the clients ? I woudl have found it a bit problematic if we had to do some more manual modifications of the generated code, but maybe contributing such feature there might solve the ptoblem (for us and others). Or maybe it's already there and we do not use it?

potiuk avatar Jan 11 '22 11:01 potiuk

After some digging around looks like there is a lot of debate around this feature, and understandably so given the different ways pagination can be handled across different APIs.

Some of the main issues I looked into for reference:

  • https://github.com/OAI/OpenAPI-Specification/issues/1835
  • https://github.com/OAI/OpenAPI-Specification/issues/1706

Looks like Transposit has created an OpenAPI extension to handle some of the most common types of pagination to address this issue: https://www.transposit.com/blog/2019.12.10-using-extensions-to-support-pagination-in-openapi/, but this implementation isn't open sourced. How it's done there is how I envisioned something like this working.

fredthomsen avatar Jan 13 '22 20:01 fredthomsen