skuber icon indicating copy to clipboard operation
skuber copied to clipboard

Proposal: Skuber should make it easier to stream large lists of resources

Open doriordan opened this issue 6 years ago • 0 comments

Skuber now supports using the limit and continue flags in ListOptions, which makes it possible for clients to retrieve large lists of resources in chunks, thus improving efficiency and reducing memory consumption, and mitigating related issues. However it currently requires the client application to itself implement the paging/continuation processing described at https://kubernetes.io/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks , even though that algorithm is general rather than client-specific. This work could be abstracted by skuber into a new variant of the list API method (e.g. listChunked) that returns an Akka stream of chunk elements, where a chunk is basically a ListResource[O <: ObjectResource] type. Each chunk would be capped in size by a limit parameter`. Example potential usage - something like:

 val podListChunkedSource: Source[PodList] = k8s.listChunked[PodList](limit=10)

doriordan avatar Feb 23 '19 10:02 doriordan