skuber
skuber copied to clipboard
Proposal: Skuber should make it easier to stream large lists of resources
Skuber now supports using the limit
and continue
flags in ListOptions
, which makes it possible for clients to retrieve large lists of resources in chunks, thus improving efficiency and reducing memory consumption, and mitigating related issues.
However it currently requires the client application to itself implement the paging/continuation processing described at https://kubernetes.io/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks , even though that algorithm is general rather than client-specific.
This work could be abstracted by skuber into a new variant of the list
API method (e.g. listChunked
) that returns an Akka stream of chunk elements, where a chunk is basically a ListResource[O <: ObjectResource]
type. Each chunk would be capped in size by a limit
parameter`.
Example potential usage - something like:
val podListChunkedSource: Source[PodList] = k8s.listChunked[PodList](limit=10)