Support paged LIST queries from the Kubernetes API

[smarterclayton]

Feature Description

• One-line feature description (can be used as a release note): Allow API consumers - especially those that must retrieve large sets of data - to retrieve results in pages so as to reduce the memory and size impact of those very large queries.
• Primary contact (assignee): @smarterclayton
• Responsible SIGs: sig-api-machinery, sig-scalability
• Design proposal link (community repo): https://github.com/kubernetes/community/pull/896
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: deads2k lavalamp
• Approver (likely from SIG/area to which feature belongs): lavalamp
• Feature target (which target equals to which milestone):
  • Alpha release target (1.8)
  • Beta release target (1.9)
  • Stable release target ()

[dixudx]

/cc

[smarterclayton]

This work is part of the SLO improvements we have for kubernetes - helping us meet the requirements of much denser clusters. The key goal for alpha is to prove that it will help us reach the next scale limit for Kubernetes, and will take work across several releases to get there.

[idvoretskyi]

@smarterclayton @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-feature-requests can you confirm that this feature targets 1.8?

If yes, please, update the features tracking spreadsheet with the feature data, otherwise, let's remove this item from 1.8 milestone.

Thanks

[smarterclayton]

Yes, it was delivered for 1.8.

[smarterclayton]

Beta for 1.9

[smarterclayton]

Goals for 1.9 - expose in CLI for get.go Goals for 1.10 - expose for all CLI commands, GA

[zacharysarah]

@smarterclayton :wave: Please open a documentation PR and add a link to the 1.9 tracking spreadsheet. Thanks in advance!

[zacharysarah]

@smarterclayton Bump for docs ☝️

/cc @idvoretskyi

[idvoretskyi]

@smarterclayton @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-feature-requests any updates on the docs status?

A friendly reminder on docs deadline tomorrow.

/cc @zacharysarah

[smarterclayton]

Bah, I knew I forgot something yesterday. Ended up having to write more API docs since there wasn't a graceful place to hang the topic.

https://github.com/kubernetes/website/pull/6540

On Thu, Nov 30, 2017 at 4:09 PM, Ihor Dvoretskyi [email protected] wrote:

@smarterclayton https://github.com/smarterclayton @kubernetes/sig-api-machinery-feature-requests https://github.com/orgs/kubernetes/teams/sig-api-machinery-feature-requests @kubernetes/sig-scalability-feature-requests https://github.com/orgs/kubernetes/teams/sig-scalability-feature-requests any updates on the docs status?

A friendly reminder on docs deadline tomorrow.

/cc @zacharysarah https://github.com/zacharysarah

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/365#issuecomment-348321380, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p8hXe9e85TlEoQ876BfHH-TVeDQIks5s7xl2gaJpZM4OktK6 .

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[smarterclayton]

This was beta in 1.9, we have not yet moved it to GA. Docs are complete, CLI commands support it.

[idvoretskyi]

@smarterclayton when you are estimating the GA stage?

[smarterclayton]

1.11 unless we have a reason not to.

On Fri, Mar 9, 2018 at 1:31 PM, Ihor Dvoretskyi [email protected] wrote:

@smarterclayton https://github.com/smarterclayton when you are estimating the GA stage?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/365#issuecomment-371904715, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p_MmLq3d8PZ-uwmSm_J3tqS6zrntks5tcsqbgaJpZM4OktK6 .

[justaugustus]

@smarterclayton Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

cc @idvoretskyi

[smarterclayton]

@deads2k and I chatted briefly. I think we need to discuss in api-machinery whether we want to go one step further and expose this for end users (like user interfaces) that want to do human focused paging before we move to stable. So at minimum this is 1.12.

[justaugustus]

Thanks for the update, Clayton!

[justaugustus]

@smarterclayton --

It looks like this feature is currently in the Kubernetes 1.12 Milestone.

If that is still accurate, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y)
  • Beta release target (x.y)
  • Stable release target (x.y)

Set the following:

• Description
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

[jwierzbo-intel]

@smarterclayton Any plans to integrate this feature for extension-api-service (https://github.com/kubernetes/apiserver)?

As I see it also doesn't work for CRD objects..

EDIT: I found how to enable paging in extension api-server: flag Paging in k8s.io/apiserver/pkg/storage/storagebackend/config.go has to be set to true

But still don't know how to enable it for CRD models..

[justaugustus]

@smarterclayton -- Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12? If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

[justaugustus]

Slack update:

deads2k [10:28 AM]
@justaugustus We wanted to solicit more comments first.  We mentioned it in apimachinery here: https://docs.google.com/document/d/1x9RNaaysyO0gXHIr1y50QFbiL1x8OWnk2v3XnrdkT5Y/edit#heading=h.lgl8gmvpt98r and we should nail down conformance before doing so, though the api is backwards compatible

justaugustus [10:33 AM]
@deads2k so hold until 1.13 at least?

deads2k [10:47 AM]
I expect so

Removed from the 1.12 milestone.

[alecmerdler]

Are there any plans to add pagination support to custom resources?

[liggitt]

Are there any plans to add pagination support to custom resources?

Yes. It was an oversight they weren't enabled already. Opened https://github.com/kubernetes/kubernetes/pull/67861 to enable them

[kacole2]

Hi This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[cben]

/remove-lifecycle rotten

[kacole2]

Hello @smarterclayton @cben , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also require a KEP to be included.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[MikeSpreitzer]

There is a problem with https://github.com/kubernetes/kubernetes/pull/67861 , it was incomplete. It did not affect the sample-apiserver, and thus not anything derived from that.

[MikeSpreitzer]

I have been looking, and did not find a deep place where one common mod would cover all servers. In the main apiserver, for example, the pagination goes into the storage factory that appears in the pkg.master.ExtraConfig; other servers have their own independent ExtraConfig structs, and the one for sample-apiserver is empty. Inquiring minds want to know, why doesn't that have a StorageFactory while the main apiserver's ExtraConfig does?