Consistent Reads from Cache

[ghost]

Enhancement Description

• One-line enhancement description : Consistent Reads from Cache
• Kubernetes Enhancement Proposal:

Kubernetes Get and List requests are guaranteed to be "consistent reads" if the resourceVersion parameter is not provided. Consistent reads are served from etcd using a "quorum read".

But often the watch cache contains sufficiently up-to-date data to serve the read request, and could serve it far more efficiently.

This KEP proposes a mechanism to serve most reads from the watch cache while still providing the same consistency guarantees as serving the read from etcd.

• Discussion Link:
• KEP link: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2340-Consistent-reads-from-cache/README.md
• Primary contact (assignee): @serathius
• Responsible SIGs: sig-api-machinery
• Enhancement target (which target equals to which milestone):
  • [x] Alpha 1.28
    • [x] KEP (k/enhancements) update PR(s):
      • https://github.com/kubernetes/enhancements/pull/4047
    • [x] Code (k/k) update PR(s): https://github.com/kubernetes/kubernetes/pull/118508
    • [x] Docs (k/website) update PR(s): https://github.com/kubernetes/website/pull/42083
    • [x] Scalability tests: https://github.com/kubernetes/test-infra/pull/30094
  • [x] Beta 1.31
    • [x] KEP (k/enhancements) update PR(s):
      • https://github.com/kubernetes/enhancements/pull/4134
    • [x] Code (k/k) update PR(s):
      • https://github.com/kubernetes/kubernetes/pull/125115
      • https://github.com/kubernetes/kubernetes/pull/124612(previous https://github.com/kubernetes/kubernetes/pull/123264)
      • https://github.com/kubernetes/kubernetes/pull/124483
      • https://github.com/kubernetes/kubernetes/pull/123513
      • [x] https://github.com/kubernetes/perf-tests/issues/2540
      • [x] https://github.com/kubernetes/kubernetes/issues/123185
      • [x] https://github.com/kubernetes/kubernetes/issues/123187
      • [x] https://github.com/kubernetes/kubernetes/issues/123192
      • [x] https://github.com/kubernetes/kubernetes/pull/123513

All tasks done for beta

[ghost]

#2331 created for migrating the old keps into new template.

[lasomethingsomething]

/sig api-machinery

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[k8s-triage-robot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[serathius]

Reviving the KEP as agreed on SIG-Apimachinery meeting May 31st 2023. https://youtu.be/sueYXW89ZHg /assign

PR https://github.com/kubernetes/enhancements/pull/4047 to move KEP to implementable

[serathius]

/stage alpha

[serathius]

/milestone v1.28

[jpbetz]

/lead opt-in

[Atharva-Shinde]

Hello @shekhar-rajak @serathius 👋, Enhancements team here.

Just checking in as we approach Enhancements freeze on Thursday, 16th June 2023.

Looks like this enhancement is targeting for stage alpha for v1.28

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [X] KEP status is marked as implementable for latest-milestone:v1.28
• [ ] KEP readme has an updated detailed test plan section filled out
• [X] KEP readme has up to date graduation criteria
• [X] KEP has a production readiness review that has been completed and merged into k/enhancements.

There are some nits remaining to be addressed:

• Update kep.yaml with latest last update date.
• Add acknowledgement under the Test Plan section.

I won't decline this KEP for enhancements freeze because of these minor changes but would be great to get a PR ready addressing them.

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you :)

[serathius]

Don't understand the unmarked points.

KEP readme using the latest template has been merged into the k/enhancements repo. KEP readme has an updated detailed test plan section filled out

Tests follows the latest template (maybe roughly) and uses a detailed test plan. Please provide criteria that you used to decide that they are not.

[Atharva-Shinde]

Hey @serathius , the second unmarked point is a subset of the first unmarked point. As mentioned in my comment above, there is an acknowledgement at the start of the Test Plan section quoting:

[ ] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

which this KEP currently doesn't have and we need that to be included so that the KEP is in compliance with the latest KEP readme template.

[serathius]

Makes sense, suggestion for future. Separate questionnaire to KEP authors (also applies to PRR) from KEPs design document into separate file. Having agreements in the middle of the design makes it less readable, doesn't contribute to overall design and also cannot be automatically validated via ./hack/verify-kep.sh.

[serathius]

Sent https://github.com/kubernetes/enhancements/pull/4094

[Atharva-Shinde]

Thanks for the PR :) We appreciate your suggestion, I'll bring this up in our retrospection meeting.

[Atharva-Shinde]

Also @serathius, it would be great if you update this field in the kep.yaml alongside.

[serathius]

Done

[taniaduggal]

Hello @shekhar-rajak 1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Rishit-dagli]

Hey @serathius , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[serathius]

Not sure what needs to be documented as this is just a pure feature gate with no user facing effects aside of performance. cc @jpbetz @deads2k any suggestions?

[Atharva-Shinde]

Hey again @serathius :wave:

Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

Here’s the enhancement’s state for the upcoming code freeze:

• [X] All the PRs that are related to your enhancement are linked in the above issue description (for tracking purposes). This includes code, tests, and documentation related PR/s.
• [ ] All code related PR/s are merged or are in merge-ready state ( i.e they have approved and lgtm labels applied) by the code freeze deadline. This includes any tests related PR/s too.

For this enhancement, it looks like the following code related PR is open and it needs to be merged or should be in merge-ready state before the code freeze commences :

• https://github.com/kubernetes/test-infra/pull/30094

These are the code freeze related PR's that I found on this KEP issue

• https://github.com/kubernetes/kubernetes/pull/118508
• https://github.com/kubernetes/test-infra/pull/30094

Please keep the issue description up-to-date with all the PR/s that are associated with this KEP and let me know if there are other PR/s in k/k we should be tracking for this KEP.

As always, we are here to help if any questions come up. Thanks!

[serathius]

Results of benchmarking consistent reads from cache against reads from etcd. Added performance tests to https://testgrid.k8s.io/sig-scalability-experiments, following 2 scenarios:

• Small objects - 300`000 configmaps each 1KB of size.
• Large objects - 300 configmaps each 1MB of size.

In both cases we put load of 1 LIST per second with selector selecting no objects.

Comparing resource usage and latency with and without consistent list from watch cache enabled.

• 2-10 times reduction in CPU usage
• 20-50 times reduction of latency

	Handled List requests [qps]	kube-apiserver CPU [cores]			etcd CPU [cores]			LIST latency [ms]
		50%ile	90%ile	99%ile	50%ile	90%ile	99%ile	50%ile	90%ile	99%ile
Baseline	0	0.10	0.11	0.12	0.18	0.19	0.19	25.00	45.00	49.50
Enabled Large Objects	1	0.09	0.11	0.11	0.18	0.19	0.19	25.00	45.00	49.49
Disabled Large Objects	1	3.13	3.14	3.16	1.73	16.16	16.37	1438.49	1856.13	1985.61
Enabled Small Objects	1	0.63	0.64	0.68	0.23	2.11	2.16	499.32	582.04	648.00
Disabled Small Objects	0.86	6.92	70.85	71.41	3.57	3.72	3.75	10493.83	17910.71	21800.00

[taniaduggal]

Hey @serathius , Docs Shadow here! could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[Atharva-Shinde]

Hey @serathius 👋 Enhancements Lead here, With https://github.com/kubernetes/kubernetes/pull/118508 and https://github.com/kubernetes/test-infra/pull/30094 merged as per the issue description, this enhancement is now tracked for v1.28 Code Freeze. Thanks!

[serathius]

Done https://github.com/kubernetes/website/pull/42083

[Rishit-dagli]

Hello @shekhar-rajak @serathius wave: please take a look at Documenting for a release - PR Ready for Review to get your docs PR ready for review before Tuesday 25th July 2023. Thank you!

Ref: https://github.com/kubernetes/website/pull/42083

[npolshakova]

Hello 👋, 1.29 Enhancements Lead here.

If you wish to progress this enhancement in v1.29, please have the SIG lead opt-in your enhancement by adding the lead-opt-in and milestone v1.29 labels before the Production Readiness Review Freeze.

/remove-label lead-opted-in

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.28 now, If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[wojtek-t]

/milestone v1.30 /label lead-opted-in /label stage/beta /remove-label stage/alpha