Codify the the kubernetes-etcd contract into etcd robustness tests

[serathius]

What would you like to be added?

Etcd robustness tests are pretty great way to validate generic contracts, however it's pretty huge problem space and it's hard to know what should be done next.

To drive robustness testing direction and priorities I would like to propose to first to focus on Kubernetes-Etcd contract. Work would include clarifying, documenting and then codifying it into etcd robustness tests.

The end result would be an etcd model that work as a fake that Kubernetes project can use to cross test the contract. A fake is a lightweight implementation that behaves like a real system and can be used for testing.

Plan

• [x] Collect the collective knowledge of the implicit Kubernetes-etcd contract in a document and collect feedback requires joining [email protected] or [email protected] to comment.
• [x] https://github.com/etcd-io/website/pull/676
• [x] Validate K8s-etcd contract in etcd robustness tests
  • [x] https://github.com/etcd-io/etcd/pull/15817
  • [x] https://github.com/etcd-io/etcd/pull/15827
  • [x] https://github.com/etcd-io/etcd/pull/15840
  • [x] https://github.com/etcd-io/etcd/pull/16072
  • [x] Implement list-watch protocol
  • [x] Implement pagination https://github.com/etcd-io/etcd/pull/16111
  • [x] Update watch validation to match contract watch validation https://github.com/etcd-io/etcd/pull/16136
  • [x] PrevKey https://github.com/etcd-io/etcd/issues/16838
  • [x] Use WithRequireLeader
  • [x] https://github.com/etcd-io/etcd/pull/17833
  • [ ] Implement non-recursive watch

Why is this needed?

Kubernetes is currently the main user of etcd, still there are a number of untested assumptions in the K8s-etcd contract. Changes to these underlying features may cause Kubernetes to behave incorrectly, or even not work at all.

[chaochn47]

RemainingItemCount: Allow the API servers to show a count of remaining items in the response to a chunking list request

I think that etcd historically broke the contract between etcd and kubernetes regarding the RemainingItemCount feature. This is a good example to add to the contract doc, wdyt? @serathius

Right. 3.5 does not have this improvement, but is correct. So we are good (but slow) . I don't think we should modify the wire-format in scope of v3 (i.e. semantic of requests, that is not driven by an explicit opt-in). In particular we need to make sure api-server e.g. 1.19 can still run on etcd-3.6+.

Thus api-server e.g. 1.24 should be able to discover whether it runs against etcd-3.6 that supports 'enhanced limiting' (e.g. v3client.IsCapabilitySupported(FAST_RANGE_WITHOUT_COUNT) and if so call range in that mode.

https://github.com/etcd-io/etcd/issues/13068

[serathius]

Thanks @chaochn47 for pointing this out, will look into that. Feel free to leave the comment on the document itself (requires being member of [email protected] or [email protected])

[chaochn47]

Thanks @serathius, I added comments to the doc. I hope it will get some attention from k8s folks as well ~

[serathius]

Most of the contract was added to robustness tests, they only remaining part is compaction.

[serathius]

When working on ConsistentListsFromCache in k8s I noticed some non-obvious details that we skipped in the definition of original contract document. Things like using clientv3.WithRequireLeader https://github.com/kubernetes/kubernetes/blob/fe9414d86ed44a667d018ee86c021041d82ac9f9/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go#L875-L884 or checking event.IsCreate() to ensure nil prevKV is ok https://github.com/kubernetes/kubernetes/blob/fe9414d86ed44a667d018ee86c021041d82ac9f9/staging/src/k8s.io/apiserver/pkg/storage/etcd3/event.go#L58-L75.

I think we should revisit K8s code to confirm that Kubernetes Traffic robustness tests setup the client and requests in the same way and we validate all the fields used by K8s. Current implementation of list&watch is also very naive as it doesn't handle all request types supported by K8s. For example setup a watch on single object, we should consider implementing this.

The end goal would be to unify those two codebases via a single interface as was discussed on SIG apimachinery.

cc @MadhavJivrajani