etcd icon indicating copy to clipboard operation
etcd copied to clipboard

Optimize Range WithKeysOnly to avoid loading values in memory

Open serathius opened this issue 5 months ago • 5 comments

What would you like to be added?

Kubernetes 1.34 will start using Range WithKeysOnly. Main reason to use this method is to reduce the overhead of loading values in memory when they are not needed. While using WithKeysOnly avoids transferring them to client, it doesn't help with etcd memory

Noticed that it loads values in memory just to throw them out little later: https://github.com/etcd-io/etcd/blob/810d69619e33aa09bfdc0449dc65ceb34e589ab6/server/etcdserver/txn/range.go#L150-L152

TODO:

  • Add thorough tests for range so we don't break it.
  • Identify places that use value in etcd server. For example when sorting by value
  • Implement support for KeysOnly method for keyindex
  • Use KeysOnly method when we don't need value, like when we don't sort by it.
  • Benchmark to show improvement

Please note this task is complicated and not good for first contributor.

Why is this needed?

Align with user expectation that using WithKeysOnly is more efficient on server.

serathius avatar Jul 22 '25 21:07 serathius

/cc @fuweid @joshjms @nwnt

serathius avatar Jul 22 '25 21:07 serathius

I'll try to take a look at it for more context

joshjms avatar Jul 23 '25 11:07 joshjms

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Sep 29 '25 00:09 github-actions[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Dec 07 '25 00:12 github-actions[bot]

/assign

nwnt avatar Dec 16 '25 04:12 nwnt