etcd icon indicating copy to clipboard operation
etcd copied to clipboard
Published 3 weeks ago • verified publisher icon etcd-io

add txn_put_get benchmark for #18667

Open hwdef opened this issue 1 month ago • 9 comments

Ref: #18667 #20545

This PR adds a benchmark for txn_put_get, specifically tailored for issue #18667.

Regarding the original issue, there are some existing solutions. However, we are concerned that they might impact performance. The challenge is that we currently lack a dedicated benchmark case to quantify the extent of this potential impact. Some discussions in https://github.com/etcd-io/etcd/pull/20545

The key to this benchmark case is to perform both the PUT and GET within a single transaction while utilizing a prefix query. This way, when there is https://github.com/ahrtr/etcd/commit/76ac23fda65ab27a86139acd5d5941e773531618, the GET/RANGE requests can not be skipped, which degraded performance, and vice versa.

I'll attach the benchmark results shortly.

cc @siyuanfoundation @serathius @ahrtr

hwdef avatar Nov 19 '25 17:11 hwdef

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hwdef Once this PR has been reviewed and has the lgtm label, please assign siyuanfoundation for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot avatar Nov 19 '25 17:11 k8s-ci-robot

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 69.23%. Comparing base (bc12c94) to head (aef7d12). :warning: Report is 50 commits behind head on main.

Additional details and impacted files

see 22 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #20953      +/-   ##
==========================================
+ Coverage   69.20%   69.23%   +0.03%     
==========================================
  Files         422      422              
  Lines       34841    34841              
==========================================
+ Hits        24110    24121      +11     
+ Misses       9330     9324       -6     
+ Partials     1401     1396       -5

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update bc12c94...aef7d12. Read the comment docs.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Nov 19 '25 17:11 codecov[bot]

/retest

hwdef avatar Nov 20 '25 01:11 hwdef

/test pull-etcd-unit-test-arm64

hwdef avatar Nov 20 '25 02:11 hwdef

original:

benchmark --endpoints http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 --conns 3 --clients 3 txn-put-get
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #3] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #3] Subchannel picks a new address "127.0.0.1:22379" to connect
2025/12/03 15:41:02 INFO: [core] original dial target is: "etcd-endpoints://0xc0001fc600/127.0.0.1:2379"
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #4] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #4] Subchannel picks a new address "127.0.0.1:32379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "127.0.0.1:2379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #5] Channel created for target "etcd-endpoints://0xc0001fc600/127.0.0.1:2379"
2025/12/03 15:41:02 INFO: [core] [Channel #5] parsed dial target is: resolver.Target{URL:url.URL{Scheme:"etcd-endpoints", Opaque:"", User:(*url.Userinfo)(nil), Host:"0xc0001fc600", Path:"/127.0.0.1:2379", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/12/03 15:41:02 INFO: [core] [Channel #5] Channel authority set to "127.0.0.1:2379"
2025/12/03 15:41:02 INFO: [core] [Channel #5] Resolver state updated: {
  "Addresses": null,
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:2379",
          "ServerName": "127.0.0.1:2379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:22379",
          "ServerName": "127.0.0.1:22379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:32379",
          "ServerName": "127.0.0.1:32379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": {
    "Config": {
      "Config": null,
      "Methods": {}
    },
    "Err": null
  },
  "Attributes": null
} (service config updated)
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #3] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #1] Channel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #5] Channel switches to new LB policy "round_robin"
2025/12/03 15:41:02 INFO: [roundrobin] [0xc000254990] Created
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #9] Subchannel created
2025/12/03 15:41:02 INFO: [core] [Channel #1 SubChannel #4] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #10] Subchannel created
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #11] Subchannel created
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #10] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #11] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #5] Channel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #9] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #10] Subchannel picks a new address "127.0.0.1:22379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #9] Subchannel picks a new address "127.0.0.1:2379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #11] Subchannel picks a new address "127.0.0.1:32379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #5] Channel exiting idle mode
2025/12/03 15:41:02 INFO: [core] original dial target is: "etcd-endpoints://0xc0000ee400/127.0.0.1:2379"
2025/12/03 15:41:02 INFO: [core] [Channel #14] Channel created for target "etcd-endpoints://0xc0000ee400/127.0.0.1:2379"
2025/12/03 15:41:02 INFO: [core] [Channel #14] parsed dial target is: resolver.Target{URL:url.URL{Scheme:"etcd-endpoints", Opaque:"", User:(*url.Userinfo)(nil), Host:"0xc0000ee400", Path:"/127.0.0.1:2379", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/12/03 15:41:02 INFO: [core] [Channel #14] Channel authority set to "127.0.0.1:2379"
2025/12/03 15:41:02 INFO: [core] [Channel #14] Resolver state updated: {
  "Addresses": null,
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:2379",
          "ServerName": "127.0.0.1:2379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:22379",
          "ServerName": "127.0.0.1:22379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:32379",
          "ServerName": "127.0.0.1:32379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": {
    "Config": {
      "Config": null,
      "Methods": {}
    },
    "Err": null
  },
  "Attributes": null
} (service config updated)
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #10] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #14] Channel switches to new LB policy "round_robin"
2025/12/03 15:41:02 INFO: [roundrobin] [0xc00058bb30] Created
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #9] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #5] Channel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #16] Subchannel created
2025/12/03 15:41:02 INFO: [core] [Channel #5 SubChannel #11] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #17] Subchannel created
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #18] Subchannel created
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #16] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #14] Channel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #18] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #14] Channel exiting idle mode
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #17] Subchannel Connectivity change to CONNECTING
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #16] Subchannel picks a new address "127.0.0.1:22379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #18] Subchannel picks a new address "127.0.0.1:2379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #17] Subchannel picks a new address "127.0.0.1:32379" to connect
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #16] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #17] Subchannel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #14] Channel Connectivity change to READY
2025/12/03 15:41:02 INFO: [core] [Channel #14 SubChannel #18] Subchannel Connectivity change to READY
10000 / 10000 [---------------------------------------------] 100.00% 220 p/s

Summary:
  Total:        45.6897 secs.
  Slowest:      0.2107 secs.
  Fastest:      0.0022 secs.
  Average:      0.0137 secs.
  Stddev:       0.0066 secs.
  Requests/sec: 218.8677

Response time histogram:
  0.0022 [1]    |
  0.0231 [9607] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0439 [382]  |∎
  0.0648 [1]    |
  0.0856 [0]    |
  0.1065 [0]    |
  0.1273 [3]    |
  0.1482 [0]    |
  0.1690 [0]    |
  0.1899 [2]    |
  0.2107 [4]    |

Latency distribution:
  10% in 0.0093 secs.
  25% in 0.0105 secs.
  50% in 0.0126 secs.
  75% in 0.0159 secs.
  90% in 0.0194 secs.
  95% in 0.0220 secs.
  99% in 0.0281 secs.
  99.9% in 0.0481 secs.

modified:

$ benchmark --endpoints http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 --conns 3 --clients 3 txn-put-get
2025/12/03 15:51:24 INFO: [core] original dial target is: "etcd-endpoints://0xc000436400/127.0.0.1:2379"
2025/12/03 15:51:24 INFO: [core] [Channel #5] Channel created for target "etcd-endpoints://0xc000436400/127.0.0.1:2379"
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #3] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "127.0.0.1:32379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #3] Subchannel picks a new address "127.0.0.1:2379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #5] parsed dial target is: resolver.Target{URL:url.URL{Scheme:"etcd-endpoints", Opaque:"", User:(*url.Userinfo)(nil), Host:"0xc000436400", Path:"/127.0.0.1:2379", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/12/03 15:51:24 INFO: [core] [Channel #5] Channel authority set to "127.0.0.1:2379"
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #4] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #1] Channel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #5] Resolver state updated: {
  "Addresses": null,
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:2379",
          "ServerName": "127.0.0.1:2379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:22379",
          "ServerName": "127.0.0.1:22379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:32379",
          "ServerName": "127.0.0.1:32379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": {
    "Config": {
      "Config": null,
      "Methods": {}
    },
    "Err": null
  },
  "Attributes": null
} (service config updated)
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #4] Subchannel picks a new address "127.0.0.1:22379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #5] Channel switches to new LB policy "round_robin"
2025/12/03 15:51:24 INFO: [roundrobin] [0xc0001fc150] Created
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #4] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #9] Subchannel created
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #10] Subchannel created
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #11] Subchannel created
2025/12/03 15:51:24 INFO: [core] [Channel #5] Channel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #5] Channel exiting idle mode
2025/12/03 15:51:24 INFO: [core] [Channel #1 SubChannel #3] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] original dial target is: "etcd-endpoints://0xc000436600/127.0.0.1:2379"
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #10] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #12] Channel created for target "etcd-endpoints://0xc000436600/127.0.0.1:2379"
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #11] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #11] Subchannel picks a new address "127.0.0.1:22379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #12] parsed dial target is: resolver.Target{URL:url.URL{Scheme:"etcd-endpoints", Opaque:"", User:(*url.Userinfo)(nil), Host:"0xc000436600", Path:"/127.0.0.1:2379", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/12/03 15:51:24 INFO: [core] [Channel #12] Channel authority set to "127.0.0.1:2379"
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #10] Subchannel picks a new address "127.0.0.1:2379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #9] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #9] Subchannel picks a new address "127.0.0.1:32379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #12] Resolver state updated: {
  "Addresses": null,
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:2379",
          "ServerName": "127.0.0.1:2379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:22379",
          "ServerName": "127.0.0.1:22379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:32379",
          "ServerName": "127.0.0.1:32379",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": {
    "Config": {
      "Config": null,
      "Methods": {}
    },
    "Err": null
  },
  "Attributes": null
} (service config updated)
2025/12/03 15:51:24 INFO: [core] [Channel #12] Channel switches to new LB policy "round_robin"
2025/12/03 15:51:24 INFO: [roundrobin] [0xc0001634a0] Created
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #13] Subchannel created
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #17] Subchannel created
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #18] Subchannel created
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #13] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #12] Channel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #12] Channel exiting idle mode
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #9] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #5] Channel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #17] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #10] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #18] Subchannel Connectivity change to CONNECTING
2025/12/03 15:51:24 INFO: [core] [Channel #5 SubChannel #11] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #18] Subchannel picks a new address "127.0.0.1:22379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #17] Subchannel picks a new address "127.0.0.1:2379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #13] Subchannel picks a new address "127.0.0.1:32379" to connect
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #17] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #12] Channel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #18] Subchannel Connectivity change to READY
2025/12/03 15:51:24 INFO: [core] [Channel #12 SubChannel #13] Subchannel Connectivity change to READY
10000 / 10000 [---------------------------------------------] 100.00% 220 p/s

Summary:
  Total:        45.7123 secs.
  Slowest:      0.2121 secs.
  Fastest:      0.0022 secs.
  Average:      0.0137 secs.
  Stddev:       0.0058 secs.
  Requests/sec: 218.7596

Response time histogram:
  0.0022 [1]    |
  0.0232 [9536] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0442 [457]  |∎
  0.0652 [3]    |
  0.0862 [0]    |
  0.1072 [0]    |
  0.1282 [0]    |
  0.1491 [0]    |
  0.1701 [0]    |
  0.1911 [0]    |
  0.2121 [3]    |

Latency distribution:
  10% in 0.0094 secs.
  25% in 0.0104 secs.
  50% in 0.0124 secs.
  75% in 0.0159 secs.
  90% in 0.0198 secs.
  95% in 0.0226 secs.
  99% in 0.0292 secs.
  99.9% in 0.0390 secs.

I think the design of this benchmark case is fine, but the test results still show that the original and modified versions are the same.
Perhaps this indicates that removeNeedlessRangeReqs() has little impact on performance? :)

hwdef avatar Dec 03 '25 08:12 hwdef

/cc @ahrtr

siyuanfoundation avatar Dec 04 '25 15:12 siyuanfoundation

/retest

hwdef avatar Dec 15 '25 16:12 hwdef

I significantly increased the key-space-size and key-size, but the test results still showed no change.

hwdef avatar Dec 15 '25 16:12 hwdef

@hwdef: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-etcd-coverage-report aef7d12cb4e8f0b035ccf3a1310d655a56ff8d72 link true /test pull-etcd-coverage-report

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot avatar Dec 15 '25 16:12 k8s-ci-robot