iroha icon indicating copy to clipboard operation
iroha copied to clipboard

[BUG] CPU leak during load testing

Open timofeevmd opened this issue 1 year ago • 1 comments

OS and Environment

k8s, linux

GIT commit hash

2ffcd007e3f794f8ee32488a0446bc11c1242cd0

Minimum working example / Steps to reproduce

run the job in Jenkins https://jenkins.soramitsu.co.jp/job/iroha2/job/deploy/job/iroha2-perf/ with the following parameters

testScenario maxPerf
logLevel INFO
configuration standard
concurrentUsers 2
times 7200
deachLevelLasting 10
separatedByRampsLasting 10
startingFrom 0

leave other parameters unchanged

iroha2 config recommended

"COMMIT_TIME_LIMIT_MS": 10000,
    "MAX_TRANSACTIONS_IN_BLOCK": 20,
    "ACTOR_CHANNEL_CAPACITY": 100,
    "GOSSIP_BATCH_SIZE": 500
    "GOSSIP_PERIOD_MS": 1000
  },
  "BLOCK_SYNC": {
    "GOSSIP_PERIOD_MS": 1000,
    "BLOCK_BATCH_SIZE": 4,
    "ACTOR_CHANNEL_CAPACITY": 100
  },

analytical info

  1. profiler
  2. iroha2 current state
  3. iroha2 TPS / queue status
  4. iroha2 cluster monitoring

Actual result

  • when running the script
  • the load should increase evenly
  • transactions should be accepted by the computer
  • no CPU leaks should occur

Expected result

When the load is applied, a leak in the CPU is observed, judging by the logs from the profiler, the problem is related to the regular calculation of the hash of blocks and transactions. Analytical information can be found at the link to the profiler

  1. profiler
  2. iroha2 current state
  3. iroha2 TPS / queue status
  4. iroha2 cluster monitoring
  5. OpenSearch

logs downloaded from the parca profile.pb.gz

Logs in JSON format

Log contents
look at the profiler logs

Who can help to reproduce?

@RamilMus

Notes

No response

timofeevmd avatar Mar 22 '24 08:03 timofeevmd

verify with rc21

mversic avatar Apr 24 '24 09:04 mversic

the problem persists when assembling Linux-based Alpine, the problem is solved when assembling Image based on Debian

timofeevmd avatar Jul 10 '24 11:07 timofeevmd