hyperledger
[BUG] general degradation of the work of Iroha2
OS and Environment
linux, k8s
GIT commit hash
2ffcd007e3f794f8ee32488a0446bc11c1242cd0
Minimum working example / Steps to reproduce
The test was carried out with different load intensity from 1 to 15 virtual users, the test result was similar - a systematic decrease in tps
testing was carried out in several approaches I tried to apply loads from 1500 to 125 tps in order to find a TPS indicator with which we can work stably. But the bad news is that the overall degradation pattern of Iroha2 was observed in every test. The presence or absence of a profiler does not affect the degradation of the system
load model
- assemble Iroha2 with the parca profiler, at the request of @BAStos525 @RamioMustafin the service is available via this link
- start 4 virtual users, about 360tps and 3rps (if you turn off the profiler the indicator will be higher)
preconditions: genesis.json contains 5 domains 1200 users each user has 10,000 assets
test script jenkins job to start load testing
params:
testScenario=load
simulationClass=jp.co.soramitsu.load.simulation.transaction.StabilitySimulation
logLevel=INFO
configuration=standard
intensity=4
rampDuration=4
stageDuration=3600
jenkins job to clean up load generator
- sending an asset from one user to another without waiting for the transaction status
analytics tools profiler
profiler before and after degradation iroha2 current state iroha2 TPS/queue status iroha2 cluster monitoring OpenSearch
Actual result
The expected TPS level is at least 350 throughout the scenario There are no CPU or RAM leaks Actual state of the world for each peer
Expected result
TPS reaches the desired value and then begins to decrease to minimum values Based on the profiler data, I assume that Iroha spends a large amount of time updating the state of the world
Logs in JSON format
Log contents
Replace this text with a JSON log,
so it doesn't grow too large and has highlighting.
Who can help to reproduce?
No response
Notes
No response
the problem persists when assembling Linux-based Alpine, the problem is solved when assembling Image based on Debian