incubator-pegasus icon indicating copy to clipboard operation
incubator-pegasus copied to clipboard

Using Chaos Mesh to enhance pegasus's stability

Open zhouqiang-cl opened this issue 3 years ago • 2 comments

Hello, I am Chaos Mesh maintainer. Chaos Mesh is a open source chaos engineering platform for Kubernetes.

As you may have known, chaos engineering is a promising method that can guarantee system resilience. We have been practicing chaos engineering for long on databases and have found many issues in various fault scernarios (for example data consistency issue caused by excessive recovery time). Some of the issue records at https://github.com/orgs/chaos-mesh/projects/1. There are also many adopters who are using Chaos Mesh to test their systems, such as PingCAP, Daily motion, Xpeng Motors, Meituan-Dianping, Apache pulsar, etc.

That said, I would like to invite you to use Chaos Mesh to enhance pegasus's stability. You can directly use Chaos Mesh or integrate it with your test platform. Chaos Mesh supports rich injection methods, such as pod/container kill, network partition, network interruption, CPU/memory usage, time rollback, disk failure, etc. Welcome to use it to enhance the stability of pegasus. If you have any questions, submit an issue at https://github.com/chaos-mesh/chaos-mesh, or send me an email [email protected]

Hope this helps and look forward to further communications. ❤️

zhouqiang-cl avatar Aug 12 '20 07:08 zhouqiang-cl

@zhouqiang-cl You guys are pretty nice :) We actually have an objective to integrate Chaos Mesh in our system. I think this is a very practical and innovative tool to improve our testing process.

Nevertheless, there're some problems the Pegasus needs to solve out first. The prerequisite for Chaos Mesh is a Pegasus deployment automated inside k8s. But this is not trivial since Pegasus is a stateful service. We do not have "tidb-operator"-like stuff.

neverchanje avatar Aug 13 '20 05:08 neverchanje

@zhouqiang-cl You guys are pretty nice :) We actually have an objective to integrate Chaos Mesh in our system. I think this is a very practical and innovative tool to improve our testing process.

Nevertheless, there're some problems the Pegasus needs to solve out first. The prerequisite for Chaos Mesh is a Pegasus deployment automated inside k8s. But this is not trivial since Pegasus is a stateful service. We do not have "tidb-operator"-like stuff.

Sorry for the late reply. Very glad that you have the interest, If you have any problem in using Chaos Mesh, we are very pleasure if we can help

Waiting for your great news ❤️

zhouqiang-cl avatar Aug 14 '20 02:08 zhouqiang-cl