jepsen
jepsen copied to clipboard
jepsen-io
[docker] rabbitmq test fails with "rabbitmq-server: unrecognized service"
from jepsen-control docker container:
$ cd rabbitmq && lein test :only jepsen.rabbitmq-test
lein test jepsen.rabbitmq-test
INFO jepsen.os.debian - :n5 setting up debian
INFO jepsen.os.debian - :n2 setting up debian
INFO jepsen.os.debian - :n1 setting up debian
INFO jepsen.os.debian - :n4 setting up debian
INFO jepsen.os.debian - :n3 setting up debian
INFO jepsen.os.debian - Installing #{man}
INFO jepsen.os.debian - Installing #{man}
INFO jepsen.os.debian - Installing #{man}
INFO jepsen.os.debian - Installing #{man}
INFO jepsen.os.debian - Installing #{man}
INFO jepsen.rabbitmq - :n2 Nuking rabbit
INFO jepsen.rabbitmq - :n4 Nuking rabbit
INFO jepsen.rabbitmq - :n5 Nuking rabbit
INFO jepsen.rabbitmq - :n3 Nuking rabbit
INFO jepsen.rabbitmq - :n1 Nuking rabbit
INFO jepsen.rabbitmq - Fetching deb package
INFO jepsen.rabbitmq - Fetching deb package
INFO jepsen.rabbitmq - Fetching deb package
INFO jepsen.rabbitmq - Installing rabbitmq
INFO jepsen.rabbitmq - Installing rabbitmq
INFO jepsen.rabbitmq - Installing rabbitmq
INFO jepsen.rabbitmq - :n2 Nuking rabbit
INFO jepsen.rabbitmq - :n1 Nuking rabbit
INFO jepsen.rabbitmq - :n3 Nuking rabbit
INFO jepsen.rabbitmq - :n4 Nuking rabbit
INFO jepsen.rabbitmq - :n5 Nuking rabbit
INFO jepsen.rabbitmq - :n3 Rabbit dead
INFO jepsen.rabbitmq - :n1 Rabbit dead
lein test :only jepsen.rabbitmq-test/rabbit-test
ERROR in (rabbit-test) (FutureTask.java:122)
Uncaught exception, not in assertion.
expected: nil
actual: java.util.concurrent.ExecutionException: java.lang.RuntimeException: rabbitmq-server: unrecognized service
...
Caused by: java.lang.RuntimeException: rabbitmq-server: unrecognized service
at jepsen.control$throw_on_nonzero_exit.invoke (control.clj:105)
jepsen.control$exec_STAR_.doInvoke (control.clj:121)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invoke (core.clj:624)
jepsen.control$exec.doInvoke (control.clj:135)
clojure.lang.RestFn.invoke (RestFn.java:436)
jepsen.rabbitmq$reify__8631.teardown_BANG_ (rabbitmq.clj:97)
jepsen.db$eval5744$fn__5745$G__5734__5749.invoke (db.clj:4)
jepsen.db$eval5744$fn__5745$G__5733__5754.invoke (db.clj:4)
clojure.lang.AFn.applyToHelper (AFn.java:160)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invoke (core.clj:626)
clojure.core$partial$fn__4228.doInvoke (core.clj:2468)
clojure.lang.RestFn.invoke (RestFn.java:421)
jepsen.core$on_nodes$fn__7547.invoke (core.clj:85)
clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
clojure.lang.AFn.call (AFn.java:18)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
java.lang.Thread.run (Thread.java:745)
hm, rabbitmq is installed only on n1,n2 and n3, but isn't it supposed to be install on all 5 nodes?
adding :nodes [:n1 :n2 :n3 :n4 :n5] to rabbitmq-test makes it try to start rabbitmq on 4 nodes?! btw how can I enable DEBUG logging level?
I have the same issue, but when run again. It stuck at another step root@control:/jepsen/rabbitmq# lein test jepsen.rabbitmq-test
lein test jepsen.rabbitmq-test
INFO jepsen.os.debian - :n5 setting up debian INFO jepsen.os.debian - :n4 setting up debian INFO jepsen.os.debian - :n3 setting up debian INFO jepsen.os.debian - :n2 setting up debian INFO jepsen.os.debian - :n1 setting up debian INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.rabbitmq - :n4 Nuking rabbit INFO jepsen.rabbitmq - :n1 Nuking rabbit INFO jepsen.rabbitmq - :n3 Nuking rabbit INFO jepsen.rabbitmq - :n2 Nuking rabbit INFO jepsen.rabbitmq - :n5 Nuking rabbit INFO jepsen.rabbitmq - Fetching deb package INFO jepsen.rabbitmq - Installing rabbitmq INFO jepsen.rabbitmq - :n4 Rabbit dead INFO jepsen.rabbitmq - :n2 Rabbit dead INFO jepsen.rabbitmq - :n1 Rabbit dead INFO jepsen.rabbitmq - Setting cookie INFO jepsen.rabbitmq - Setting cookie INFO jepsen.rabbitmq - uploading config INFO jepsen.rabbitmq - uploading config INFO jepsen.rabbitmq - Starting rabbitmq INFO jepsen.rabbitmq - Starting rabbitmq
Do you know why it has different issues while starting the test ?
I haven't worked on the rabbit tests in 3 years, but my guess is that whatever version of Debian (and possibly Jepsen) you're using no longer runs the rabbitmq service in the same way that it did circa 2015.
@aphyr, Can you suggest any tool to test rabbitmq that is easier to read than Jepsen(clojure)? I have spent a week to read on that but still can't understand how it works. If you have a document that guide step by step, please help to share. Thanks much for your answer
Sorry @vutruongqa, I don't really know any other tool that does what Jepsen does. At least, not in this degree of depth. There are a few Jepsen clones out there but they're usually oriented towards testing one particular system, or just inducing faults without correctness testing.
Oh, and as for support, well, there's a lot of documentation in Jepsen. The README links to the API docs, architecture overview, there are extensive code comments and docstrings, a comprehensive tutorial (as @vjuranek noted), and there are several free resources for learning Clojure; Clojure for the Brave and True is quite nice.
As far as bitrot goes, I'd love to spend some time bringing the RabbitMQ tests back up to speed with modern Jepsen, but my backlog right now is measured in years, not days. :-(
If you'd like commercial support, there are also engineers who work on Jepsen on a contract basis, and they might have some availability to help in your testing. I understand if you're budget constrained, though.