Can't find ceph config dump in output directories

[ommoreno]

In librbdfio.py, before every test, dump_config is called to save the ceph config into a file called ceph_settings.out. I don't think this ever completes successfully because that file is never created.

[bengland2]

No error checking is done on pdsh calls (yet). It's easy to add now but in the meantime, did you try running the commands in the CBT log standalone and see whether they worked?

[ommoreno]

Interestingly enough, it does work when I invoke the command manually on the head node. The file is created and I can see it in the tmp directory of the storage node.

09:42:09 - DEBUG - cbt - CheckedPopen continue_if_error=True args=pdsh -f 128 -R ssh -w root@cephdev05,root@cephdev04,root@cephdev03,root@cephdev02,root@cephdev01 sudo /usr/bin/ceph -c /home/bmpa/tmp_cbt/ceph/ceph.conf --admin-daemon /var/run/ceph/ceph-osd.0.asok config show > /home/bmpa/tmp_cbt/00000000/LibrbdFio/osd_ra-00000128/op_size-00004096/concurrent_procs-080/iodepth-032/randrw/readmix-0/ceph_settings.out

I just doesn't seem to work when it runs in CBT. One thing I noticed is that the config show command doesn't work on any nodes that are not the first one. This is because it's asking for ceph-osd.0.asok, which only exists on the first node.

cephdev02: admin_socket: exception getting command descriptions: [Errno 2] No such file or directory pdsh@ft01: cephdev02: ssh exited with exit code 22

Not like it really matters if we allow pdsh to continue_if_error and in most cases, the config should be the same across nodes.