Proper cleanup when an application fails to call unifyfs_unmount

[sandrain]

System information

Summit

Describe the problem you're observing

When an application fails to call unifyfs_unmount, the UnifyFS server daemon encounters a segmentation fault during its cleanup process.

Maybe, we would need to capture when the application process is completed/killed and then accordingly cleanup data structures.

Describe how to reproduce the problem

Launch multiple applications with unifyfsd, and omit unifyfs_unmount from one or more applications. For instance:

1. launch the unifyfs server daemon (unifyfs start ...)
2. run write-static
3. run sysio-stat-static (this doesn't explicitly call unifyfs_unmount unless -u option is specified)
4. run read-statici
5. cleanup (unifyfs stop ...)

Include any warning or errors or releveant debugging data

The below backtrace is from the margotree branch, but it should be similar in the dev branch.

(gdb) bt
#0  0x000020000085fbf0 in raise () from /lib64/libc.so.6
#1  0x0000200000861f6c in abort () from /lib64/libc.so.6
#2  0x00002000008a8d10 in __libc_message () from /lib64/libc.so.6
#3  0x00002000008b3820 in malloc_consolidate () from /lib64/libc.so.6
#4  0x00002000008ba030 in free () from /lib64/libc.so.6
#5  0x0000000010007884 in cleanup_app_client (app=0x200008000a30, client=0x2000100436f0)
    at /autofs/nccs-svm1_techint/home/hs2/projects/UnifyFS/server/src/unifyfs_server.c:989
#6  0x0000000010007a9c in cleanup_app_client (client=<optimized out>, app=0x200008000a30)
    at /autofs/nccs-svm1_techint/home/hs2/projects/UnifyFS/server/src/unifyfs_server.c:715
#7  cleanup_application (app=0x200008000a30)
    at /autofs/nccs-svm1_techint/home/hs2/projects/UnifyFS/server/src/unifyfs_server.c:716
#8  0x0000000010004824 in unifyfs_exit ()
    at /autofs/nccs-svm1_techint/home/hs2/projects/UnifyFS/server/src/unifyfs_server.c:597
#9  main (argc=<optimized out>, argv=0x7fffcbf6df48)
    at /autofs/nccs-svm1_techint/home/hs2/projects/UnifyFS/server/src/unifyfs_server.c:421
(gdb)