Design: Use busybox as the UnifyFS shell

[tonyhutter]

In light of the https://github.com/LLNL/UnifyFS/pull/425, I wanted to offer an alternative idea for a UnifyFS shell. I'd like to propose building busybox (https://en.wikipedia.org/wiki/BusyBox) against UnifyFS and using that as our shell. Benefits:

• It gives us all the unix commands, with all their command line options, and a shell (ash), in a single binary, allowing us to run commands directly in /unifyfs. This gives us a lot of power. For example:

   $ echo "hello world" > /unifyfs/helloworld.txt
   $ ls -lStr /unifyfs
   /unifyfs $ tar -cf mycheckpoints.tar.gz checkpoints/*
   $ wget -O /unifyfs/file https://path/to/file
   $ grep foo /unifyfs/checkpoints* 

• Effectively removes the need for a FUSE driver
• Gives us the option to write shell scripts for our automated tests.
• Gives our users a familiar shell environment to manipulate their files.
• Saves us from having to write, test, document, maintain, and support a custom shell for the lifetime of the UnifyFS project.
• Gives us the ability to build other commands against Unifyfs and use them with the busybox commands. For example, we could build bash against Unify, and have it use the busybox commands.

Thoughts?

[MichaelBrim]

Assuming this would be easy to do, I would support this. Having written a custom shell program myself in the past, the arguments regarding maintenance and non-compatibility with existing programs are all too familiar.

[sandrain]

I also think it's a good idea, as long as it's not becoming an overkill.

[tonyhutter]

I've been playing around with busybox and I'm now able to get it to build against UnifyFS using gotcha. It's super broken right now, but the fact that it even builds and runs is encouraging. Here's the extent of what I can do:

# Launch busybox's shell (ash)
$ ./busybox ash

$ touch /unifyfs/newfile
2020-02-03T15:16:31 tid=35493 @ __wrap_write() [unifyfs-sysio.c:1030] write was called for fd 82
trying to create /unifyfs/newfile
2020-02-03T15:16:31 tid=35493 @ unifyfs_fid_open() [unifyfs.c:1261] unifyfs_get_fid_from_path() gave -1 (gfid = 633515734)
2020-02-03T15:16:31 tid=35493 @ invoke_client_metaget_rpc() [margo_client.c:355] invoking the metaget rpc function in client
2020-02-03T15:16:31 tid=35493 @ invoke_client_metaget_rpc() [margo_client.c:364] Got response ret=2022
2020-02-03T15:16:31 tid=35493 @ unifyfs_fid_open() [unifyfs.c:1358] Creating a new entry for /unifyfs/newfile.
2020-02-03T15:16:31 tid=35493 @ unifyfs_fid_open() [unifyfs.c:1362] shm_super_buf = 0x20001bfe0000; free_fid_stack = 0x20001bfe0004; free_chunk_stack = 0x20001bff8c0c; unifyfs_filelist = 0x20001bfe010c; chunks = 0x20001c000000
2020-02-03T15:16:31 tid=35493 @ unifyfs_fid_alloc() [unifyfs.c:940] unifyfs_stack_pop() gave 4
2020-02-03T15:16:31 tid=35493 @ unifyfs_fid_create_file() [unifyfs.c:984] Filename /unifyfs/newfile got unifyfs fd 4
2020-02-03T15:16:31 tid=35493 @ invoke_client_metaset_rpc() [margo_client.c:322] invoking the metaset rpc function in client
2020-02-03T15:16:31 tid=35493 @ invoke_client_metaset_rpc() [margo_client.c:331] Got response ret=0
2020-02-03T15:16:31 tid=35493 @ unifyfs_fid_open() [unifyfs.c:1394] UNIFYFS_open generated fd 4 for file /unifyfs/newfile
2020-02-03T15:16:31 tid=35493 @ __wrap_open() [unifyfs-sysio.c:775] UNIFYFS_open generated fd 0 for file /unifyfs/newfile
opened fd = 16384
2020-02-03T15:16:31 tid=35493 @ unifyfs_intercept_fd() [unifyfs.c:452] Changing fd from exposed 16384 to internal 0
2020-02-03T15:16:31 tid=35493 @ __wrap_close() [unifyfs-sysio.c:2324] closing fd 0
2020-02-03T15:16:31 tid=35493 @ unifyfs_intercept_fd() [unifyfs.c:452] Changing fd from exposed 16384 to internal 0

$ echo "hello world" > /unifyfs/hello.txt
2020-02-03T15:09:40 tid=31165 @ __wrap_write() [unifyfs-sysio.c:1030] write was called for fd 82
2020-02-03T15:09:40 tid=31165 @ unifyfs_fid_open() [unifyfs.c:1261] unifyfs_get_fid_from_path() gave -1 (gfid = 1440053875)
2020-02-03T15:09:40 tid=31165 @ invoke_client_metaget_rpc() [margo_client.c:355] invoking the metaget rpc function in client
2020-02-03T15:09:40 tid=31165 @ invoke_client_metaget_rpc() [margo_client.c:364] Got response ret=2022
2020-02-03T15:09:40 tid=31165 @ unifyfs_fid_open() [unifyfs.c:1358] Creating a new entry for /unifyfs/hello.txt.
2020-02-03T15:09:40 tid=31165 @ unifyfs_fid_open() [unifyfs.c:1362] shm_super_buf = 0x20001bfe0000; free_fid_stack = 0x20001bfe0004; free_chunk_stack = 0x20001bff8c0c; unifyfs_filelist = 0x20001bfe010c; chunks = 0x20001c000000
2020-02-03T15:09:40 tid=31165 @ unifyfs_fid_alloc() [unifyfs.c:940] unifyfs_stack_pop() gave 2
2020-02-03T15:09:40 tid=31165 @ unifyfs_fid_create_file() [unifyfs.c:984] Filename /unifyfs/hello.txt got unifyfs fd 2
2020-02-03T15:09:40 tid=31165 @ invoke_client_metaset_rpc() [margo_client.c:322] invoking the metaset rpc function in client
2020-02-03T15:09:40 tid=31165 @ invoke_client_metaset_rpc() [margo_client.c:331] Got response ret=0
2020-02-03T15:09:40 tid=31165 @ unifyfs_fid_open() [unifyfs.c:1394] UNIFYFS_open generated fd 2 for file /unifyfs/hello.txt
2020-02-03T15:09:40 tid=31165 @ __wrap_open() [unifyfs-sysio.c:775] UNIFYFS_open generated fd 0 for file /unifyfs/hello.txt


$ var=$(< /unifyfs/hello)
2020-02-03T15:15:23 tid=34504 @ __wrap_write() [unifyfs-sysio.c:1030] write was called for fd 82
2020-02-03T15:15:23 tid=34913 @ unifyfs_get_fid_from_path() [unifyfs.c:507] File found: unifyfs_filelist[1].filename = /unifyfs/hello
2020-02-03T15:15:23 tid=34913 @ unifyfs_fid_open() [unifyfs.c:1261] unifyfs_get_fid_from_path() gave 1 (gfid = 1545952423)
2020-02-03T15:15:23 tid=34913 @ invoke_client_metaget_rpc() [margo_client.c:355] invoking the metaget rpc function in client
# NA -- Warning -- /g/g0/hutter2/opendir/deps/mercury/src/na/na_sm.c:2362
 # na_sm_progress_expected(): Ignored expected message received (canceled?)

Nothing else works (cat, cd, dd, etc) and busybox seems to silently crash when echo'ing to a file. So don't pop the champagne yet...

[tonyhutter]

I've been able to get the Unify client to build in a hacky way so that you don't need to use MPI (which was causing a bunch of problems with busybox). Now, I can get busybox to work a little better, but am running into an issue where busybox wants to call fcntl(fd, F_DUPFD_CLOEXEC, 16384) which is basically saying "make a copy of fd as a new file descriptor that's no less than 16384". We currently don't wrap fcntl(), so I'm going to add in the functionality for this and see how far the rabbit hole I can go...