new xsos v1 branch: split all modules into separate files

[ryran]

As it says on the tin ... with xsos v1, we'll be trading the single-file convenience for a path to less code complexity -- easier to find things, do maintenance; easier to add new features, etc. This branch does not add any new features (except potentially sliiightly less memory usage & load time when not using all modules).

The next step will be to rewrite the core in python and then slowly start migrating the modules to python as well, with the option for them to be in any language.

[ryran]

Check it out @lathan

[gangelop]

So much stuff!

[gangelop]

I have a question about XSOS_ASSET_PATH

# XSOS_ASSET_PATH (directory path)
#   xsos was split into multiple files starting with xsos v1
#   This directory must contain all of xsos' helper modules
    : ${XSOS_ASSET_PATH:="/usr/share/xsos"}

Is this going to be a strict requirement? Would it make sense to try and look for a module path in the dirname of wherever xsos is called from?

[ryran]

@lathan

Would it make sense to try and look for a module path in the dirname of wherever xsos is called from?

Yeah it's tempting isn't it? Something like:

# XSOS_ASSET_PATH (directory path)
    : ${XSOS_ASSET_PATH:="${0%/*}"}

This works if you call xsos from that directory directly, e.g.:

╔═[rsaw@xala 16:57 ~]$ ll /g/xsos/xsos
-rwxrwxr-x. 1 rsaw rsaw 25747 Jun 23 16:48 /g/xsos/xsos
╔═[rsaw@xala 16:57 ~]$ /g/xsos/xsos >/dev/null
╔═[rsaw@xala 16:58 ~]$ cd /g/xsos
╔═[rsaw@xala 16:58 xsos]$ ./xsos >/dev/null
╔═[rsaw@xala 16:58 xsos]$ 

But as soon as you move the executable to some bindir in your path (or create a symlink to it where it is, for that matter), things break.

╔═[rsaw@xala 16:58 xsos]$ ll $(which xsos)
lrwxrwxrwx. 1 root root 12 Jun  8 12:48 /usr/bin/xsos -> /g/xsos/xsos
╔═[rsaw@xala 16:58 xsos]$ xsos >/dev/null
/usr/bin/xsos: line 546: /usr/bin/modules/OS: No such file or directory
╔═[rsaw@xala 16:58 xsos]$ 

You may know this, but another cool thing about all those config variables is that you can override them in your shell environment without modifying xsos.

If using bash, you can permanently change any setting by putting it (as an exported environment variable) in one of your config files like ~/.bash_profile or some /etc/profile.d/xxx.sh file. Can even run xsos like:

XSOS_ASSET_PATH=. ./xsos

[gangelop]

See latest comment for summary of ramblings.

I was thinking of something "smarter". For example, checking first the "current" directory, then checking /usr/share/xsos/.

Of course, when programs try to be smart, they sometimes end up being very stupid. So I was trying to think if there is any other program that I'm aware of that works similarly. The only thing that comes to mind is how things like bash check for config files, but configs != modules.

The main thing that bugs me here is that I feel like it should be possible to just run xsos from the git repo, with no extra steps, or parameters, or env variables. So if this feeling is a good feeling, I'm assuming that some other project has already solved a similar problem. But maybe this is a bad feeling and the proper way is to just define a default path.

[gangelop]

More ramblings incoming!

See latest comment for summary of ramblings.

So another thing comes to mind. Python modules! How does python decide in which paths to look for modules when you do for example import this? Turns out, this is documented.

Basically, the python interpreter looks at three places and uses the first match. The first place it looks is "the directory containing the input script (or the current directory)."

Then I remembered, I've actually used this - writing python modules and importing them in a script in the same directory.

So at this point I think my feels are good feels.

[gangelop]

To summarize my ramblings:

My suggestion is to make xsos look in multiple locations for the modules with the current directory taking precedence - essentially something similar to how python looks for modules.

More specifically, this could be handled by the x function and $XSOS_ASSET_PATH could be a list of paths similar to the shell's $PATH and python's sys.path, and the first entry on the list would be ., then ./modules, then other stuff.

[ryran]

@lathan DONE.

Here's what I set it to by default:

$ grep -A3 '# XSOS_IMPORT_PATH' xsos
# XSOS_IMPORT_PATH (colon-delimited array of directory paths)
#   xsos was split into multiple files starting with xsos v1
#   These directories will be searched (in order) for modules
    : ${XSOS_IMPORT_PATH:="./modules:$HOME/.local/share/xsos:/usr/share/xsos"}

Demonstration:

╔═[rsaw@xala 01:30 ~]$ cd /g/xsos
╔═[rsaw@xala 01:31 xsos]$ ll
total 68
drwxrwxr-x. 2 rsaw rsaw  4096 Jul  1 01:07 modules
-rw-rw-r--. 1 rsaw rsaw 28893 Jul  1 01:01 README.md
-rwxrwxr-x. 1 rsaw rsaw 26894 Jul  1 01:27 xsos
-rw-rw-r--. 1 rsaw rsaw  3321 Jun 23 15:09 xsos-bash-completion.bash
╔═[rsaw@xala 01:31 xsos]$ ./xsos -r
SOFTIRQ
  Backlog max is sufficient (Current value: net.core.netdev_max_backlog = 1000)
  Budget is sufficient (Current value: net.core.netdev_budget = 300)

╔═[rsaw@xala 01:31 xsos]$ cd /tmp
╔═[rsaw@xala 01:31 tmp]$ /g/xsos/xsos -r
ERROR: To use the SOFTIRQ module, one of the $XSOS_IMPORT_PATH dirs must contain 'SOFTIRQ'
Current value of $XSOS_IMPORT_PATH: './modules:/home/rsaw/.local/share/xsos:/usr/share/xsos'

Your choices:
  * cd to the dir you downloaded xsos to & run it there via ./xsos
  * Copy or symlink the xsos/modules dir to one of the dirs mentioned above
  * Copy/symlink specific (desired) modules to one of the dirs mentioned above
  * Change the $XSOS_IMPORT_PATH env variable to point to an xsos module dir
    This can be done at runtime, e.g.:
      XSOS_IMPORT_PATH=~/git/xsos/modules xsos -ya
    Or it can be done by editing your shell environment, e.g.:
      echo 'export XSOS_IMPORT_PATH=$HOME/git/xsos/modules' >>~/.bash_profile
╔═[rsaw@xala 01:31 tmp]$ mkdir -p ~/.local/share/xsos
╔═[rsaw@xala 01:32 tmp]$ cp /g/xsos/modules/SOFTIRQ ~/.local/share/xsos
╔═[rsaw@xala 01:32 tmp]$ /g/xsos/xsos -r
SOFTIRQ
  Backlog max is sufficient (Current value: net.core.netdev_max_backlog = 1000)
  Budget is sufficient (Current value: net.core.netdev_budget = 300)

╔═[rsaw@xala 01:32 tmp]$ sudo ln -sv /g/xsos/modules /usr/share/xsos
‘/usr/share/xsos’ -> ‘/g/xsos/modules’
╔═[rsaw@xala 01:33 tmp]$ xsos -d
STORAGE
  Whole Disks from /proc/partitions:
    1 disks, totaling 238 GiB (0.23 TiB)
    - - - - - - - - - - - - - - - - - - - - -
    Disk    Size in GiB
    ----    -----------
    sda     238

[ryran]

PS @lathan I had "." in the path for a while, but decided to remove it in deference to the security gods.

[gangelop]

:+1: :tada: :tada: Awesome. Thank you!

[gangelop]

Hey @ryran

I was just wondering what the status of this is. I want to work on some other issues but it looks like any work being done on master branch now will need to be manually imported here. Any advice?