Talk

I just gave a talk about this at [[https://www.socallinuxexpo.org/scale/17x][SCaLE 17x]]. Here are the [[https://www.youtube.com/watch?v=Qvb_uNkFGNQ&t=12830s][video of the talk]] and the [[https://github.com/dkogan/talk-feedgnuplot-vnlog/blob/master/feedgnuplot-vnlog.org]["slides"]].

Summary

Vnlog ("vanilla-log") is a toolkit for manipulating tabular ASCII data with labelled fields using normal UNIX tools. If you regularly use =awk= and =sort= and =uniq= and others, these tools will make you infinitely more powerful. The vnlog tools /extend/, rather than replace the standard tooling, so minimal effort is required to learn and use these tools.

Everything assumes a trivially simple log format:

A whitespace-separated table of ASCII human-readable text
A =#= character starts a comment that runs to the end of the line (like in many scripting languages)
The first line that begins with a single =#= (not =##= or =#!=) is a /legend/, naming each column. This is required, and the field names that appear here are referenced by all the tools.
Empty fields reported as =-=

This describes 99% of the format, with some extra details [[#format-details][below]]. Example:

#+BEGIN_EXAMPLE #!/usr/bin/whatever

a b c

1 2 3

comment

4 5 6 #+END_EXAMPLE

Such data can be processed directly with almost any existing tool, and /this/ toolkit allows the user to manipulate this data in a nicer way by relying on standard UNIX tools. The core philosophy is to avoid creating new knowledge as much as possible. Consequently, the vnlog toolkit relies /heavily/ on existing (and familiar!) tools and workflows. As such, the toolkit is small, light, and has a /very/ friendly learning curve.

Synopsis

I have [[https://raw.githubusercontent.com/dkogan/vnlog/master/dji-tsla.tar.gz][two sets of historical stock data]], from the start of 2018 until now (2018/11):

#+BEGIN_SRC sh :results output :exports both < dji.vnl head -n 4 #+END_SRC

#+RESULTS: : # Date Open High Low Close AdjClose Volume : 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 : 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 : 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000

And

#+BEGIN_SRC sh :results output :exports both < tsla.vnl head -n 4 #+END_SRC

#+RESULTS: : # Date Open High Low Close AdjClose Volume : 2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339 : 2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300 : 2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600

I can add whitespace to make the headers more legible by humans:

#+BEGIN_SRC sh :results output :exports both < dji.vnl head -n 4 | vnl-align #+END_SRC

I can pull out the closing prices:

#+BEGIN_SRC sh :results output :exports both < dji.vnl vnl-filter -p Close | head -n4 #+END_SRC

#+RESULTS: : # Close : 25289.27 : 25080.50 : 25286.49

=vnl-filter= is primarily a wrapper around =awk= or =perl=, allowing the user to reference columns by name. I can then plot the closing prices:

#+BEGIN_SRC sh :results file link :exports both < dji.vnl vnl-filter -p Close | feedgnuplot --lines --unset grid #+END_SRC

#+RESULTS: [[file:guide-1.svg]]

Here I kept /only/ the closing price column, so the x-axis is just the row index. The data was in reverse chronological order, so this plot is also in reverse chronological order. Let's fix that:

#+BEGIN_SRC sh :results file link :exports both < dji.vnl vnl-sort -k Date | vnl-filter -p Close | feedgnuplot --lines --unset grid #+END_SRC

#+RESULTS: [[file:guide-2.svg]]

The =vnl-sort= tool (and most of the other =vnl-xxx= tools) are wrappers around the core tools already available on the system (such as =sort=, in this case). With the primary difference being reading/writing vnlog, and referring to columns by name.

We now have the data in the correct order, but it'd be nice to see the actual dates on the x-axis. While we're at it, let's label the axes too:

#+BEGIN_SRC sh :results output :exports both < dji.vnl vnl-filter -p Date,Close | head -n4 #+END_SRC

#+RESULTS: : # Date Close : 2018-11-15 25289.27 : 2018-11-14 25080.50 : 2018-11-13 25286.49

#+BEGIN_SRC sh :results file link :exports both < dji.vnl vnl-sort -k Date | vnl-filter -p Date,Close | feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain
--xlabel 'Date' --ylabel 'Price ($)' #+END_SRC

#+RESULTS: [[file:guide-3.svg]]

What was the highest value of the Dow-Jones index, and when did it happen?

#+BEGIN_SRC sh :results output :exports both < dji.vnl vnl-sort -rgk Close | head -n2 | vnl-align #+END_SRC

#+RESULTS: : # Date Open High Low Close AdjClose Volume : 2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000

Alrighty. Looks like the high was in October. Let's zoom in on that month:

#+BEGIN_SRC sh :results file link :exports both < dji.vnl vnl-sort -k Date | vnl-filter 'Date ~ /2018-10/' -p Date,Close | feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain
--xlabel 'Date' --ylabel 'Price ($)' #+END_SRC

#+RESULTS: [[file:guide-4.svg]]

OK. Is this thing volatile? What was the largest single-day gain?

#+BEGIN_SRC sh :results output :exports both < dji.vnl vnl-filter -p '.,d=diff(Close)' | head -n4 | vnl-align #+END_SRC

#+RESULTS: : # Date Open High Low Close AdjClose Volume d
: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 -
: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 -208.77 : 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 205.99

#+BEGIN_SRC sh :results output :exports both < dji.vnl vnl-filter -p '.,d=diff(Close)' | vnl-sort -rgk d | head -n2 | vnl-align #+END_SRC

#+RESULTS: : # Date Open High Low Close AdjClose Volume d
: 2018-02-02 26061.79 26061.79 25490.66 25520.96 25520.96 522880000 1175.21

Whoa. So the best single-gain day was 2018-02-02: the dow gained 1175.21 points between closing on Feb 1 and Feb 2. But it actually lost ground that day! What if I looked at the difference between the opening and closing in a single day?

#+BEGIN_SRC sh :results output :exports both < dji.vnl vnl-filter -p '.,d=Close-Open' | vnl-sort -rgk d | head -n2 | vnl-align #+END_SRC

#+RESULTS: : # Date Open High Low Close AdjClose Volume d
: 2018-02-06 24085.17 24946.23 23778.74 24912.77 24912.77 823940000 827.6

I guess by that metric 2018-02-06 was better. Let's join the Dow-jones index data and the TSLA data, and let's look at them together:

#+BEGIN_SRC sh :results output :exports both vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date | head -n4 | vnl-align #+END_SRC

#+RESULTS: : # Date Open_dji High_dji Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla : 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 342.33 348.58 339.04 348.44 348.44 4486339
: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 342.70 347.11 337.15 344.00 344.00 5036300
: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 333.16 344.70 332.20 338.73 338.73 5448600

#+BEGIN_SRC sh :results output :exports both vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date | vnl-filter -p '^Close' | head -n4 | vnl-align #+END_SRC

#+RESULTS: : # Close_dji Close_tsla : 25289.27 348.44
: 25080.50 344.00
: 25286.49 338.73

#+BEGIN_SRC sh :results file link :exports both vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date | vnl-filter -p '^Close' | feedgnuplot --domain --points --unset grid
--xlabel 'DJI price ($)' --ylabel 'TSLA price ($)' #+END_SRC

#+RESULTS: [[file:guide-5.svg]]

Huh. Apparently there's no obvious, strong correlation between TSLA and Dow-Jones closing prices. And we saw that with just a few shell commands, without dropping down into a dedicated analysis system.

Build and installation vnlog is a part of Debian/buster and Ubuntu/cosmic (18.10) and later. On those boxes you can simply

#+BEGIN_EXAMPLE $ sudo apt install vnlog libvnlog-dev libvnlog-perl python3-vnlog #+END_EXAMPLE

to get the binary tools, the C API, the perl and python3 interfaces respectively.

** Install on non-Debian boxes Most of this is written in an interpreted language, so there's nothing to build or install, and you can run the tools directly from the source tree:

#+BEGIN_EXAMPLE $ git clone https://github.com/dkogan/vnlog.git $ cd vnlog $ ./vnl-filter ..... #+END_EXAMPLE

The python and perl libraries can be run from the tree by setting the =PYTHONPATH= and =PERL5LIB= environment variables respectively. For the C library, you should =make=, and then point your =CFLAGS= and =LDLIBS= and =LD_LIBRARY_PATH= to the local tree.

If you do want to install to some arbitrary location to simplify the paths, do this:

#+BEGIN_EXAMPLE $ make $ PREFIX=/usr/local make install #+END_EXAMPLE

This will install /all/ the components into =/usr/local=.

Description Vnlog data is nicely readable by both humans and machines. Any time your application invokes =printf()= for either diagnostics or logging, consider writing out vnlog-formatted data. You retain human readability, but gain the power all the =vnl-...= tools provide.

Vnlog tools are designed to be very simple and light. There's an ever-growing list of other tools that do vaguely the same thing. Some of these:

https://github.com/BurntSushi/xsv
https://csvkit.readthedocs.io/
https://github.com/johnkerl/miller
https://github.com/jqnatividad/qsv
https://github.com/greymd/teip
https://github.com/eBay/tsv-utils-dlang
https://www.gnu.org/software/datamash/
https://stedolan.github.io/jq/
https://github.com/benbernard/RecordStream
https://github.com/dinedal/textql
https://www.visidata.org/
http://harelba.github.io/q/
https://github.com/BatchLabs/charlatan
https://github.com/dbohdan/sqawk

Many of these provide facilities to run various analyses, and others focus on data types that aren't just a table (json for instance). Vnlog by contrast doesn't analyze anything, and targets the most trivial possible data format. This makes it very easy to run any analysis you like in any tool you like. The main envisioned use case is one-liners, and the tools are geared for that purpose. The above mentioned tools are much more powerful than vnlog, so they could be a better fit for some use cases. I claim that

90% of the time you want to do simple things, and vnlog is a great fit for the task
If you really do need to do something complex, you shouldn't be in the shell writing oneliners anymore, and a fully-fledged analysis system (numpy, etc) is more appropriate

In the spirit of doing as little as possible, the provided tools are wrappers around tools you already have and are familiar with. The provided tools are:

=vnl-filter= is a tool to select a subset of the rows/columns in a vnlog and/or to manipulate the contents. This is an =awk= wrapper where the fields can be referenced by name instead of index. 20-second tutorial:

#+BEGIN_SRC sh :results none :exports code vnl-filter -p col1,col2,colx=col3+col4 'col5 > 10' --has col6 #+END_SRC

will read the input, and produce a vnlog with 3 columns: =col1= and =col2= from the input, and a column =colx= that's the sum of =col3= and =col4= in the input. Only those rows for which /both/ =col5 > 10= is true /and/ that have a non-null value for =col6= will be output. A null entry is signified by a single =-= character.

#+BEGIN_SRC sh :results none :exports code vnl-filter --eval '{s += x} END {print s}' #+END_SRC

#+RESULTS:

will evaluate the given awk program on the input, but the column names work as you would hope they do: if the input has a column named =x=, this would produce the sum of all values in this column.

=vnl-sort=, =vnl-uniq=, =vnl-join=, =vnl-tail=, =vnl-ts= are wrappers around the corresponding commandline tools. These work exactly as you would expect also: the columns can be referenced by name, and the legend comment is handled properly. These are wrappers, so all the commandline options those tools have "just work" (except options that don't make sense in the context of vnlog). As an example, =vnl-tail -f= will follow a log: data will be read by =vnl-tail= as it is written into the log (just like =tail -f=, but handling the legend properly). And you already know how to use these tools without even reading the manpages! Note: I use the Linux kernel and the tools from GNU Coreutils exclusively, but this all has been successfully tested on FreeBSD and OSX also. Please let me know if something doesn't work.
=vnl-align= aligns vnlog columns for easy interpretation by humans. The meaning is unaffected
=Vnlog::Parser= is a simple perl library to read a vnlog
=vnlog= is a simple python library to read a vnlog. Both python2 and python3 are supported
=libvnlog= is a C library to simplify reading and writing a vnlog. Clearly all you /really/ need for writing is =printf()=, but this is useful if we have lots of columns, many containing null values in any given row, and/or if we have parallel threads writing to a log. In my usage I have hundreds of columns of sparse data, so this is handy
=vnl-make-matrix= converts a one-point-per-line vnlog to a matrix of data. I.e.

#+BEGIN_EXAMPLE $ cat dat.vnl

i j x

0 0 1 0 1 2 0 2 3 1 0 4 1 1 5 1 2 6 2 0 7 2 1 8 2 2 9 3 0 10 3 1 11 3 2 12

$ < dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp Writing to '/tmp/x.matrix'

$ cat /tmp/x.matrix 1 2 3 4 5 6 7 8 9 10 11 12 #+END_EXAMPLE

All the tools have manpages that contain more detail. And more tools will probably be added with time.

Format details The high-level description of the vnlog format from [[#Summary][above]] is sufficient to read/write "normal" vnlog data, but there are a few corner cases that should be mentioned. To reiterate, the format description from above describes vnlog as:

A whitespace-separated table of ASCII human-readable text
A =#= character starts a comment that runs to the end of the line (like in many scripting languages)
The first line that begins with a single =#= (not =##= or =#!=) is a /legend/, naming each column. This is required, and the field names that appear here are referenced by all the tools.
Empty fields reported as =-=

For a few years now I've been using these tools myself, and supporting others as they were passing vnlog data around. In the process I've encountered some slightly-weird data, and patched the tools to accept it. So today the included vnlog tools are /very/ permissive, and accept any vnlog data that can possibly be accepted. Other vnlog tools may not be quite as permissive, and may not be able to interpret "weird" data. Points of note, describing the included vnlog tools:

Leading and trailing whitespace is ignored. Everywhere. So this data file will be read properly, with the =x= column containing 1 and 3:

#+begin_example

x y

1 2 3 4 #+end_example

Empty (or whitespace-only) lines anywhere are ignored, and treated as a comment
An initial =#= comment without field names is treated as a comment, and we continue looking for the legend in the following lines. So this data file will be read properly:

#+begin_example

comment

x y

1 2 3 4 #+end_example

Trailing comments are supported, like in most scripting languages. So this data file will be read properly:

#+begin_example

x y

1 2 # comment 3 4 #+end_example

Field names are /very/ permissive: anything that isn't whitespace is supported. So this data file will be read properly:

#+begin_example

x y # 1+ -

1 2 3 4 5 11 12 13 14 15 #+end_example

We can pull out the =#= and =1+= and =-= columns:

#+begin_src sh vnl-filter -p '#,1+,-' #+end_src

And we can even operate on them, if we use whitespace to indicate field boundaries:

#+begin_src sh vnl-filter -p 'x=1+ + 5' #+end_src

Note that this implies that trailing comments in a legend line are /not/ supported: the extra =#= characters will be used for field names. Field names containing =,= or === are currently not accepted by =vnl-filter=, but /are/ accepted by the other tools (=vnl-sort= and such). I'll make =vnl-filter= able to work with those field names too, eventually, but as a user, the simplest thing to do is to not pass around data with such field names.

Duplicated labels are supported whenever possible. So

#+begin_example

x y z z

1 2 3 4 11 12 13 14 #+end_example

will work just fine, unless we're operating on =z=. With this data, both of these commands work:

#+begin_src sh vnl-filter -p x vnl-filter -p z #+end_src

Picking =z= selects both of the =z= columns. But neither of these commands can work with the non-unique =z= column:

#+begin_src sh vnl-filter -p s=z+1 vnl-sort -k z #+end_src

Workflows and recipes ** Storing disjoint data

A common use case is a complex application that produces several semi-related subsets of data at once. Example: a moving vehicle is reporting both its own position and the observed positions of other vehicles; at any given time any number of other vehicles may be observed. Two equivalent workflows are possible:

a single unified vnlog stream for /all/ the data
several discrete vnlog streams for each data subset

Both are valid approaches

*** One unified vnlog stream Here the application produces a /single/ vnlog that contains /all/ the columns, from /all/ the data subsets. In any given row, many of the columns will be empty (i.e. contain only =-= ). For instance, a row describing a vehicle own position will not have data about any observations, and vice versa. It is inefficient to store all the extra =-= but it makes many things much nicer, so it's often worth it. =vnl-filter= can be used to pull out the different subsets. Sample =joint.vnl=:

#+BEGIN_EXAMPLE

time x_self x_observation

1 10 - 2 20 - 2 - 100 3 30 - 3 - 200 3 - 300 #+END_EXAMPLE

Here we have 3 instances in time. We have no observations at =time= 1, one observation at =time= 2, and two observations at =time= 3. We can use =vnl-filter= to pull out the data we want:

#+BEGIN_EXAMPLE $ < joint.vnl vnl-filter -p time,self

time x_self

1 10 2 20 2 - 3 30 3 - 3 - #+END_EXAMPLE

If we only care about our own positions, the =+= modifier in picked columns in =vnl-filter= is very useful here:

#+BEGIN_EXAMPLE $ < joint.vnl vnl-filter -p time,+self

time x_self

1 10 2 20 3 30

$ < joint.vnl vnl-filter -p time,+observation

time x_observation

2 100 3 200 3 300 #+END_EXAMPLE

Note that the default is =--skipempty=, so if we're /only/ looking at =x_self= for instance, then we don't even need to =+= modifier:

#+begin_example $ < joint.vnl vnl-filter -p self

x_self

10 20 30 #+end_example

Also, note that the =vnlog= C interface works very nicely to produce these datafiles:

You can define lots and lots of columns, but only fill some of them before calling =vnlog_emit_record()=. The rest will be set to =-=.
You can create multiple contexts for each type of data, and you can populate them with data independently. And when calling =vnlog_emit_record_ctx()=, you'll get a record with data for just that context.

*** Several discrete vnlog streams

Conversely, the application can produce /separate/ vnlog streams for /each/ subset of data. Depending on what is desired, exactly, =vnl-join= can be used to re-join them:

#+BEGIN_EXAMPLE $ cat self.vnl

time x_self

1 10 2 20 3 30

$ cat observations.vnl

time x_observation

2 100 3 200 3 300

$ vnl-join -j time -a- self.vnl observations.vnl

time x_self x_observation

1 10 - 2 20 100 3 30 200 3 30 300 #+END_EXAMPLE

** Data statistics

A common need is to compute basic statistics from your data. Many of the alternative toolkits listed above provide built-in facilities to do this, but vnlog does not: it's meant to be unixy, where each tool has very limited scope. Thus you can either do this with =awk= like you would normally, or you can use other standalone tools to perform the needed computations. For instance, I can generate some data:

#+BEGIN_EXAMPLE $ seq 2 100 | awk 'BEGIN {print "# x"} {print log($1)}' > /tmp/log.vnl #+END_EXAMPLE

Then I can compute the mean with =awk=:

#+BEGIN_EXAMPLE $ < /tmp/log.vnl vnl-filter --eval '{sum += x} END {print sum/NR}' 3.67414 #+END_EXAMPLE

Or I can compute the mean (and other stuff) with a separate standalone tool:

#+BEGIN_EXAMPLE $ < /tmp/log.vnl ministat x +----------------------------------------------------------------------------+ | xx | | x xxxxxxx | | xx xxxxxxxxxxxx| | x x xxxxxxxxxxxxxxxxxxxxxxx| |x x x x x x x x x xx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |____A____M| | +----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 99 0.693147 4.60517 3.93183 3.6741353 0.85656382 #+END_EXAMPLE

=ministat= is not a part of the vnlog toolkit, but the vnlog format is generic so it works just fine.

** Powershell-style filtering of common shell commands

Everything about vnlog is generic and simple, so it's easy to use it to process data that wasn't originally meant to be used this way. For instance filtering the output of =ls -l= to report only file names and sizes, skipping directories, and sorting by file sizes:

#+BEGIN_EXAMPLE $ ls -l

total 320 -rw-r--r-- 1 dima dima 5044 Aug 25 15:04 Changes -rw-r--r-- 1 dima dima 12749 Aug 25 15:04 Makefile -rw-r--r-- 1 dima dima 69789 Aug 25 15:04 README.org -rw-r--r-- 1 dima dima 33781 Aug 25 15:04 README.template.org -rw-r--r-- 1 dima dima 5359 Aug 25 15:04 b64_cencode.c drwxr-xr-x 4 dima dima 4096 Aug 25 15:04 completions drwxr-xr-x 3 dima dima 4096 Aug 25 15:04 lib drwxr-xr-x 3 dima dima 4096 Aug 25 15:04 packaging drwxr-xr-x 2 dima dima 4096 Aug 25 15:04 test -rwxr-xr-x 1 dima dima 5008 Aug 25 15:04 vnl-align -rwxr-xr-x 1 dima dima 56637 Aug 25 15:04 vnl-filter -rwxr-xr-x 1 dima dima 5678 Aug 25 15:04 vnl-gen-header -rwxr-xr-x 1 dima dima 29815 Aug 25 15:04 vnl-join -rwxr-xr-x 1 dima dima 3631 Aug 25 15:04 vnl-make-matrix -rwxr-xr-x 1 dima dima 8372 Aug 25 15:04 vnl-sort -rwxr-xr-x 1 dima dima 5822 Aug 25 15:04 vnl-tail -rwxr-xr-x 1 dima dima 4439 Aug 25 15:04 vnl-ts -rw-r--r-- 1 dima dima 559 Aug 25 15:04 vnlog-base64.h -rw-r--r-- 1 dima dima 8169 Aug 25 15:04 vnlog.c -rw-r--r-- 1 dima dima 12677 Aug 25 15:04 vnlog.h

$ (echo '# permissions num_links user group size month day time name'; ls -l | tail -n +2) | vnl-filter 'permissions !~ "^d"' -p name,size | vnl-sort -gk size | vnl-align

name size

vnlog-base64.h 559 vnl-make-matrix 3631 vnl-ts 4439 vnl-align 5008 Changes 5044 b64_cencode.c 5359 vnl-gen-header 5678 vnl-tail 5822 vnlog.c 8169 vnl-sort 8372 vnlog.h 12677 Makefile 12749 vnl-join 29815 README.template.org 33781 vnl-filter 56637 README.org 69789 #+END_EXAMPLE

With a bit of shell manipulation, these tools can be applied to a whole lot of different data streams that know nothing of vnlog.

C interface ** Writing vnlog files *** Basic usage For most uses, vnlog files are simple enough to be generated with plain prints. But then each print statement has to know which numeric column we're populating, which becomes effortful with many columns. In my usage it's common to have a large parallelized C program that's writing logs with hundreds of columns where any one record would contain only a subset of the columns. In such a case, it's helpful to have a library that can output the log files. This is available. Basic usage looks like this:

In a shell:

#+BEGIN_SRC sh :results none :exports code vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' > vnlog_fields_generated.h #+END_SRC

#+RESULTS:

In a C program test.c:

#+BEGIN_SRC C #include "vnlog_fields_generated.h"

int main() { vnlog_emit_legend();

vnlog_set_field_value__w(-10);
vnlog_set_field_value__x(40);
vnlog_set_field_value__y("asdf");
vnlog_emit_record();

vnlog_set_field_value__z(0.3);
vnlog_set_field_value__x(50);
vnlog_set_field_value__w(-20);
vnlog_set_field_value__binary("\x01\x02\x03", 3);
vnlog_emit_record();

vnlog_set_field_value__w(-30);
vnlog_set_field_value__x(10);
vnlog_set_field_value__y("whoa");
vnlog_set_field_value__z(0.5);
vnlog_emit_record();

return 0;

} #+END_SRC

Then we build and run, and we get

#+BEGIN_EXAMPLE $ cc -o test test.c -lvnlog

$ ./test

w x y z binary

-10 40 asdf - - -20 50 - 0.2999999999999999889 AQID -30 10 whoa 0.5 - #+END_EXAMPLE

The binary field in base64-encoded. This is a rarely-used feature, but sometimes you really need to log binary data for later processing, and this makes it possible.

So you

Generate the header to define your columns
Call =vnlog_emit_legend()=
Call =vnlog_set_field_value__...()= for each field you want to set in that row.
Call =vnlog_emit_record()= to write the row and to reset all fields for the next row. Any fields unset with a =vnlog_set_field_value__...()= call are written as null: =-=

This is enough for 99% of the use cases. Things get a bit more complex if we have have threading or if we have multiple vnlog ouput streams in the same program. For both of these we use vnlog /contexts/.

*** Contexts

To support independent writing into the same vnlog (possibly by multiple threads; this is reentrant), each log-writer should create a context, and use it when talking to vnlog. The context functions will make sure that the fields in each context are independent and that the output records won't clobber each other:

#+BEGIN_SRC C void child_writer( // the parent context also writes to this vnlog. Pass NULL to // use the global one struct vnlog_context_t* ctx_parent ) { struct vnlog_context_t ctx; vnlog_init_child_ctx(&ctx, ctx_parent);

while(records)
{
    vnlog_set_field_value_ctx__xxx(&ctx, ...);
    vnlog_set_field_value_ctx__yyy(&ctx, ...);
    vnlog_set_field_value_ctx__zzz(&ctx, ...);
    vnlog_emit_record_ctx(&ctx);
}

vnlog_free_ctx(&ctx); // required only if we have any binary fields

} #+END_SRC

If we want to have multiple independent vnlog writers to /different/ streams (with different columns and legends), we do this instead:

=file1.c=: #+BEGIN_SRC C #include "vnlog_fields_generated1.h"

void f(void) { // Write some data out to the default context and default output (STDOUT) vnlog_emit_legend(); ... vnlog_set_field_value__xxx(...); vnlog_set_field_value__yyy(...); ... vnlog_emit_record(); } #+END_SRC

=file2.c=: #+BEGIN_SRC C #include "vnlog_fields_generated2.h"

void g(void) { // Make a new session context, send output to a different file, write // out legend, and send out the data struct vnlog_context_t ctx; vnlog_init_session_ctx(&ctx); FILE* fp = fopen(...); vnlog_set_output_FILE(&ctx, fp); vnlog_emit_legend_ctx(&ctx); ... vnlog_set_field_value__a(...); vnlog_set_field_value__b(...); ... vnlog_free_ctx(&ctx); // required only if we have any binary fields vnlog_emit_record(); } #+END_SRC

Note that it's the user's responsibility to make sure the new sessions go to a different =FILE= by invoking =vnlog_set_output_FILE()=. Furthermore, note that the included =vnlog_fields_....h= file defines the fields we're writing to; and if we have multiple different vnlog field definitions in the same program (as in this example), then the different writers /must/ live in different source files. The compiler will barf if you try to =#include= two different =vnlog_fields_....h= files in the same source.

*** Remaining APIs

=vnlog_printf(...)= and =vnlog_printf_ctx(ctx, ...)= write to a pipe like =printf()= does. This exists primarily for comments.
=vnlog_clear_fields_ctx(ctx, do_free_binary)= clears out the data in a context and makes it ready to be used for the next record. It is rare for the user to have to call this manually. The most common case is handled automatically (clearing out a context after emitting a record). One area where this is useful is when making a copy of a context:

#+BEGIN_SRC C struct vnlog_context_t ctx1; // .... do stuff with ctx1 ... add data to it ...

struct vnlog_context_t ctx2 = ctx1; // ctx1 and ctx2 now both have the same data, and the same pointers to // binary data. I need to get rid of the pointer references in ctx1

vnlog_clear_fields_ctx(&ctx1, false); #+END_SRC

=vnlog_free_ctx(ctx)= frees memory for an vnlog context. Do this before throwing the context away. Currently this is only needed for context that have binary fields, but this should be called for all contexts anyway, in case this changes in a later revision

** Reading vnlog files The basic usage goes like this:

#+begin_src c #include <stdio.h> #include <stdbool.h> #include <vnlog/vnlog-parser.h> bool parse_vnlog(const char* filename) { FILE* fp = fopen(filename); if(fp == NULL) return false;

vnlog_parser_t ctx;
if(VNL_OK != vnlog_parser_init(&ctx, fp))
    return false;

// String in the "time" column for the most-recently-parsed row
const char*const* time_record = vnlog_parser_record_from_key(&ctx, "time");
if(time_record == NULL)
{
    vnlog_parser_free(&ctx);
    return false;
}

int i_record = 0;
vnlog_parser_result_t result;
while(VNL_OK == (result = vnlog_parser_read_record(&ctx, fp)))
{
    for(int i=0; i<ctx.Ncolumns; i++)
        printf("Record %d: %s = %s\n",
               i_record,
               ctx.record[i].key, ctx.record[i].value);
    printf("Record %d: time = %s\n",
           i_record, *time_record);
    i_record++;
}

vnlog_parser_free(&ctx);
return true;

} #+end_src

The usage should be clear from this example. See =vnlog-parser.h= for details.

** Base64 interface The C interface supports writing base64-encoded binary data using Chris Venter's libb64. The base64-encoder used here was slightly modified: the output appears all on one line, making is suitable to appear in a vnlog field. If we're writing a vnlog with =printf()= directly without using the =vnlog.h= interface described above, we allow this modified base64 encoder to be invoked by itself. Usage:

#+BEGIN_SRC C void* binary_buffer = ...; int binary_buffer_len = ...;

char base64_buffer[vnlog_base64_dstlen_to_encode(binary_buffer_len)]; vnlog_base64_encode( base64_buffer, sizeof(base64_buffer), binary_buffer, binary_buffer_len ); #+END_SRC

Clearly the above example allocates the base64 buffer on the stack, so it's only suitable for small-ish data chunks. But if you have lots and lots of data, probably writing it as base64 into a vnlog isn't the best thing to do.

Python interface Reading vnlog data into a python program is simple. The =vnlog= Python module provides three different ways to do that:

slurp the whole thing into a numpy array using the =slurp()= function. Basic usage:

#+begin_src python import vnlog log_numpy_array,list_keys,dict_key_index =
vnlog.slurp(filename_or_fileobject) #+end_src

This parses out the legend, and then calls =numpy.loadtxt()=. Null data values (=-=) are not supported
Iterate through the records: =vnlog= class, used as an iterator. Basic usage:

#+begin_src python import vnlog for d in vnlog.vnlog(filename_or_fileobject): print(d['time'],d['height']) #+end_src

Null data values are represented as =None=
Parse incoming lines individually: =vnlog= class, using the =parse()= method. Basic usage:

#+begin_src python import vnlog parser = vnlog.vnlog() for l in file: parser.parse(l) d = parser.values_dict() if not d: continue print(d['time'],d['height']) #+end_src

Most of the time you'd use options 1 or 2 above. Option 3 is the most general, but also the most verbose

numpy interface If we need to read data into numpy specifically, nicer tools are available than the generic =vnlog= Python module. The built-in =numpy.loadtxt= =numpy.savetxt= functions work well (with the caveat that =numpy.loadtxt()= should be followed by =numpysane.atleast_dims(..., -2)= to make sure that a data array of shape =(Nrows,Ncols)= is returned even if =Nrows==1=. For example to write to standard output a vnlog with fields =a=, =b= and =c=:

#+BEGIN_SRC python numpy.savetxt(sys.stdout, array, fmt="%g", header="a b c") #+END_SRC

Note that numpy automatically adds the =#= to the header. To read a vnlog from a file on disk, do something like

#+BEGIN_SRC python array = numpysane.atleast_dims(numpy.loadtxt('data.vnl'), -2) #+END_SRC

These functions know that =#= lines are comments, but don't interpret anything as field headers. That's easy to do, so I'm not providing any helper libraries. I might do that at some point, but in the meantime, patches are welcome.

Compatibility

I use GNU/Linux-based systems exclusively, but everything has been tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and CentOS. I can imagine there's something I missed when testing on non-Linux systems, so please let me know if you find any issues.

Caveats and bugs

These tools are meant to be simple, so some things are hard requirements. A big one is that columns are whitespace-separated. There is /no/ mechanism for escaping or quoting whitespace into a single field. I think supporting something like that is more trouble than it's worth.

Manpages ** vnl-filter #+BEGIN_EXAMPLE NAME vnl-filter - filters vnlogs to select particular rows, fields

SYNOPSIS $ cat run.vnl

 # time x   y   z   temperature
 3      1   2.3 4.8 30
 4      1.1 2.2 4.7 31
 6      1   2.0 4.0 35
 7      1   1.6 3.1 42


 $ <run.vnl vnl-filter -p x,y,z | vnl-align

 # x  y   z
 1   2.3 4.8
 1.1 2.2 4.7
 1   2.0 4.0
 1   1.6 3.1


 $ <run.vnl vnl-filter -p i=NR,time,'dist=sqrt(x*x + y*y + z*z)' | vnl-align

 # i time   dist
 1   3    5.41572
 2   4    5.30471
 3   6    4.58258
 4   7    3.62905


 $ <run.vnl vnl-filter 'temperature >= 35' | vnl-align

 # time x  y   z  temperature
 6      1 2.0 4.0 35
 7      1 1.6 3.1 42



 $ <run.vnl vnl-filter --eval '{s += temperature} END { print "mean temp: " s/NR}'

 mean temp: 34.5


 $ <run.vnl vnl-filter -p x,y | feedgnuplot --terminal 'dumb 80,30' --unset grid --domain --lines --exit

   2.3 +---------------------------------------------------------------------+
       |           +          +          ***************         +           |
       |                                                **************       |
       |                                                              *******|
   2.2 |-+                                                       ************|
       |                                                 ********            |
       |                                         ********                    |
   2.1 |-+                              *********                          +-|
       |                        ********                                     |
       |                ********                                             |
       |            ****                                                     |
     2 |-+         *                                                       +-|
       |           *                                                         |
       |           *                                                         |
       |           *                                                         |
   1.9 |-+         *                                                       +-|
       |           *                                                         |
       |           *                                                         |
       |           *                                                         |
   1.8 |-+         *                                                       +-|
       |           *                                                         |
       |           *                                                         |
   1.7 |-+         *                                                       +-|
       |           *                                                         |
       |           *                                                         |
       |           *          +           +           +          +           |
   1.6 +---------------------------------------------------------------------+
      0.98         1         1.02        1.04        1.06       1.08        1.1

DESCRIPTION This tool is largely a frontend for awk to operate on vnlog files. Vnlog is both an input and an output. This tool makes it very simple to select specific rows and columns for output and to manipulate the data in various ways.

This is a UNIX-style tool, so the input/output of this tool is strictly
STDIN/STDOUT. Furthermore, in its usual form this tool is a filter, so
the format of the output is *exactly* the same as the format of the
input. The exception to this is when using "--eval", in which the output
is dependent on whatever expression we're evaluating.

This tool is convenient to process both stored data or live data; in the
latter case, it's very useful to pipe the streaming output to
"feedgnuplot --stream" to get a realtime visualization of the incoming
data.

This tool reads enough of the input file to get a legend, at which point
it constructs an awk program to do the main work, and execs to awk (it's
possible to use perl as well, but this isn't as fast).

Input/output data format The input/output data is vnlog: a plain-text table of values. Any lines beginning with "#" are treated as comments, and are passed through. The first line that begins with "#" but not "##" or "#!" is a legend line. After the "#", follow whitespace-separated field names. Each subsequent line is whitespace-separated values matching this legend. For instance, this is a valid vnlog file:

 #!/usr/bin/something
 ## more comments
 # x y z
 -0.016107 0.004362 0.005369
 -0.017449 0.006711 0.006711
 -0.018456 0.014093 0.006711
 -0.017449 0.018791 0.006376

"vnl-filter" uses this format for both the input and the output. The
comments are preserved, but the legend is updated to reflect the fields
in the output file.

A string "-" is used to indicate an undefined value, so this is also a
valid vnlog file:

 # x y z
 1 2 3
 4 - 6
 - - 7

Filtering To select specific columns, pass their names to the "-p" option (short for "--print" or "--pick", which are synonyms). In its simplest form, to grab only columns "x" and "y", do

 vnl-filter -p x,y

See the detailed description of "-p" below for more detail.

To select specific *rows*, we use *matches* expressions. Anything on the
"vnl-filter" commandline and not attached to any "--xxx" option is such
an expression. For instance

 vnl-filter 'size > 10'

would select only those rows whose "size" column contains a value > 10.
See the detailed description of matches expressions below for more
detail.

Context lines "vnl-filter" supports the context output options ("-A", "-B" and "-C") exactly like the "grep" tool. I.e to print out all rows whose "size" column contains a value > 10 but also include the 3 rows immediately before and after such matching rows, do this:

 vnl-filter -C3 'size > 10'

"-B" reports the rows *before* matching ones and "-A" the rows *after*
matching ones. "-C" reports both. Note that this applies *only* to
*matches* expressions: records skipped because they fail "--has" or
"--skipempty" are *not* included in contextual output.

Backend choice By default, the parsing of arguments and the legend happens in perl, which then constructs a simple awk script, and invokes "mawk" to actually read the data and to process it. This is done because awk is lighter weight and runs faster, which is important because our data sets could be quite large. We default to "mawk" specifically, since this is a simpler implementation than "gawk", and runs much faster. If for whatever reason we want to do everything with perl, this can be requested with the "--perl" option.

Special functions For convenience we support several special functions in any expression passed on to awk or perl (named expressions, matches expressions, "--eval" strings). These generally maintain some internal state, and vnl-filter makes sure that this state is consistent. Note that these are evaluated after "--skipcomments" and "--has". So any record skipped because of a "--has" expression, for instance, will not be considered in "prev()", "diff()" and so on.

*   rel(x) returns value of "x" relative to the first value of "x". For
    instance we might want to see the time or position relative to the
    start, not relative to some absolute beginning. Example:

     $ cat tst.vnl

     # time x
     100    200
     101    212
     102    209


     $ <tst.vnl vnl-filter -p 't=rel(time),x=rel(x)

     # t x
     0 0
     1 12
     2 9

*   diff(x) returns the difference between the current value of "x" and
    the previous value of "x". The first row will always be "-".
    Example:

     $ <tst.vnl vnl-filter -p x,'d1=diff(x),d2=diff(diff(x))' | vnl-align

     # x d1 d2
       1  -  -
       8  7  7
      27 19 12
      64 37 18
     125 61 24

*   sum(x) returns the cumulative sum of "x". As diff(x) can be thought
    of as a derivative, sum(x) can be thought of as an integral. So
    "diff(sum(x))" would return the same value as "x" (except for the
    first row; "diff()" always returns "-" for the first row).

    Example:

     $ <tst.vnl vnl-filter -p 'x,s=sum(x),ds=diff(sum(x))' | vnl-align

     # x  s   ds
       1   1   -
       8   9   8
      27  36  27
      64 100  64
     125 225 125

*   prev(x) returns the previous value of "x". One could construct
    "sum()" and "rel()" using this, if they weren't already available.

*   latestdefined(x) returns the most recent value of "x" that isn't
    "-". If "x" isn't "-", this simply returns "x".

ARGUMENTS Matches expressions Anything on the commandline not attached to any "--xxx" option is a matches expression. These are used to select particular records (rows) in a data file. For each row, we evaluate all the expressions. If all the expressions evaluate to true, that row is output. This expression is passed directly to the awk (or perl) backend.

Example: to select all rows that have valid data in column "a" *or*
column "b" *or* column "c" you can

 vnl-filter 'a != "-" || b != "-" || c != "-"'

or

 vnl-filter --perl 'defined a || defined b || defined c'

As with the named expressions given to "-p" (described above), these are
passed directly to awk, so anything that can be done with awk is
supported here.

-p|--print|--pick expr These option provide the mechanism to select specific columns for output. For instance to pull out columns called "lat", "lon", and any column whose name contains the string "feature_", do

 vnl-filter -p lat,lon,'feature_.*'

or, equivalently

 vnl-filter --print lat --print lon --print 'feature_.*'

We look for exact column name matches first, and if none are found, we
try a regex. If there was no column called exactly "feature_", then the
above would be equivalent to

 vnl-filter -p lat,lon,feature_

This mechanism is much more powerful than just selecting columns. First
off, we can rename chosen fields:

 vnl-filter -p w=feature_width

would pick the "feature_width" field, but the resulting column in the
output would be named "w". When renaming a column in this way regexen
are *not* supported, and exact field names must be given. But the string
to the right of the "=" is passed on directly to awk (after replacing
field names with column indices), so any awk expression can be used
here. For instance to compute the length of a vector in separate columns
"x", "y", and "z" you can do:

 vnl-filter -p 'l=sqrt(x*x + y*y + z*z)'

A single column called "l" would be produced.

We can also *exclude* columns by preceding their name with "!". This
works like you expect. Rules:

*   The pick/exclude directives are processed in order given to produce
    the output picked-column list

*   If the first "-p" item is an exclusion, we implicitly pick *all* the
    columns prior to processing the "-p".

*   The exclusion expressions match the *output* column names, not the
    *input* names.

*   We match the exact column names first. If that fails, we match as a
    regex

Example. To grab all the columns *except* the temperature(s) do this:

 vnl-filter -p !temperature

To grab all the columns that describe *something* about a robot (columns
whose names have the string "robot_" in them), but *not* its temperature
(i.e. *not* "robot_temperature"), do this:

 vnl-filter -p robot_,!temperature

--has a,b,c,... Used to select records (rows) that have a non-empty value in a particular field (column). A null value in a column is designated with a single "-". If we want to select only records that have a value in the "x" column, we pass "--has x". To select records that have data for all of a given set of columns, the "--has" option can be repeated, or these multiple columns can be given in a whitespace-less comma-separated list. For instance if we want only records that have data in both columns "x" and "y" we can pass in "--has x,y" or "--has x --has y". If we want to combine multiple columns in an or (select rows that have data in any of a given set of columns), use a matches expression, as documented below.

If we want to select a column *and* pick only rows that have a value in
this column, a shorthand syntax exists:

 vnl-filter --has col -p col

is equivalent to

 vnl-filter -p +col

Note that just like the column specifications in "-p" the columns given
to "--has" must match exactly *or* as a regex. In either case, a unique
matching column must be found.

-l|list-columns Instead of doing any processing, parse the input to get the available columns, print those out, and exit

-A N|--after-context N Output N lines following each matches expression, even those lines that do not themselves match. This works just like the "grep" options of the same name. See "Context lines"

-B N|--before-context N Output N lines preceding each matches expression, even those lines that do not themselves match. This works just like the "grep" options of the same name. See "Context lines"

-C N|--context N Output N lines preceding and following each matches expression, even those lines that do not themselves match. This works just like the "grep" options of the same name. See "Context lines"

--eval expr Instead of printing out all matching records and picked columns, just run the given chunk of awk (or perl). In this mode of operation, "vnl-filter" acts just like a glorified awk, that allows fields to be accessed by name instead of by number, as it would be in raw awk.

Since the expression may print *anything* or nothing at all, the output
in this mode is not necessarily itself a valid vnlog stream. And no
column-selecting arguments should be given, since they make no sense in
this mode.

In awk the expr is a full set of pattern/action statements. So to print
the sum of columns "a" and "b" in each row, and at the end, print the
sum of all values in the "a" column

 vnl-filter --eval '{print a+b; suma += a} END {print suma}'

In perl the arbitrary expression fits in like this:

 sub evalexpr
 {
   eval expression;     # evaluate the arbitrary expression
 }
 while(<>) # read each line
 {
   chomp;
   next unless matches; # skip non-matching lines
   evalexpr();
 }

--function|--sub Evaluates the given expression as a function that can be used in other expressions. This is most useful when you want to print something that can't trivially be written as a simple expression. For instance:

 $ cat tst.vnl
 # s
 1-2
 3-4
 5-6

 $ < tst.vnl
   vnl-filter --function 'before(x) { sub("-.*","",x); return x }' \
              --function 'after(x)  { sub(".*-","",x); return x }' \
              -p 'b=before(s),a=after(s)'
 # b a
 1 2
 3 4
 5 6

See the CAVEATS section below if you're doing something
sufficiently-complicated where you need this.

--function-abs|--sub-abs Convenience option to add an absolute-value "abs()" function. This is only useful for awk programs (the default, no "--perl" given) since perl already provides "abs()" by default.

--begin|--BEGIN Evaluates the given expression in the BEGIN {} block of the generated awk (or perl) program.

--end|--END Evaluates the given expression in the END {} block of the generated awk (or perl) program.

--[no]skipempty Do [not] skip records where all fields are blank. By default we do skip all empty records; to include them, pass "--noskipempty"

--skipcomments Don't output non-legend comments

--perl By default all procesing is performed by "mawk", but if for whatever reason we want perl instead, pass "--perl". Both modes work, but "mawk" is noticeably faster. "--perl" could be useful because it is more powerful, which could be important since a number of things pass commandline strings directly to the underlying language (named expressions, matches expressions, "--eval" strings). Note that while variables in perl use sigils, column references should not use sigils. To print the sum of all values in column "a" you'd do this in awk

 vnl-filter --eval '{suma += a} END {print suma}'

and this in perl

 vnl-filter --perl --eval '{$suma += a} END {say $suma}'

The perl strings are evaluated without "use strict" or "use warnings" so
I didn't have to declare $suma in the example.

With "--perl", empty strings ("-" in the vnlog file) are converted to
"undef".

--dumpexprs Used for debugging. This spits out all the final awk (or perl) program we run for the given commandline options and given input. This is the final program, with the column references resolved to numeric indices, so one can figure out what went wrong.

--unbuffered Flushes each line after each print. This makes sure each line is output as soon as it is available, which is crucial for realtime output and streaming plots.

--stream Synonym for "--unbuffered"

CAVEATS This tool is very lax in its input validation (on purpose). As a result, columns with names like %CPU and "TIME+" do work (i.e. you can more or less feed in output from "top -b"). The downside is that shooting yourself in the foot is possible. This tradeoff is currently tuned to be very permissive, which works well for my use cases. I'd be interested in hearing other people's experiences. Potential pitfalls/unexpected behaviors:

*   All column names are replaced in all eval strings without regard to
    context. The earlier example that reports the sum of values in a
    column: "vnl-filter --eval '{suma += a} END {print suma}'" will work
    fine if we *do* have a column named "a" and do *not* have a column
    named "suma". But will not do the right thing if any of those are
    violated. For instance, if a column "a" doesn't exist, then "awk"
    would see "suma += a" instead of something like "suma += $5". "a"
    would be an uninitialized variable, which evaluates to 0, so the
    full "vnl-filter" command would not fail, but would print 0 instead.
    It's the user's responsibility to make sure we're talking about the
    right columns. The focus here was one-liners so hopefully nobody has
    so many columns, they can't keep track of all of them in their head.
    I don't see any way to resolve this without seriously impacting the
    scope of the tool, so I'm leaving this alone.

*   It is natural to use vnlog as a database. You can run queries with
    something like

     vnl-filter 'key == 5'

    This works. But unlike a real database this is clearly a linear
    lookup. With large data files, this would be significantly slower
    than the logarithmic searches provided by a real database. The
    meaning of "large" and "significant" varies, and you should test it.
    In my experience vnlog "databases" scale surprisingly well. But at
    some point, importing your data to something like sqlite is well
    worth it.

*   When substituting column names I match *either* a word-nonword
    transition ("\b") *or* a whitespace-nonword transition. The word
    boundaries is what would be used 99% of the time. But the keys may
    have special characters in them, which don't work with "\b". This
    means that whitespace becomes important: "1+%CPU" will not be parsed
    as expected, which is correct since "+%CPU" is also a valid field
    name. But "1+ %CPU" will be parsed correctly, so if you have weird
    field names, put the whitespace into your expressions. It'll make
    them more readable anyway.

*   Strings passed to "-p" are split on "," *except* if the "," is
    inside balanced "()". This makes it possible to say things like
    "vnl-filter --function 'f(a,b) { ... }' -p 'c=f(a,b)'". This is
    probably the right behavior, although some questionable looking
    field names become potentially impossible: "f(a" and "b)" *could*
    otherwise be legal field names, but you're probably asking for
    trouble if you do that.

*   Currently there're two modes: a pick/print mode and an "--eval"
    mode. Then there's also "--function", which adds bits of "--eval" to
    the pick/print mode, but it feels maybe insufficient. I don't yet
    have strong feelings about what this should become. Comments welcome

#+END_EXAMPLE

** vnl-align #+BEGIN_EXAMPLE NAME vnl-align - aligns vnlog columns for easy interpretation by humans

SYNOPSIS $ cat tst.vnl

 # w x y z
 -10 40 asdf -
 -20 50 - 0.300000
 -30 10 whoa 0.500000


 $ vnl-align tst.vnl

 # w  x   y      z
 -10 40 asdf -
 -20 50 -    0.300000
 -30 10 whoa 0.500000

DESCRIPTION The basic usage is

 vnl-align logfile

The arguments are assumed to be the vnlog files. If no arguments are
given, the input comes from STDIN.

This is very similar to "column -t", but handles "#" lines properly:

1. The first "#" line is the legend. For the purposes of alignment, the
leading "#" character and the first column label are treated as one
column

2. All other "#" lines are output verbatim.

#+END_EXAMPLE

** vnl-sort #+BEGIN_EXAMPLE NAME vnl-sort - sorts an vnlog file, preserving the legend

SYNOPSIS $ cat a.vnl # a b AA 11 bb 12 CC 13 dd 14 dd 123

 Sort lexically by a:
 $ <a.vnl vnl-sort -k a
 # a b
 AA 11
 CC 13
 bb 12
 dd 123
 dd 14

 Sort lexically by a, ignoring case:
 $ <a.vnl vnl-sort -k a --ignore-case
 # a b
 AA 11
 bb 12
 CC 13
 dd 123
 dd 14

 Sort lexically by a, then numerically by b:
 $ <a.vnl vnl-sort -k a -k b.n
 # a b
 AA 11
 CC 13
 bb 12
 dd 14
 dd 123

 Sort lexically by a, then numerically by b in reverse:
 $ <a.vnl vnl-sort -k a -k b.nr
 # a b
 AA 11
 CC 13
 bb 12
 dd 123
 dd 14


 Sort by month and then day:
 $ cat dat.vnl
 # month day
 March 5
 Jan 2
 Feb 1
 March 30
 Jan 21

 $ <dat.vnl vnl-sort -k month.M -k day.n
 # month day
 Jan 2
 Jan 21
 Feb 1
 March 5
 March 30

DESCRIPTION Usage: vnl-sort [options] logfile logfile logfile ... < logfile

This tool sorts given vnlog files in various ways. "vnl-sort" is a
wrapper around the GNU coreutils "sort" tool. Since this is a wrapper,
most commandline options and behaviors of the "sort" tool are present;
consult the sort(1) manpage for detail. The differences from GNU
coreutils "sort" are

*   The input and output to this tool are vnlog files, complete with a
    legend

*   The columns are referenced by name, not index. So instead of saying

      sort -k1

    to sort by the first column, you say

      sort -k time

    to sort by column "time".

*   The fancy "KEYDEF" spec from "sort" is only partially supported. I
    only allow us to sort by full *fields*, so the start/stop positions
    don't make sense. I *do* support the "OPTS" to change the type of
    sorting in a given particular column. For instance, to sort by month
    and then by day, do this (see example above):

      vnl-sort -k month.M -k day.n

*   "--files0-from" is not supported due to lack of time. If somebody
    really needs it, talk to me.

*   "--output" is not supported due to an uninteresting technical
    limitation. The output always goes to standard out.

*   "--field-separator" is not supported because vnlog assumes
    whitespace-separated fields

*   "--zero-terminated" is not supported because vnlog assumes
    newline-separated records

*   By default we call the "sort" tool to do the actual work. If the
    underlying tool has a different name or lives in an odd path, this
    can be specified by passing "--vnl-tool TOOL"

Past that, everything "sort" does is supported, so see that man page for
detailed documentation. Note that all non-legend comments are stripped
out, since it's not obvious where they should end up.

COMPATIBILITY I use GNU/Linux-based systems exclusively, but everything has been tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and CentOS. I can imagine there's something I missed when testing on non-Linux systems, so please let me know if you find any issues.

vnlog
vnlog copied to clipboard

Metadata

a b c

comment

i j x

x y

comment

x y

x y

x y # 1+ -

x y z z

time x_self x_observation

time x_self

time x_self

time x_observation

x_self

time x_self

time x_observation

time x_self x_observation

name size

w x y z binary

← Metadata

Owner

Metadata

vnlog vnlog copied to clipboard

Metadata

a b c

comment

i j x

x y

comment

x y

x y

x y # 1+ -

x y z z

time x_self x_observation

time x_self

time x_self

time x_observation

x_self

time x_self

time x_observation

time x_self x_observation

name size

w x y z binary

← Metadata

Owner

Metadata

vnlog
vnlog copied to clipboard