Feature request: machine friendly outputs

[mateuszkwiatkowski]

As a user I'd like to be able to easily parse outputs of vm. It can be achieved using xo(1) tool.

[mateuszkwiatkowski]

@churchers would you accept a PR with this feature implemented?

[churchers]

I'd definitely be interested in seeing a PR for it

My only worry is adding large amounts of code to support a fairly non-essential feature. One of the downsides of sh is that many things that would be straight forward in a programming language end up being highly convoluted.

I did have a quick look at xo(1) the other day and it seemed like being able to generate meaningful semantic output for all the main list/info commands would require quite a lot of hacking. Especially to include the nice features like multiple output formats.

[churchers]

I've had a bit of a look at writing wrapper functions for the list commands, included below. If XO_OUTPUT is empty it just outputs a table via column as normal.

Seems to work well with XML output. The JSON looks good at first glance but contains an object/array with multiple elements called machine. I can't see an obvious way of getting it to create an actual array and I haven't bothered trying to parse it yet to see if it's usable. Think it's a good start though to support existing & xo output without making a mess of the existing code.

(Edit: in fact there's a few JSON errors which is a shame as that seems the most useful format. Needs surrounding in {} for a start which is easy, but also is missing a comma between the machines and errors on duplicate object keys as expected)

Probably could be cleaned up and organised a bit more. (For a start I've just realised I could drop XO_OUTPUT and just use XO_FORMAT)

#!/bin/sh

XO_OUTPUT="1"
XO_FORMAT="J"

table::start(){
    TABLE_NAME="$1"
    TABLE_DEPTH="$2"
    TABLE_WRAP="$3"

    # create opening tags
    if [ -n "${XO_OUTPUT}" -a -n "${TABLE_NAME}" ]; then
        xo -p"${XO_FORMAT}" --open "${TABLE_NAME}"
    fi
}

table::fields(){
    local _field="$1"
    local _line

    while [ -n "${_field}" ]; do
        if [ -n "${XO_OUTPUT}" ]; then
            # create an xo format line
            _line="${_line}${_line:+,}\{:${_field}\}"
        else
            # just add headings to top of output
            _line="${_line}${_line:+^}${_field}"
        fi

        shift
        _field="$1"
    done

    if [ -n "${XO_OUTPUT}" ]; then
        # use this as xo format spec
        XO_SPEC="${_line}"
    else
        _line=$(echo "${_line}" | tr '[:lower:]' '[:upper:]')
        OUTPUT="${_line}\n"
    fi
}

table::data(){
    local _data="$1"
    local _line _xo_opts

    # machine output?
    if [ -n "${XO_OUTPUT}" ]; then

        _xo_opts="-p${XO_FORMAT}"
        [ -n "${TABLE_DEPTH}" ] && _xo_opts="${_xo_opts} --depth ${TABLE_DEPTH}"
        [ -n "${TABLE_WRAP}" ] && _xo_opts="${_xo_opts} --wrap ${TABLE_WRAP}"

        xo ${_xo_opts} "${XO_SPEC}\n" "$@"
        return 0
    fi

    # not xo, just add to OUTPUT
    while [ -n "${_data}" ]; do
        _line="${_line}${_line:+^}${_data}"

        shift
        _data="$1"
    done

    OUTPUT="${OUTPUT}${_line}\n"
}

table::end(){
    if [ -n "${XO_OUTPUT}" ]; then
        # close tags
        [ -n "${TABLE_NAME}" ] && xo -p"${XO_FORMAT}" --close "${TABLE_NAME}"
    else
        # dump all the content of OUTPUT via column to sort out alignment
        printf "${OUTPUT}" | column -ts^
    fi
}

table::start "bhyve/machines" 2 "machine"
table::fields "name" "datastore" "loader" "cpu" "memory" "vnc" "autostart" "state"
table::data "bsd1" "default" "bhyveload" "4" "512M" "-" "No" "Stopped"
table::data "bsd2" "default" "bhyveload" "4" "512M" "-" "No" "Stopped"
table::end 

[churchers]

I've had more of a look at this and do have a working example to create reasonable xml/json output for the list command. You do have to jump through a lot of hoops to get valid JSON though as xo doesn't expose the functions needed to create arrays/lists. The libxo guys have actually created a few issues related to it on the main project - https://github.com/juniper/libxo/issues

[mateuszkwiatkowski]

Thank you for investing your time in this! I really appreciate this. I thought that this tool would do the job because it's widely used in FreeBSD's base. Let's see how libxo guys will respond. Or maybe there's some better way to do that.

[mateuszkwiatkowski]

@churchers we have kind of workaround for this: https://gitlab.com/runhyve/vm-webhooks I consider this temporary solution for our problem.

[mateuszkwiatkowski]

@churchers I submitted a PoC with support for JSON lists: https://github.com/churchers/vm-bhyve/pull/402 It uses JSON support in column(1) from... util-linux. :-) Let me know what you think about this direction.

Example output:

$ sudo vm list -j | jq .
{
  "machines": [
    {
      "name": "1_freebsd",
      "datastore": "default",
      "loader": "bhyveload",
      "cpu": "2",
      "memory": "2048M",
      "vnc": "-",
      "auto": "Yes [1]",
      "state": "Running (29311)"
    },
    {
      "name": "1_test",
      "datastore": "default",
      "loader": "bhyveload",
      "cpu": "4",
      "memory": "2048M",
      "vnc": "-",
      "auto": "No",
      "state": "Stopped"
    },
    {
      "name": "1_test-099",
      "datastore": "default",
      "loader": "grub",
      "cpu": "4",
      "memory": "4096M",
      "vnc": "-",
      "auto": "No",
      "state": "Stopped"
    }
  ]
}

[elasmo]

@churchers I submitted a PoC with support for JSON lists: #402 It uses JSON support in column(1) from... util-linux. :-) Let me know what you think about this direction. ...

Nice work! Why did you choose to rely on a third-party tool to achieve this? Not particularly involved with the inner workings and goals of vm-bhyve, but I'm a daily user. Producing json output would really help when integrating with other tools. XO(1) seems to me like a better option as it's part of FreeBSD base. It seems that the issues @churchers were referring to above now is closed.

[mateuszkwiatkowski]

Hi @elasmo,

It took me less than 30 minutes to achieve my goal with column from util-linux. I don't think I could do the same with xo in such short time but I agree, implementation using tools available in base system would be better.