JSON-SEQ is painful

[marten-seemann]

When we switched to JSON-SEQ a while back (https://github.com/quicwg/qlog/issues/172), we assumed that using record separators instead of newline-delimited JSON wouldn't cause too much pain. I think it's time to reevaluate this assumption.

vim

It also turns out that existing software is not that great at handling record record separators. Here's a screenshot of vim messing up the line character count (note that last line, where it usually tells you the cursor position, and it now displays 12-13).

jq

jq is actually the tool that's the most compatible with JSONSEQ. However, one MUST use the --seq flag, using standard jq results in the following error, which is not very helpful for the user:

jq: parse error: Invalid numeric literal at line 1, column 2

I had to use a search engine to figure out how to convince jq to use record separators. Definitely not the end of the world, but a subpar developer experience nonetheless.

grep

Another example is grep. Try to extract the transport:connection_started event from this qlog file using grep:

{
    "qlog_format":"JSON-SEQ",
    "qlog_version":"draft-03",
    "title":"quic-go qlog",
    "configuration":{"code_version":"(devel)"},
    "trace":{
        "vantage_point":{"type":"client"},
        "common_fields":{"ODCID":"65b480747721258697d6","group_id":"65b480747721258697d6","reference_time":1722494434168.3032,"time_format":"relative"},
    },
}
{
    "time":0.038458,
    "name":"transport:connection_started",
    "data":{"ip_version":"ipv6","src_ip":"::","src_port":55464,"dst_ip":"127.0.0.1","dst_port":6121,"src_cid":"36ff0294","dst_cid":"65b480747721258697d6"},
}

First of all, note how GitHub renders the record separator here: � (I love seeing this character on my screen!).

But back to grep: I haven't figured out a way to grep for the transport:connection_started event, and apparently it's not easy to change the record separator.

greping is very important when debugging, I often find myself filtering a qlog file for the event type(s) that I'm interested in.