structured-logging-schema

structured-logging-schema copied to clipboard

Hey there,

I don't think that using JSON objects is the best choice for structured logging at all. In my opinion, JSON arrays are more suited:

• They are easily parsed by JSON.parse and all major JSON toold as well
• They are faster to parse
• They are faster to stringify
• They are more lightweight

I agree that the payload should be any object the user provides, but using objects for the root log record does not sound well for me.

Yes, the negative is that arrays are not as human readable as objects are, but with a first field specifying the schema version, it's just as machine readable.

And command-line filters using JSON agnostic tools (e.g. jq) should be just as easy as for objects.

Thus, if a unified standard would be crafted, I highly prefer arrays over objects; as I do use them within my logging library oddlog. And yes, I'm not sure how to include it within the readme because of the array-based log record format.

But since it's not used by any other library that I know of, I highly doubt that this would succeed :crying_cat_face:

I would propose something along this lines (... means additional fields tolerated):

RECORD:   [SCHEMA, PROCESS, LOGGER, LOG]<EOL>

SCHEMA:   [version : number, flags (optional) : string, ...]

flags that come to my mind:

• S - timestamp in seconds instead of milliseconds
• R - timestamp is not absolute (e.g. relative to application start, performance.now(), etc.)
• C - timestamp has custom format

PROCESS:  null | object | [process_id : number, host : string, platform : string, ...]

LOGGER:   null | [name : string, ...]

LOG:      [timestamp : number, severity : number, message : string, PAYLOAD, SOURCE (optional), ...]
PAYLOAD:  null | object
SOURCE:   [file : string, row : number, col : number]

Examples would look like this:

[[1],[345,"zerg","Linux 4.17.11"],["my-app:http"],[1459529098960,4,"hello world",{"lorem": "ipsum"},["lib/index.js",82,12]]]
[[1],[345,"zerg","Linux 4.17.11"],["my-app:http"],[1459529098960,4,"hello world",{"lorem": "ipsum"}]]

This schema would have the additional benefit of clear separation of the information sources (schema, process, logger, log). Even so the records are meant to be read by machines, they are easy to investigate by humans as well.