structured-logging-schema
structured-logging-schema copied to clipboard
Mission: Agree on a common format for application structured logs
Schema Comparison between Structured Logging Libraries
The goal of this document is to summarize the differences between various structured logging libraries, in order to encourage a discussion so that a consensus can be reached for a common format.
Please use the GitHub issues for discussion, and submit GitHub Pull requests to add/fix the information in the tables.
Message and Payload
Behold the following logging command that you might find in an application. There is a "message" string, and a "payload" object:
logger.info("Hello World", {"animal": "cat", "numLegs": 4})
^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Message Payload object
Some libraries merge the payload object with the root object, so the event will look like this:
{
"timestamp": "2018-06-18T23:16:45.000Z",
"message": "Hello World",
"animal": "cat",
"numLegs": 4
}
While other libraries nest the payload object beneath some key, with a result like this:
{
"timestamp": "2018-06-18T23:16:45.000Z",
"message": "Hello World",
"data": {
"animal": "cat",
"numLegs": 4
}
}
Library | Message JSON Key | Payload Object JSON Key |
---|---|---|
bunyan (JavaScript) | msg |
Merged with root |
katip (Haskell) | msg |
data |
logrus (Go) | msg |
Merged with root |
logstash | message |
Merged with root |
ougai (Ruby) | msg |
Merged with root |
pygogo (Python) | message |
Merged with root |
roarr (JavaScript) | message |
context |
semantic logger (Ruby) | message |
payload |
serilog (C#) | @m / @mt |
Merged with root |
slog (Rust) | msg |
Merged with root |
structlog (Python) | msg |
HELP NEEDED |
Timestamp JSON Key and format
All libraries automatically add a timestamp to all events.
Library | Timestamp JSON Key | Timestamp format |
---|---|---|
bunyan (JavaScript) | time |
ISO 8601 |
katip (Haskell) | at |
ISO 8601 |
logrus (Go) | time |
ISO 8601 |
logstash | timestamp / @timestamp |
ISO 8601 / varies |
ougai (Ruby) | time |
ISO 8601 |
pygogo (Python) | time |
ISO 8601 |
roarr (JavaScript) | time |
Epoch milliseconds |
semantic logger (Ruby) | timestamp |
ISO 8601 |
serilog (C#) | Timestamp / @t |
ISO 8601 |
slog (Rust) | ts |
ISO 8601 |
structlog (Python) | timestamp |
ISO 8601 |
Log Level
All libraries require that that all events are tagged with a level, chosen from a pre-defined list of available levels.
Log Level JSON Key
Library | Log Level JSON Key |
---|---|
bunyan (JavaScript) | level |
katip (Haskell) | sev |
logrus (Go) | level |
logstash | HELP NEEDED |
ougai (Ruby) | level |
pygogo (Python) | level |
roarr (JavaScript) | context.logLevel |
semantic logger (Ruby) | level |
serilog (C#) | Level / @l |
slog (Rust) | level |
structlog (Python) | level |
Available Log Levels
Library | VERBOSE | TRACE | DEBUG | INFO | NOTICE | WARNING | ERROR | CRITICAL | ALERT | FATAL | PANIC | EMERGENCY |
---|---|---|---|---|---|---|---|---|---|---|---|---|
bunyan (JavaScript) | N/A | 10 |
20 |
30 |
N/A | 40 |
50 |
N/A | N/A | 60 |
N/A | N/A |
katip (Haskell) | N/A | N/A | Debug |
Info |
Notice |
Warning |
Error |
Critical |
Alert |
N/A | N/A | Emergency |
logrus (Go) | N/A | N/A | debug |
info |
N/A | warning |
error |
N/A | N/A | fatal |
panic |
N/A |
logstash | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED |
ougai (Ruby) | N/A | 10 |
20 |
30 |
N/A | 40 |
50 |
N/A | N/A | 60 |
N/A | N/A |
pygogo (Python) | N/A | N/A | DEBUG |
INFO |
N/A | WARNING |
ERROR |
CRITICAL |
N/A | N/A | N/A | N/A |
roarr (JavaScript) | N/A | 10 |
20 |
30 |
N/A | 40 |
50 |
N/A | N/A | 60 |
N/A | N/A |
semantic logger (Ruby) | N/A | trace |
debug |
info |
N/A | warn |
error |
N/A | N/A | fatal |
N/A | N/A |
serilog (C#) | Verbose |
N/A | Debug |
Information |
N/A | Warning |
Error |
N/A | N/A | Fatal |
N/A | N/A |
slog (Rust) | N/A | TRACE |
DEBUG |
INFO |
N/A | WARN |
ERROR |
CRITICAL |
N/A | N/A | N/A | N/A |
structlog (Python) | N/A | N/A | debug |
info |
N/A | warning |
error |
critical |
N/A | N/A | N/A | N/A |
System Context Data
Some libraries automatically add additional fields to all events (such as process ID and thread ID) and/or allow you to configure a few pre-determined global fields suchs the current environment and application.
Library | Environment | Application | Namespace | Package | Machine Hostname | Process ID | Thread ID |
---|---|---|---|---|---|---|---|
bunyan (JavaScript) | N/A | N/A | N/A | N/A | hostname |
pid |
N/A |
katip (Haskell) | env |
app (Array of strings) |
ns (Array of strings) |
N/A | host |
pid |
thread |
logrus (Go) | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
logstash | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED | HELP NEEDED |
ougai (Ruby) | N/A | app |
N/A | N/A | hostname |
pid |
N/A |
pygogo (Python) | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
roarr (JavaScript) | N/A | context.application |
context.namespace |
context.package |
context.hostname |
N/A | N/A |
semantic logger (Ruby) | environment |
application |
N/A | N/A | host |
pid |
thread |
serilog (C#) | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
slog (Rust) | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
structlog (Python) | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
HTTP Request Context Data
Logging of HTTP requests is very common. Let's figure out a common format for HTTP method, host, url, headers, remote IP, remote IP geo-lookup, response status code, response size, etc...
!!! TODO !!!