dazl
dazl copied to clipboard
Configurable abstraction layer for Go logging frameworks
dazl
Configurable abstraction layer for Go logging frameworks
Dazl is not just another Go logging framework. We're not here to reinvent Go logging for the nth time. Dazl is an abstraction layer that provides a unified interface and configuration format on top of existing logging frameworks, using a proven approach adapted from popular frameworks in other languages like slf4j.
Dazl is a pluggable logging abstraction with support for multiple existing backend frameworks:
Dazl loggers add numerous features on top of existing frameworks:
- Decouples Go libraries from specific logging implementations
- Makes logging configurable via YAML configuration files
- Structured logging with support for JSON or console encoding and user-defined fields
- Hierarchical loggers, inheritance, sampling and other advanced features
- Supports runtime configuration changes for easy debugging
User Guide
-
Use cases
- Logging in Go libraries
- Configuration of Go services
-
Getting started
- Add the dazl dependency
- Import a logging backend
- Configure logging
- Run the application
-
Go API
- Initializing the framework
- Working with loggers
- Log levels
- Structured logging
-
Configuration files
-
Encoders
- JSON
- Console
- Writers
- Loggers
- Reference
-
Encoders
-
Runtime configuration changes
- Changing the log level
-
Custom logging frameworks
- Custom encoders
- Custom writers
Use cases
There are numerous benefits to using a logging abstraction, but dazl is designed to serve a couple of important use cases for two types of applications in particular.
Logging in Go libraries
Go libraries designed to be imported and used by other Go modules can use dazl to avoid adding a dependency on a specific logging framework, tying their users to the same framework. Additionally, dazl enables your users to configure loggers and log formats indepenedently, with no added work for you. The users of your library ought to be able to select their own logging framework and configure the format and severity of log outputs. Simply add a dependency on the dazl logger, and leave it up to your users to import whichever logging backend they desire.
Configuration of Go application logging
One of the most common use cases for Go applications is in cloud applications deployed in containers and on platforms like Kubernetes. Most Go logging frameworks provide programmatic APIs for configuring loggers, levels, formats, and other logging options. Using dazl in Go-based services enables your users to configure logging independent of the code, eliminating the need to recompile code to modify the verbosity or format of application logs.
Getting started
Add the dazl dependencies
To start using dazl, first add the framework to your go.mod
:
go get -u github.com/atomix/dazl
Additionally, add the dependency for one of the logging backendsL
go get -u github.com/atomix/dazl/zap
Import a logging backend
Import the logging backend in your module main
:
main.go
package main
import (
"github.com/atomix/dazl"
_ "github.com/atomix/dazl/zap"
)
var log = dazl.GetLogger()
func main() {
log.Info("Hello world!")
}
To avoid adding unnecessary dependencies on logging frameworks, logging backends should only be imported within
your module main
.
Configure logging
Create a logging.yaml
file to configure the logging framework:
logging.yaml
encoders:
json:
fields:
- message
- level:
format: uppercase
- caller:
format: short
- timestamp:
format: iso8601
writers:
stdout:
encoder: json
rootLogger:
level: info
outputs:
- stdout
To print logs to an output, the logging.yaml
configuration must configure one or more writers
and configure the
rootLogger
or child loggers
with outputs
to output messages from those loggers to the configured writer(s).
Run the application
Finally, when you run the application main
from the same directory as your logging.yaml
file, you should see
the logs printed to stdout:
go run ./main.go
For more information check out the example module.
Go API
Initializing the logging framework
The dazl logger does not log to any output unless a logging framework is imported. To maintain independence from any
particular logging backend, applications should only import a specific logging framework from within a main
file.
Libraries designed to be imported by other projects should never import a logging backend themselves. Instead, leave
the specific logging framework implementation up to your users.
The logging backend is configured by importing the framework into your application's main
package.
Logging with zap
To configure dazl to use the zap logging backend, add the zap
framework
to your module's go.mod
:
go get -u github.com/atomix/dazl/zap
Then import the github.com/atomix/dazl/zap
framework implementation in your main
package:
package main
import _ "github.com/atomix/dazl/zap"
func main() {
...
}
Logging with zerolog
To configure dazl to use the zerolog logging backend, add the zerolog
framework
to your module's go.mod
:
go get -u github.com/atomix/dazl/zerolog
Then import the github.com/atomix/dazl/zerolog
framework implementation in your main
package:
package main
import _ "github.com/atomix/dazl/zerolog"
func main() {
...
}
Loggers
The typical usage of the framework is to create a Logger
once at the top of each Go package:
var log = dazl.GetLogger()
Package loggers are assigned the name of the package calling GetLogger()
or GetPackageLogger()
. So, if you
call dazl.GetLogger()
from the github.com/atomix/atomix/runtime
package, the logger will be assigned the
name github.com/atomix/atomix/runtime
. The naming strategy becomes important for
logger configuration and, in particular, inheritance.
const author = "kuujo"
var log = dazl.GetLogger()
func main() {
log.Infof("The author of dazl is %s", author)
}
2023-03-31T01:13:30.607Z INFO main.go:12 The author of dazl is kuujo
Alternatively, custom loggers can be retrieved via GetLogger
:
var log = dazl.GetLogger("foo/bar")
Logger names must be formatted in path format, with each element separated by a /
. This format is used
to establish a hierarchy for inheritence of logger configurations.
All loggers are descendants of the root logger:
var log = dazl.GetRootLogger()
Log levels
Dazl supports a fairly standard set of log levels for loggers:
-
debug
-
info
-
warn
-
error
-
fatal
-
panic
The levels for each logger can be configured individually via their configuration:
loggers:
github.com/atomix/atomix/runtime:
level: warn
The Logger
interface exposes methods for simple logging, formatted logging, and structured logging with typed
fields for each log level:
-
Debug(args ...any)
-
Debugf(msg string, args ...any)
-
Debugw(msg string, fields ...Field)
-
Info(args ...any)
-
Infof(msg string, args ...any)
-
Infow(msg string, fields ...Field)
-
Warn(args ...any)
-
Warnf(msg string, args ...any)
-
Warnw(msg string, fields ...Field)
- ...
Messages will only be written to log outputs if the configured level of the logger is higher than the message level.
Structured logging
Structured logging is supported for the JSON encoding, and JSON fields are configurable via
the Logger
API.
The simplest way to add fields to your structured logs is to call one of the *w
methods on the Logger
interface.
These methods accept an arbitrary number of ...Field
s to write to the logs. Fields are typed and named values
that can be constructed via functions in the dazl
package:
log.Warnw("Something went wrong!",
dazl.String("user", user.Name),
dazl.Uint64("user-id", user.ID))
Alternatively, you can create a structured logger with a fixed set of fields using the WithFields
method:
var log = dazl.GetLogger().WithFields(
dazl.String("user", user.Name),
dazl.Uint64("id", user.ID))
log.Warn("Something went wrong!")
When the logger is output to a JSON encoded writer, the above code will log the fields as part of the JSON object:
{"timestamp":"2023-04-07T19:24:09-07:00","logger":"2/4","message":"Something went wrong!","user":"Jordan Halterman","id":5678}
Configuration files
Loggers can be configured via a YAML configuration file. The configuration files may be in one of many locations on the file system:
-
logging.yaml
-
~/logging.yaml
-
/etc/dazl/logging.yaml
The configuration file contains a set of loggers
which specifies the level and outputs of each logger,
writers
which specify where to write log messages, and encoders
defining how to encode log messages.
Configuring encoders
The encoders
section of the configuration defines how dazl encodes log messages. Dazl supports two
encodings: json
and console
. Each encoder defines the set of fields
to output and optionally the
format of the fields.
encoders:
json:
fields:
...
console:
fields:
...
Encoders are referenced by writers
and used to encode messages.
Note that support for some encoding options such as renaming keys or formatting fields depends on whether those features are supported the underlying logging framework. If some requested features are not supported by the imported logging framework, dazl may panic at startup.
JSON encoder
The json
encoder configuration defines the set of fields to include in all JSON formatted messages. Each
JSON field also supports an additional key
to override the default JSON key for that field:
encoders:
json:
# A list of fields to include
fields:
# The log message
- message:
# The JSON key for the field
key: msg
# The logger name
- name:
# The JSON key for the field
key: logger
# The log level
- level:
# The JSON key for the field
key: level
# The level format: 'uppercase' or 'lowercase'
format: uppercase
# The time at which the message was logged
- timestamp:
# The JSON key for the field
key: time
# The time format: 'iso8601' or 'unix'
format: iso8601
# The line of code at which the message was logged
- caller:
# The JSON key for the field
key: caller
# The caller format: 'short' or 'long'
format: short
# The log stacktrace
- stacktrace:
# The JSON key for the field
key: trace
Console encoder
The console
encoder configuration defines the colums to encode with each log message:
encoders:
console:
# A list of fields to include
fields:
# The log message
- message
# The logger name
- name
# The log level
- level:
# The level format: 'uppercase' or 'lowercase'
format: uppercase
# The time at which the message was logged
- timestamp:
# The time format: 'iso8601' or 'unix'
format: iso8601
# The line of code at which the message was logged
- caller:
# The caller format: 'short' or 'long'
format: short
# The log stacktrace
- stacktrace
Encoder fields
Both the json
and console
encoders support the following set of fields:
-
message
-
name
-
level
-
timestamp
-
caller
-
stacktrace
If any field is excluded from the encoder's field list, dazl will attempt to exclude that field from the log output, but some logging backends may not support this ability and therefore may include those fields in their output.
Level formats
The format of the log level can be configured for each encoder via the format
key:
encoders:
json:
level:
format: uppercase
Defined level formats include:
-
uppercase
- upper case level name -
lowercase
- lower case level name
Note that support for level formats depends on support from the imported logging backend. Dazl may panic at startup if the underlying logging framework does not support the configured level format.
Timestamp formats
The format of the timestamp can be configured for each encoder via the format
key:
encoders:
json:
timestamp:
format: ISO8601
Defined timestamp formats include:
-
ISO8601
-
unix
Note that support for timestamp formats depends on support from the imported logging backend. Dazl may panic at startup if the underlying logging framework does not support the configured timestamp format.
Caller formats
The format of the caller can be configured for each encoder via the format
key:
encoders:
json:
caller:
format: uppercase
Defined caller formats include:
-
short
-
full
Note that support for caller formats depends on support from the imported logging backend. Dazl may panic at startup if the underlying logging framework does not support the configured level format.
Configuring writers
# A set of named writers for loggers to write to.
# All writers must specify an 'encoder' to use
writers:
# The stdout writer
stdout:
# The name of the encoder to use for the writer
encoder: console
# The stderr writer
stderr:
# The name of the encoder to use for the writer
encoder: console
# Remaining writers are files
file:
# The path to the file
path: ./example.log
# The name of the encoder to use for the writer
encoder: json
Configuring loggers
The root logger
The default logging configuration is configured via the rootLogger
key:
rootLogger:
level: info
All loggers inherit their default configuration from the root logger. The root logger configuration should at
least specify a minimum log level
for all loggers, and at least one outputs
to a writer.
rootLogger:
level: info
outputs:
- stdout
Named loggers
Once the rootLogger
has been defined, all other loggers that are descendants of the root logger may be defined and
configured in the loggers
section of the configuration file:
loggers:
github.com/atomix/atomix/runtime:
level: info
github.com/atomix/atomix/sidecar:
level: warn
Outputs
Once you've defined the set of writers to which to write your application logs, named loggers can be directed to
those writers via their configured outputs
:
rootLogger:
level: info
outputs:
- stdout
Each output must specify a writer
to write to. As with log levels, loggers inherit the outputs of their ancestors, so
with an output to the stdout
writer in the root logger configuration, all loggers will have an output to stdout
.
Output levels
In some cases, you may want to restrict the verbosity of logs to one output without restricting the verbosity of all
messages for a logger. For example, you may want to write info
and higher messages to the console for
human readability, and debug
and higher messages to a file for later debugging. Each output supports a level
that can be used to filter the logs to that output:
loggers:
github.com/atomix/atomix:
# Pass through all messages over 'debug' level to the outputs
level: debug
outputs:
# Limit the messages from this logger to stdout to 'info' level
- stdout:
level: info
# Pass through all messages to the 'file' writer
- file
writers:
stdout:
encoder: console
file:
path: ./app.log
encoder: json
Output overrides
Descendants may override their ancestor loggers' output configurations. This can be done by simply specifying the
same output name. For example, to override the log level for the rootLogger
's stdout
output:
# Configure the stdout writer to use console encoding
writers:
stdout:
encoder: console
rootLogger:
level: info
outputs:
# Output the root logger to stdout
- stdout
loggers:
github.com/atomix/atomix/runtime:
outputs:
# Limit outputs from this logger to stdout to minimum of 'warn' level
stdout:
level: warn
Sampling
Samplers can be added to either loggers to reduce the number of messages logged:
loggers:
github.com/atomix/atomix/runtime:
sample:
basic:
interval: 10
Samplers can also be added to individual outputs to limit only the messages to that particular output:
loggers:
github.com/atomix/atomix/runtime:
outputs:
stdout:
sample:
basic:
interval: 10
Basic sampler
The basic
sampler logs every nth message below maxLevel
according to the configured interval
:
loggers:
github.com/atomix/atomix/runtime:
sample:
basic:
interval: 10
maxLevel: debug
Random sampler
The random
sampler randomly logs messages below maxLevel
by choosing a random integer between 0
and interval
:
loggers:
github.com/atomix/atomix/runtime:
sample:
random:
interval: 10
maxLevel: debug
Inheritance
The path-like format used for logger names is used to establish a hierarchy of loggers. The dazl configuration enables developers and their users to configure individual loggers at runtime. Log levels are inherited by descendants of a logger. This enables users to easily enable logging for specific Go packages, their subpackages, or entire Go modules with a single configuration change:
# Enable debug logging for all Atomix code
loggers:
github.com/atomix:
level: debug
You can set the dazl.Level
for a logger at startup time via configuration files or at runtime
via the Logger
API to control the granularity of a logger's output:
dazl.GetLogger("github.com/atomix").SetLevel(dazl.DebugLevel)
The root logger is the ancestor of all other loggers and can be configured via GetRootLogger
:
dazl.GetRootLogger().SetLevel(dazl.InfoLevel)
If the level for a logger is not explicitly set, it will inherit its level from its nearest ancestor in
the logger hierarchy. For example, setting the github.com/atomix/atomix/runtime
logger to the debug
level will change the loggers for all loggers in the github.com/atomix/atomix/runtime/...
packages
to the debug
level.
Reference
A reference configuration file detailing and documenting all the available configuration
options in logging.yaml
is available in the examples directory of this repo.
Runtime configuration changes
Changing the log level
You can set the dazl.Level
for a logger at startup time via configuration files or at runtime
via the Logger
API to control the granularity of a logger's output:
dazl.GetLogger("github.com/atomix").SetLevel(dazl.DebugLevel)
The root logger is the ancestor of all other loggers and can be configured via GetRootLogger
:
dazl.GetRootLogger().SetLevel(dazl.InfoLevel)
Custom logging frameworks
Dazl provides several existing implementations of logging frameworks:
- zap
- zerolog
Logging frameworks are implemented by implementing the Framework
interface:
const name = "example"
type Framework struct{}
func (f Framework) Name() string {
return name
}
Frameworks should register when imported:
func init() {
dazl.Register(&Framework{})
}
Encoding
Framework
implementations should implement one or more of the *EncodingFramework
interfaces to indicate support
for encoding formats. Frameworks implement these interfaces to provide Encoder
s which are used by dazl to create new
Writer
s:
type JSONEncoder struct{}
func (e *JSONEncoder) NewWriter(writer io.Writer) (dazl.Writer, error) {
...
}
JSON encoding
To implement support for JSON encoding, implement the JSONEncodingFramework
interface:
func (f Framework) JSONEncoder() Encoder {
return &JSONEncoder{}
}
Console encoding
To implement support for JSON encoding, implement the ConsoleEncodingFramework
interface:
func (f Framework) ConsoleEncoder() Encoder {
return &ConsoleEncoder{}
}
Encoding options
Encoder
implementations may support configuration options by implementing optional interfaces with the
following methods:
-
WithMessageKey(key string) (dazl.Encoder, error)
-
WithNameEnabled() (dazl.Encoder, error)
-
WithNameKey(key string) (dazl.Encoder, error)
-
WithLevelEnabled() (dazl.Encoder, error)
-
WithLevelKey(key string) (dazl.Encoder, error)
-
WithLevelFormat(format dazl.LevelFormat) (dazl.Encoder, error)
-
WithTimestampEnabled() (dazl.Encoder, error)
-
WithTimestampKey(key string) (dazl.Encoder, error)
-
WithTimestampFormat(format dazl.TimestampFormat) (dazl.Encoder, error)
-
WithCallerEnabled() (dazl.Encoder, error)
-
WithCallerKey(key string) (dazl.Encoder, error)
-
WithCallerFormat(format dazl.CallerFormat) (dazl.Encoder, error)
-
WithStacktraceEnabled() (dazl.Encoder, error)
-
WithStacktraceKey(key string) (dazl.Encoder, error)
Log writers
Encoder
s create Writer
s for use by the dazl Logger
. All Writer
s must implement the following methods:
-
WithName(name string) dazl.Writer
-
WithSkipCalls(calls int) dazl.Writer
-
Debug(msg string)
-
Info(msg string)
-
Warn(msg string)
-
Error(msg string)
-
Panic(msg string)
-
Fatal(msg string)
Writer options
Writer
implementations may optionally support additional features by implementing optional interfaces
by adding the following methods:
-
WithStringField(name string, value string) dazl.Writer
-
WithIntField(name string, value int) dazl.Writer
-
WithInt32Field(name string, value int32) dazl.Writer
-
WithInt64Field(name string, value int64) dazl.Writer
-
WithUintField(name string, value uint) dazl.Writer
-
WithUint32Field(name string, value uint32) dazl.Writer
-
WithUint64Field(name string, value uint64) dazl.Writer
-
WithFloat32Field(name string, value float32) dazl.Writer
-
WithFloat64Field(name string, value float64) dazl.Writer
-
WithBoolField(name string, value bool) dazl.Writer
-
WithStringSliceField(name string, values []string) dazl.Writer
-
WithIntSliceField(name string, values []int) dazl.Writer
-
WithInt32SliceField(name string, values []int32) dazl.Writer
-
WithInt64SliceField(name string, values []int64) dazl.Writer
-
WithUintSliceField(name string, values []uint) dazl.Writer
-
WithUint32SliceField(name string, values []uint32) dazl.Writer
-
WithUint64SliceField(name string, values []uint64) dazl.Writer
-
WithFloat32SliceField(name string, values []float32) dazl.Writer
-
WithFloat64SliceField(name string, values []float64) dazl.Writer
-
WithBoolSliceField(name string, values []bool) dazl.Writer
-
WithErrorField(name string, err error) dazl.Writer