slog-sampling
slog-sampling copied to clipboard
🚨 slog sampling: drop repetitive log records
Slog sampling policy
A middleware that samples incoming records which caps the CPU and I/O load of logging while attempting to preserve a representative subset of your logs.
Sampling fixes throughput by dropping repetitive log entries.
See also:
-
slog-multi:
slog.Handler
chaining, fanout, routing, failover, load balancing... -
slog-formatter:
slog
attribute formatting -
slog-sampling:
slog
sampling policy
HTTP middlewares:
-
slog-gin: Gin middleware for
slog
logger -
slog-echo: Echo middleware for
slog
logger -
slog-fiber: Fiber middleware for
slog
logger -
slog-chi: Chi middleware for
slog
logger -
slog-http:
net/http
middleware forslog
logger
Loggers:
-
slog-zap: A
slog
handler forZap
-
slog-zerolog: A
slog
handler forZerolog
-
slog-logrus: A
slog
handler forLogrus
Log sinks:
-
slog-datadog: A
slog
handler forDatadog
-
slog-betterstack: A
slog
handler forBetterstack
-
slog-rollbar: A
slog
handler forRollbar
-
slog-loki: A
slog
handler forLoki
-
slog-sentry: A
slog
handler forSentry
-
slog-syslog: A
slog
handler forSyslog
-
slog-logstash: A
slog
handler forLogstash
-
slog-fluentd: A
slog
handler forFluentd
-
slog-graylog: A
slog
handler forGraylog
-
slog-quickwit: A
slog
handler forQuickwit
-
slog-slack: A
slog
handler forSlack
-
slog-telegram: A
slog
handler forTelegram
-
slog-mattermost: A
slog
handler forMattermost
-
slog-microsoft-teams: A
slog
handler forMicrosoft Teams
-
slog-webhook: A
slog
handler forWebhook
-
slog-kafka: A
slog
handler forKafka
-
slog-nats: A
slog
handler forNATS
-
slog-parquet: A
slog
handler forParquet
+Object Storage
-
slog-channel: A
slog
handler for Go channels
🚀 Install
go get github.com/samber/slog-sampling
Compatibility: go >= 1.21
No breaking changes will be made to exported APIs before v2.0.0.
💡 Usage
GoDoc: https://pkg.go.dev/github.com/samber/slog-sampling
Middlewares
3 strategies are available:
- Uniform sampling: drop % of logs
- Threshold sampling: drop % of logs after a threshold
- Absolute sampling: limit logs throughput to a fixed number of records
- Custom sampler
The sampling middleware can be used standalone or with the slog-multi
helpers.
A combination of multiple sampling strategies can be chained. Eg:
- drop when a single log message is produced more than 100 times per second
- drop above 1000 log records per second (globally)
Matchers
Similar log records can be deduplicated and rate-limited using the Matcher
API.
Available Matcher
:
-
slogsampling.MatchByLevelAndMessage
(default) -
slogsampling.MatchAll
-
slogsampling.MatchByLevel
-
slogsampling.MatchByMessage
-
slogsampling.MatchBySource
-
slogsampling.MatchByAttribute
-
slogsampling.MatchByContextValue
Uniform sampling
type UniformSamplingOption struct {
// The sample rate for sampling traces in the range [0.0, 1.0].
Rate float64
// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped func(context.Context, slog.Record)
}
Example using slog-multi
:
import (
slogmulti "github.com/samber/slog-multi"
slogsampling "github.com/samber/slog-sampling"
"log/slog"
)
// Will print 33% of entries.
option := slogsampling.UniformSamplingOption{
// The sample rate for sampling traces in the range [0.0, 1.0].
Rate: 0.33,
}
logger := slog.New(
slogmulti.
Pipe(option.NewMiddleware()).
Handler(slog.NewJSONHandler(os.Stdout, nil)),
)
Threshold sampling
type ThresholdSamplingOption struct {
// This will log the first `Threshold` log entries with the same hash,
// in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0].
Tick time.Duration
Threshold uint64
Rate float64
// Group similar logs (default: by level and message)
Matcher func(ctx context.Context, record *slog.Record) string
// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped func(context.Context, slog.Record)
}
If Rate
is zero, the middleware will drop all log entries after the first Threshold
records in that interval.
Example using slog-multi
:
import (
slogmulti "github.com/samber/slog-multi"
slogsampling "github.com/samber/slog-sampling"
"log/slog"
)
// Will print the first 10 entries having the same level+message, then every 10th messages until next interval.
option := slogsampling.ThresholdSamplingOption{
Tick: 5 * time.Second,
Threshold: 10,
Rate: 0.1,
}
logger := slog.New(
slogmulti.
Pipe(option.NewMiddleware()).
Handler(slog.NewJSONHandler(os.Stdout, nil)),
)
Absolute sampling
type AbsoluteSamplingOption struct {
// This will log all entries with the same hash until max is reached,
// in a `Tick` interval as-is. Following that, it will reduce log throughput
// depending on previous interval.
Tick time.Duration
Max uint64
// Group similar logs (default: by level and message)
Matcher Matcher
// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped func(context.Context, slog.Record)
}
Example using slog-multi
:
import (
slogmulti "github.com/samber/slog-multi"
slogsampling "github.com/samber/slog-sampling"
"log/slog"
)
// Will print the first 10 entries during the first 5s, then a fraction of messages during the following intervals.
option := slogsampling.AbsoluteSamplingOption{
Tick: 5 * time.Second,
Max: 10,
Matcher: slogsampling.MatchAll(),
}
logger := slog.New(
slogmulti.
Pipe(option.NewMiddleware()).
Handler(slog.NewJSONHandler(os.Stdout, nil)),
)
Custom sampler
type CustomSamplingOption struct {
// The sample rate for sampling traces in the range [0.0, 1.0].
Sampler func(context.Context, slog.Record) float64
// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped func(context.Context, slog.Record)
}
Example using slog-multi
:
import (
slogmulti "github.com/samber/slog-multi"
slogsampling "github.com/samber/slog-sampling"
"log/slog"
)
// Will print 100% of log entries during the night, or 50% of errors, 20% of warnings and 1% of lower levels.
option := slogsampling.CustomSamplingOption{
Sampler: func(ctx context.Context, record slog.Record) float64 {
if record.Time.Hour() < 6 || record.Time.Hour() > 22 {
return 1
}
switch record.Level {
case slog.LevelError:
return 0.5
case slog.LevelWarn:
return 0.2
default:
return 0.01
}
},
}
logger := slog.New(
slogmulti.
Pipe(option.NewMiddleware()).
Handler(slog.NewJSONHandler(os.Stdout, nil)),
)
🤝 Contributing
- Ping me on twitter @samuelberthe (DMs, mentions, whatever :))
- Fork the project
- Fix open issues or request new features
Don't hesitate ;)
# Install some dev dependencies
make tools
# Run tests
make test
# or
make watch-test
👤 Contributors
💫 Show your support
Give a ⭐️ if this project helped you!
📝 License
Copyright © 2023 Samuel Berthe.
This project is MIT licensed.