jamhall
more useful logging output
As mentioned in #623 our aim is to create feature table documenting the supported S3 API surface.
I open this issue to discuss what possible approaches we have to this problem, to solve the problem in an effective way.
In terms of logs, it's worth thinking about:
- who will use them and how (community feedback welcome!)
- what solutions and standards exist in this area,
- how to generate logs.
In terms of the log standard, we have standard log formats derived in particular from:
- Apache (AFAIK Common Log Format)
- Nginx
- Traefik
- Amazon S3 (via CloudWatch)
- Elastic Load Balancing (via CloudWatch)
- CloudFront (via CloudWatch)
- Google Cloud Storage
- Azure Files (?)
- other (which one?)
It is also worth considering whether the format is structured or textual, and if textual, whether it is multiline or single-line.
Comments with examples of formats are welcome to make it easier to see what we are dealing with. I will try to collect them here regularly.
In terms of log consumer, it is worth mentioning:
- classic ELK set, including grok,
- GoAccess
- syslog
- other (what?)
Our configuration of logging is available at: https://github.com/jamhall/s3rver/blob/3154cd6514b9aa919e7949149e60180d234a2703/lib/middleware/logger.js
I think I'm going to attempt to recreate CloudTrail's logging output internally. We already do this to a short extent with S3 events. This refactor would replace all existing calls to the logger and event triggers.
Currently, the events API is only accessible via the JavaScript API with app.on('event', ...). This could be made accessible from the command line with a new --json option that should make S3rver's stdout to print as a NDJSON stream. Without this flag, logging events should continue to use a common-log-format-inspired output.
