Rotating file logger

[ghost]

Is it possible to get something similar to python: https://docs.python.org/3.6/library/logging.handlers.html#rotatingfilehandler ?

[dpc]

Does not exist yet, but should be straightforward to make.

New Drain implementing struct that wraps https://docs.rs/slog-stream/1.2.0/slog_stream/struct.Streamer.html, gives it io::Writer that other than writing to a file, also keep track of how much bytes got written, file size, creation time etc. When time comes the underlying File gets closed, renamed and moved.

Feel free to reach out for help on gitter if you'd like to give it a chance.

[przygienda]

related to https://github.com/sile/sloggers/issues/12

[vbmade2000]

I think slog-stream is not used anymore. Can we have a new Drain that take Decorator as input and wrap FullFormat Drain to write output to file ?

[Ploppz]

Now it seems like slog_term is the usual way to go about writing log entries to file. At least in our application we use FullFormat with PlainDecorator.

If log rotation was to be implemented as a Drain is would make it incompatible with slog_term. Are you still of the opinion that it should be implemented as a Drain, @dpc?

To advocade for Drain: Actually, it is kind of confusing at the moment that we use slog_term which is meant for "terminal output" to write to files. Which leads me to think that maybe it would be nice with a new slog-file crate, which can provide a Rotation Drain which can compose well with a new FullFormat Drain that is not specific to terminal output. (what I mean here is that our new FullFormat::new takes a Drain rather than a Decorator, if that makes sense)

The alternative I see is to implement log rotation just by making a std::io::Write wrapper. I'm not sure exactly if this is enough to fulfill all requirements one might have for log rotation.

Let's get this going, I think @vbmade2000 and me will be willing to do the work here if you provide some input @dpc .

[Ploppz]

Oh, I just found that you think the same :) https://github.com/slog-rs/slog/issues/128#issuecomment-296390564 Maybe we will create something then.

[garasubo]

I also came to the situation to need the file rotation functionality for slog-json. I firstly tried to implement log rotation in std::io::Write wrapper, but found that this approach might have a risk to write one line into two separate files when file rotation. So, I think it would be better to have some wrapper for Drain or functionality to replace target file in Drain side.

[Ploppz]

I have put some effort into a fork of the file-rotate crate and I think I have made some good improvements. More info in PR https://github.com/BourgondAries/file-rotate/pull/7

I also came to the situation to need the file rotation functionality for slog-json. I firstly tried to implement log rotation in std::io::Write wrapper, but found that this approach might have a risk to write one line into two separate files when file rotation. So, I think it would be better to have some wrapper for Drain or functionality to replace target file in Drain side.

file-rotate uses std::io::Write but you have some control exactly what should happen with a line when it rotates the file. I wonder this addresses your concern: https://github.com/Ploppz/file-rotate/blob/master/src/lib.rs#L279

[Zylatis]

Ooh I'm glad I found this thread, I was also noodling around with my own file rotation crate (turnstiles) and came across the very issue @garasubo described when applying to slog-json, i.e. split across multiple files. @Ploppz thanks for the link, if i understand correctly your solution was to force write to use write_all?

EDIT: okay i guess it wasn't that as changed mine to do write_all and it still splits over files :sweat_smile: , at least in the json with timestamp etc case

[Ploppz]

@Zylatis Our solution gives you the choice when to split the file. It basically scans all the bytes for \n if you so desire, to split on a new line. For my application I chose ContentLimit::Bytes however, because I wrote an application that concatenates all the log files.

I have worked file-rotate to a satisfactory state where it supports timestamp suffixes, compression, and different delete policies. It's used in production so it will be actively maintained. I recommend everyone to check it out: https://docs.rs/file-rotate/0.5.0/file_rotate/index.html