proteus icon indicating copy to clipboard operation
proteus copied to clipboard

Implement structured logging

Open cekees opened this issue 9 years ago • 1 comments

It would be nice if we could easily parse our .log file to extract information and some state at each time step. For example, the number of linear and nonlinear iterations per model per times step, wall clock times, memory usage. I think we can probably do it by switching to yaml and deciding how to structure log messages that have some quantitative content. We could probably eliminate the need for the xml part of the xdmf file, though for now the Archiver rewrite addresses the scalability problem with xml. Here are some earlier thoughts:

Some more ideas on this. We need to improve our logging as well as our archiving, and there is some overlap between logging, profiling, checkpointing, and archiving. The Python standard logging facility (https://docs.python.org/2/library/logging.html) has gotten a lot better. I wonder if we might be able to condense some of our functionality by using structured logging: https://docs.python.org/2/howto/logging-cookbook.html#implementing-structured-logging and customized handlers: https://docs.python.org/2/howto/logging.html#logging-flow https://docs.python.org/2/howto/logging.html#handlers

JSON is an obvious candidate for structured logging. YAML is a human-friendly data serialization format that is a superset of JSON: http://www.yaml.org/ YAML now has Python and Javascript support http://pyyaml.org/ https://github.com/nodeca/js-yaml

cekees avatar Sep 05 '15 21:09 cekees

@cekees Looks that needs to be still open

adimako avatar Jan 22 '19 09:01 adimako