Support for default python logging handlers

[benelot]

The handlers of the default logging library do a good job at sending information to different destinations. https://docs.python.org/3/library/logging.handlers.html

Your library does an excellent job at creating log messages. Why not combine both? At our company, we already have several custom handlers, so that would be perfect to just plug them into eliot. If you are interested, how can I help you with the implementation of it?

[benelot]

Alternatively, is there a default interface to implement for eliot destinations?

[itamarst]

Hi,

1. Half curiosity, half relevant: what are you using Eliot for?
2. Destinations are just a callable that takes a dictionary.
3. stdlib logging Handler expect logging.Record objects.

So the pseudo-code would be something like:

def destination_for_stdlib_handler(handler):
    def destination(m):
        record = makeLoggingRecordForMessage(m)  # choose INFO vs ERROR level, figure out what message would be if any... maybe message is JSON serialized dict?
        handler.emit(record)
    return destination

And example usage:

from logging.handlers import SocketHandler

import eliot
eliot.add_destinations(destination_for_stdlib_handler(SocketHandler('yourlogserver', 8080)))

There's interesting policy questions of how you turn Eliot message into a logging.Record, which depend in part on how you consume logs (do you expect them to be human readable? Will you use eliot-tree? etc.).

[itamarst]

Logging levels would plausibly be INFO by default, ERROR for a failed action message or a traceback.

[itamarst]

As next steps I would suggest:

1. If the above is enough information (and if you need more info, happy to explain it more detail), implement something for internal use, and then after you've used it a bit report back on what you learned and what patterns you chose. Then we can see if there's a more general solution that'd be helpful for everyone.
2. Alternatively, I am available for consulting work on Eliot (and more broadly, helping teams using Python get past all the random hurdles keeping them from working on actual features and bug fixes).

[benelot]

I am using Eliot to send messages to an ELK stack (logz.io in our case), so we have a nice handler for rest-calls optimized for higher throughput. I played a bit with destinations and found out the same problem you identified: logging.Records are stupid when it comes to representing eliot messages. Yes, we would love to use eliot-tree, so my current solution is just to use the rest-call sender backend that sends the json.dumps(eliotmessage) to the stack. Works well so far. If you are interested in a contribution regarding this, I can post the backend code and we can turn it into proper code for a pull request. For now I just pack it into a function called to_http(message): which calls the sender inside with the json.dump.

[itamarst]

I'd be interested in seeing the code, yes, and understanding how general purpose it is. E.g. would it work for any ElasticSearch backend?

I would recommend BTW reusing/wrapping the JSON code already in Eliot (and if that means some refactoring, that can be done), both because it allows registering custom serializers, and because I will at some point be switching it to a faster JSON library (#372).

(By the way, you can probably convince Kibana to show messages in the right order: find specific task_uuid, sort by task_level (you just need to make sure it's interpreting it as a list of integers, not a string). Not quite as pretty as eliot-tree, but it does show messages in the right order.)