Improve Lithops logging configuration

[tomwhite]

Lithops configures Python logging itself, which can be handy, but equally can make it hard to configure logging for other (non-Lithops) modules. For example, using logging.basicConfig in your code doesn't work well if Lithops has set up logging.

This issue is to discuss different ways to configure logging, and eventually add to the documentation and reflect in the examples.

1. Using Lithops logging setup

The default is to log at INFO level to the console (stderr). This is fine, but can be quite verbose, particularly for large workloads.

But you can also send Lithops logs to a file. This is a useful option as it keeps log information in case you want to debug a problem after the Lithops job has failed.

For example, set log_filename to lithops.log in your Lithops conifguration. Note that the log file will be appended too (and could therefore grow indefinitely). If you want something more sophisticated (such as rotating log file), then turn off Lithops logging setup, and configure logging yourself, as described below.

2. Configuring logging yourself

Another option is to turn off Lithops logging setup, by passing log_level=None to LithopsDagExecutor (which is forward to Lithops' FunctionExecutor). This tells Lithops not to configure logging, which means that you can configure logging yourself, and still get log messages from Lithops - it gives you full control, although you have to do a bit more work to get what you want.

Option 1: Set logging level globally to INFO with the given format. Note that Lithops will produce quite a lot of output

logging.basicConfig(level=[logging.INFO](http://logging.info/), format="%(asctime)s [%(levelname)s] %(filename)s:%(lineno)s -- %(message)s")

Optin 2: Set the logging level for Lithops to WARN, but other modules to INFO

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(filename)s:%(lineno)s -- %(message)s")
logging.getLogger("lithops").setLevel(logging.WARN)

Option 3: Send Lithops logs to a file, while sending everything else to the console

LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(filename)s:%(lineno)s -- %(message)s"
        },
    },
    "handlers": {
        "console_handler": {
            "level": "INFO",
            "formatter": "standard",
            "class": "logging.StreamHandler",
        },
        "file_handler": {
            "level": "INFO",
            "formatter": "standard",
            "class": "logging.FileHandler",
            "filename": "lithops.log",
            "mode": "a",
        },
    },
    "loggers": {
        "": {  # root logger
            "handlers": ["console_handler"],
            "level": "INFO",
            "propagate": False,
        },
        "lithops": {"handlers": ["file_handler"], "level": "INFO", "propagate": False},
    },
}
logging.config.dictConfig(LOGGING_CONFIG)

[TomNicholas]

I guess verbose logging but sent somewhere where it's not in your face if you don't want it would be the most useful default for users?

[tomwhite]

Agreed. I'll update the examples to do this.

[tomwhite]

I'll update the examples to do this.

Done in https://github.com/cubed-dev/cubed/commit/c6b30c8e959377683234432a952aa3a20931366a