Can't use own logging handlers / Can't propagate logger

[marcinhlybin]

Hi,

I have some troubles changing usp logging behavior. By default it outputs to stdout. It should be easy enough to use my own logger and handlers with propagation to the root logger.

The only way I figured out so far is to hack logging within context with following code:

import logging
import os.path, pkgutil
import usp.tree

logger = logging.getLogger('')
logger.setLevel(logging.DEBUG)
logger.propagate = True

handler = logging.FileHandler('/tmp/test.log', mode='a')
formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)

class CustomLogger():

  def __enter__(self):
    pkgpath = os.path.dirname(usp.tree.__file__)
    modules = [module.name for module in pkgutil.iter_modules([pkgpath])]
    
    for module in modules:
      l = logging.getLogger('usp.' + module)
      l.handlers = logger.handlers
      
    # Disable DEBUG for usp submodules
    logging.disable(logging.DEBUG)

  def __exit__(self, *args):
    logging.disable(logging.NOTSET)


with CustomLogger():
  page = "https://www.github.com/"
  tree = usp.tree.sitemap_tree_for_homepage(page)

It ain't pretty but it works. Any hints how to do it correctly?

[pypt]

Hey, thanks for the issue!

Custom loggers are not currently supported but PRs are welcome! Logging is already encapsulated in a class:

https://github.com/berkmancenter/mediacloud-ultimate-sitemap-parser/blob/develop/usp/log.py

so one could do something similar to how custom user agents are supported, i.e. have an "abstract logger" class to define an interface for logging, a default implementation, and an ability to pass an optional logger object to sitemap_tree_for_homepage(). Web client implementation of this approach is here:

https://github.com/berkmancenter/mediacloud-ultimate-sitemap-parser/tree/develop/usp/web_client