Setting ClearML package log level

[fjean]

I could not find a way to control the verbosity of the messages that are logged by ClearML (I'm talking about it's own messages, e.g. clearml.Task - INFO - No repository found, storing script code instead). It doesn't seem to be possible to do this in the standard way where the log level of any Python package can be changed by getting the package's root logger, i.e.

logging.getLogger("clearml").setLevel(logging.WARNING)

Is there a way to control ClearML log level, or to deactivate any logging from the ClearML package (unless it's a warning or an error)?

[jkhenning]

Hi @fjean, did you try setting it before or after importing clearml?

[fjean]

I tried before and after importing clearml, and still the messages show up. Here is some code to test the issue:

import logging
from clearml import Task

logging.basicConfig(
     level=logging.INFO,
     format= '[%(asctime)s] %(levelname)s - %(message)s',
     datefmt='%H:%M:%S'
 )
logging.getLogger("clearml").setLevel(logging.WARNING)

# Show root logger
print(logging.getLogger())
# Show ClearML logger
print(logging.getLogger("clearml"))

logging.debug("Debug message")
logging.info("Info message")

task = Task.init(
    project_name="testclearml",
    task_name="Test task",
    task_type=Task.TaskTypes.training,
)

The output is:

<RootLogger root (INFO)>
<Logger clearml (WARNING)>
[10:55:18] INFO - Info message
ClearML Task: overwriting (reusing) task id=ad1bec5cc1ea47c18261fb8bd1720f08
2022-08-08 10:55:19,232 - clearml.Task - INFO - No repository found, storing script code instead
ClearML results page: http://localhost:8080/projects/f9430eeb1f15463eab9fc99ffc4765f2/experiments/ad1bec5cc1ea47c18261fb8bd1720f08/output/log
2022-08-08 10:55:19,393 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
2022-08-08 10:55:20,548 - clearml.Task - INFO - Finished repository detection and package analysis

[jkhenning]

So clearml re-configures the logging, and probably overrides your settings. Take a look at the clearml default logging settings

These settings can be overridden in your clearml.conf file using:

logging.loggers.clearml {
   ...
}

This entire structure is fed into Python's logging.config.dictConfig()

[fjean]

I have tried putting

logging.loggers.clearml {
    level: WARNING
}

at the end of the clearml.conf file, it didn't work; I also tried this inside the sdk section of the file, didn't work either. I also tried the following (inside and also outside of the sdk section of the file), and it didn't work either:

logging {
    loggers {
        clearml {
            level: WARNING
        }
    }
}

[jkhenning]

Can you try overriding everything, like so:

logging {
    version: 1
    disable_existing_loggers: 0
    loggers {
        clearml {
            level: WARNING
        }
        boto {
            level: WARNING
        }
        "boto.perf" {
            level: WARNING
        }
        botocore {
            level: WARNING
        }
        boto3 {
            level: WARNING
        }
        google {
            level: WARNING
        }
        urllib3 {
            level: WARNING
        }
    }
}

As a top-level entry in the clearml.conf file?

[fjean]

Can you try overriding everything, like so:

logging {
    version: 1
    disable_existing_loggers: 0
    loggers {
        clearml {
            level: WARNING
        }
        boto {
            level: WARNING
        }
        "boto.perf" {
            level: WARNING
        }
        botocore {
            level: WARNING
        }
        boto3 {
            level: WARNING
        }
        google {
            level: WARNING
        }
        urllib3 {
            level: WARNING
        }
    }
}

As a top-level entry in the clearml.conf file?

Unfortunately it still shows the messages

[fjean]

When Task.init is called, it seems that the logging dictionary is correctly applied: if I print the clearml logger level before and after the call to logging.config.dictConfig in backend_config/log.py, I can see the level change from INFO to WARNING.

However, the clearml log level goes back to INFO by the time the call to Task.init has ended; the level is changed back to INFO somewhere in the init process. I could not figure out where just yet.

[fjean]

Any idea what could cause the log level to revert back to INFO before the end of the call to Task.init?

[erezalg]

Hi @fjean,

That indeed seems like a bug...I'll check internally! I think you can use, backend_config.config.logger().setLevel() to set the level after Task.init, try setting it to the original level.

[fjean]

@erezalg Indeed calling backend_config.config.logger().setLevel(logging.WARNING) after calling Task.init does suppress all other INFO log messages that used to show after that point (e.g. uploading files and all). The only INFO messages showing up are the ones from within the call to Task.init.

[idanab1010]

@erezalg Would it be possible to set the log level before calling Task.init (important also for remote execution)?

[erezalg]

Hi @idanab1010,

We should fix the issue where Task.init() overrides the config. Then you'll be able to use clearml.conf to specify log level for agents. Will update here once it's done

[qwaxys]

@fjean the logger default level can now be changed in the new release 1.8.1. If you have any questions or feedback, feel free to let us know!

[jkhenning]

Closing this issue. Please reopen if it's still relevant.