[BUG] Unexpected warnings when invoking MNMG kmeans.fit() method

[miguelusque]

Describe the bug Hi, while running the MNMG example notebook for kmeans, the following warning messages are displayed.

Even if those are warnings, I think it might be well-worth to double-check if they might be suppressed.

Steps/Code to reproduce bug Running the notebook without changes.

Expected behavior No warnings displayed.

Environment details (please complete the following information): DGX-1 server, running rapidsai/rapidsai:22.02-cuda11.5-runtime-ubuntu20.04-py3.8 container.

[miguelusque]

I managed to hide those warnings adding silence_logs=logging.ERROR as follows:

import logging
cluster = LocalCUDACluster(threads_per_worker=1, silence_logs=logging.ERROR)
client = Client(cluster)

Nevertheless, I think it is well-worth to investigate if we can do anything to have those errors hidden (either in Dask or in cuML).

Thanks!

Miguel

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[miguelusque]

I think this issue is still relevant.

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[miguelusque]

Hi,

This issue is still relevant. I have just tried it with 22.06 container, and the issue still persists.

Thanks!

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[lowener]

I found this dask issue relevant https://github.com/dask/distributed/issues/6882. I tried to use and adapt the code exemple there to ignore the logging of distributed.worker

import logging
    logger = logging.getLogger("distributed.worker")
    class IgnoreComputeFailedLogs(logging.Filter):
        def filter(self, record):
            return not (record.funcName == "execute" and record.msg.startswith("Compute Failed\n"))
    logger.addFilter(IgnoreComputeFailedLogs())

But I did not see any changes on the warning messages.

[dantegd]

Closed by #5299