Decluttering logs

[JulianGeis]

🧹 Collect unnecessary or cluttering log messages

Let’s collect log messages from PyPSA-Eur that are unnecessary, overly verbose, or deprecated — especially those that clutter the output during typical workflows.

If you see any of the following, please paste them in a comment below:

• Deprecation warnings (e.g. from pandas, xarray, numpy, etc.)
• Verbose or unclear info/debug messages
• Repeated or outdated warnings

Please include:

• The full log message
• The command you ran (if relevant)

🎯 Goal: Clean up and streamline PyPSA-Eur logs for a better user experience.

Thanks for contributing!

PyPSA specific

• [ ] Importing network from PyPSA version v0.0.0 -> Bug
• [ ] Plotting warning too verbose (When combining n.plot() with other plots on a geographical axis, ensure n.plot() is called first...)

Snakemake

• [ ] The flag 'directory' used in rule all is only valid for outputs, not inputs.

PyPSA-Eur

• [ ] build_industrial_production_per_country.py:170: FutureWarning: Allowing a non-bool 'na' in obj.str.contains is deprecated and will raise in a future version.
• [ ] build_industrial_production_per_country_tomorrow.py:93: RuntimeWarning: invalid value encountered in scalar divide
• [ ] WARNING:pypsa.network.io:The following Link have buses which are not defined: Index(['EU oil refining'], dtype='object')
• [ ] WARNING:scripts.add_electricity:tech_colors for carriers ['', 'none'] not defined in config.
• [ ] WARNING:pypsa.consistency:The following buses have no attached components, which can break the lopf: {'EU'}
• [ ]

"make test".log

[willu47]

Also relates to #1846 and #1844 which provide a mechanism for filtering and organizing logging messages using a plugin.

[coroa]

There is multiple layers to that, that will need some work:

1. There are logs from snakemake that's what #1846 and #1844 relates to and already improves tremendously.
2. There is logging from the individual script files/jobs.

This one here, i would say, concerns 2, and that needs an overhaul of mostly the configure_logging function in helpers, but not only.

Features we'd want:

• Only a bare minimum of information should end up at the standard output. Most should actually end up in the designated python logs. Currently it's the other way around tqdm progress and warnings are written to stdout but have no logging representation.
• Stdout stuff should be prefixed by the rule name or job number.

My vision is:

1. Everything that ends up at stdout/stderr by the jobs should go via the logging module (ie. tqdm needs to be wrapped, multiprocessing needs to be handled specially, warnings should feed into logging, dask distributed client needs a couple of extra plugins).
2. We remove the stream handler from logging in the jobs (unless we are in a cluster environment, where stdout isn't shared) and send the progress output via some sort of zeromq pub/sub to the logging plugin in the snakemake process (introduced in #1846), which does ALL the stdout printing.