pipeline log output file at the same time as screen

[stscijgbot]

Issue JP-564 was created by Maria Pena-Guerrero:

Hi,

Is there a way to tell the ST pipe log module to output to a file at the same time as the terminal?

Thank you.

 

[stscijgbot]

Comment by Jonathan Eisenhamer: Depends on the context of what you mean. If running from a unix command line, such as the bash shell, you can use the tee command. The following will redirect stderr to stdout. Then the tee command will write whatever is put to stdout into the file logfile.log while stdout is still shown on the screen.

strun calwebb_spec2.cfg exposure.fits 2>&1 | tee logfile.log

Note that the above form works for any command:

any_command 2>&1 | tee tosomelogfile.log

However, if you wish to do this more programmatically, the pipeline does allow configuration of its logging. See both the [general logging example|https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html#logging-configuration] and the [read about the details.|https://jwst-pipeline.readthedocs.io/en/latest/jwst/stpipe/user_logging.html]

[stscijgbot]

Comment by Maria Pena-Guerrero: Thank you for pointing me at the docs, they are helpful. However, I am trying to run the argument from a script with the following line:

Spec2Pipeline.call(step_input_file, config_file=calwebb_spec2_cfg, logcfg="pipeline-log.cfg")

And this is returning the following error:

raise ValidationError('\n'.join(messages)) E jwst.stpipe.config_parser.ValidationError: Extra value 'logcfg' in root

What is the correct way to do it?

thanks

[stscijgbot]

Comment by Jonathan Eisenhamer: Will have to explore a bit more, but the error is that call does not understand logcfg. While that is explored, try the following: First, rename the log config to stpipe-log.cfg and make sure it is in the current working directory. As specified in the detailed documentation, the configuration should automatically be found.

[stscijgbot]

Comment by Jonathan Eisenhamer: Just verified that using call does not allow the specification of a logging configuration, so the previous comment is currently the only way of handling logging. I will file an issue for this.

[eas342]

Has this now changed so you can log both to stdout and a file? It appears as though running .call in iPython ignored my stpipe-log.cfg in the local directory but when I ran

from jwst.pipeline import Detector1Pipeline
Detector1Pipeline.call('jw_uncal.fits',logcfg='stpipe-log.cfg')

and had this stpipe-log.cfg configuration file:

[*]
handler = file:pipeline.log, stdout
level = INFO

it appears to be logging both stdout and the pipeline.log file simultaneously with JWST version 1.13.4.

[melanieclarke]

Associated JP ticket is closed.