Capturing stdout with slurm

[rueberger]

We've been running our experiments with the slurm scheduler which captures stdout and stdin and pipes them to a file. This seems to confuse sacred, which doesn't capture stdout in this case.

Easiest solution by far would be just telling sacred where to look for stdout. Is there a way to do that?

Thanks! A

[Qwlouse]

That is an interesting case I hadn't thought about. So far this is not possible, no. If you are fine with only capturing stdout from within python, you could try to set the capture mode to sys. My guess is that this wouldn't be affected by stdout redirects:

from sacred.settings import SETTINGS
SETTINGS.CAPTURE_MODE = 'sys'

But I'll mark this as a feature request nonetheless.

[rueberger]

This worked just fine! Thanks!

[johny-c]

Hi, I have the same situation but the fix does not seem to work for me. Some of the output is captured some times. Did something change in the meantime with respect to output capturing?

[Qwlouse]

@johny-c I don't think anything relevant changed. If you are still interested in this: what do you mean by some output is captured? I am asking because the capture mode sys only applies to python based outputs, i.e. output that goes through sys.stdout and sys.stderr and cannot capture outputs from any other sources (e.g. compiled modules that output from C, or invocations of non-python tools). If you experience indeterminacy even in capturing python-based outputs, then that is definitely a problem, and this should be flagged as a bug.

[johny-c]

Hi @Qwlouse , I define my logger based on python's logging module. I can see my logger's console output and file output, but I cannot see the output in sacredboard's captured output. Maybe that's a sacredboard issue? Otherwise, it might have to do with the fact that I setup the logger, after I start the experiment since I want to associate the logger's file path with the run's id.

[Qwlouse]

Ohh, I found the problem: Sys-based capturing relies on replacing sys.stdout and sys.stderr with custom wrappers. But the StreamHandler still maintains a copy of the old sys.stderr and logs to that. So in fact any Sacred experiment that uses capture mode sys looses their logging. That is clearly a problem!

You might be able to work around it with something like this, which should give you most logs (except for the "experiment started" logs):

@ex.pre_run_hook
def set_logger_stream(_run):
    _run.root_logger.handlers[0].stream = sys.stderr

I have to think about how to best fix this problem in a clean way. Let me know if you have any thoughts on this, especially also concerning the general integration of sacred with logging, since I am not entirely happy with the way it is currently done.

[johny-c]

I think this might be easier to solve with an object-oriented interface which I see has been discussed in #193. Are there any news on that? In the meantime, I thought of this (probably naive) solution: I would move create_run inside the Run class as a static constructor-like method and make initialize_logging also a method of Run. Finally I would expose the Run class as a parameter of an Experiment, something like: Experiment(name=None,..., run_type=Run) This way, users would be able to subclass Run and add custom functionality, such as loggers.