drmaa-python icon indicating copy to clipboard operation
drmaa-python copied to clipboard

Using wurlitzer to capture and log stdout/stderr from libdrmaa

Open jakirkham opened this issue 7 years ago • 3 comments

Some DRMAA implementations print information to stdout/stderr. While this information can be useful, having it print to stderr or stdout can be disruptive (e.g. launching jobs with DRMAA in an IPython terminal session) as it will get mingled with other information. It would be ideal to capture this stdout/stderr and instead redirect to somewhere like a Python log, which users can redirect as they see fit.

The main trick is capturing the C-level stdout/stderr pipes and redirecting them to something else that can used with logging. Note that working with sys.stdout/sys.stdout in Python will not address this. For a good explanation of why, please read this blog post by Eli Bendersky. Fortunately a lot of work has already been done on this and there now exists a package, wurlitzer, that handles this redirection logic for us. So it would be good if we could use this to redirect stdout/stderr from libdrmaa to a logger (possibly using StringIO as an intermediary).

jakirkham avatar Jun 30 '18 20:06 jakirkham

Seems like a great idea to me.

desilinguist avatar Jun 30 '18 23:06 desilinguist

Alright I’ll write up a PR.

jakirkham avatar Jul 01 '18 02:07 jakirkham

Working on it in PR ( https://github.com/pygridtools/drmaa-python/pull/70 ).

jakirkham avatar Jul 28 '18 01:07 jakirkham