R API for peeking at logs?

[kendonB]

Ref: https://github.com/HenrikBengtsson/future.batchtools/issues/17

It would be great to be able peek into the logs from a R session in some sort of nice way.

From @HenrikBengtsson:

Having an R API to peek into/browse around in an existing (and possibly even a live/hot) batchtools job registry would be neat. If possible, it's not hard to imagine of pulling out an overview of current jobs, e.g. classical stdout, fancy color-annotated stdout, or even a GUI/Shiny dashboard. I think that such a feature request for a basic querying API is better address to the batchtools project itself.

[mllg]

You can query current/live jobs with findRunning()/findOnSystem(). And there is showLog() to inspect log files using file.show(). You can set the pager option to open the log with an editor, e.g. vim. The up-to-dateness of the log depends on the scheduler. Some schedulers write the log files after the job has terminated, others flush frequently.

ANSII colors are usually problematic and IMHO you do not want to have color codes in log files. You could set up a shiny up to browse log files though, but this should be done in an extra package to keep the dependencies for batchtools reasonable.

[kendonB]

@HenrikBengtsson, do you have any guidance for accessing these for future.batchtools generated jobs?

[HenrikBengtsson]

To clarify this feature request: Is it possible (and safe) to set up a pointer to a batchtools registry from a separate R session (by only knowing its path on the file system) and use the find*() functions to query it's status? I'd expect that this has to be read-only in order for such external monitoring processes not to corrupt the state of the registry.

[mllg]

To clarify this feature request: Is it possible (and safe) to set up a pointer to a batchtools registry from a separate R session (by only knowing its path on the file system) and use the find*() functions to query it's status?

Yes, it should be possible if you use loadRegistry("<path>", writeable = FALSE). If you encounter problems while only monitoring, it is a bug in batchtools.

[vjcitn]

can you comment further on this issue ... here is a case where findDone cannot look at this registry ... I have tried setting writeable=TRUE as well but that seemed to hang

> xx
Job Registry
  Backend  : Socket
  File dir : /udd/stvjc/VM/HOW_ITS_DONE/BATCHTOOLS/t100k
  Work dir : /udd/stvjc/VM/HOW_ITS_DONE/BATCHTOOLS
  Jobs     : 50
  Seed     : 3366
  Writeable: FALSE
> findDone(xx)
Error in assertRegistry(reg, sync = TRUE) : 
  Registry must be writeable to be synced

Enter a frame number, or 0 to exit   

1: findDone(xx)
2: assertRegistry(reg, sync = TRUE)

Selection: 0
> sessionInfo()
R version 3.5.0 beta (2018-04-10 r74578)

[mllg]

Yep, that's a bug. Fixed and tested in 326349bb9dfa8a592b00a17887ec425ab50078f3.

[vjcitn]

Thanks! Want to bump your version number?

On Thu, Apr 12, 2018 at 4:06 PM, Michel Lang [email protected] wrote:

Yep, that's a bug. Fixed and tested in 326349b https://github.com/mllg/batchtools/commit/326349bb9dfa8a592b00a17887ec425ab50078f3 .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mllg/batchtools/issues/157#issuecomment-380928591, or mute the thread https://github.com/notifications/unsubscribe-auth/AEaOwmUczw2sII4rAb8FP5c6cG_kg_Utks5tn7PigaJpZM4Qis9F .

[mllg]

Bumped.

[mllg]

NB: I've improved docs and error messages for the ro/rw stuff.