(Dashboard?) monitor for keyword tracking of logs

[sadielbartholomew]

It's hard to know where to put this given the current state of development, plans, & the Issue tracker, but this relates to the:

• UI dashboard (see #94);
• "future" Cylc Review, whatever form it should take e.g. under current plans probably integrated into the UI in some way (see e.g. https://github.com/cylc/cylc-flow/issues/2833);
• suite & task analytics (there has been lots of discussion on this including across GH Issues, but there's no specific Issue, I'll probably create one as a cover-all for UI analytics & the long-term dashboard plans).

A user has indicated they would benefit from a means in the UI to continuously track information they find important in suite log files. For example, if a keyword of interest to the user such as a certain Python error name is raised in a suite & emerges in the job.err, that could be flagged in this component as & when it occurs if set up to be tracked by the user.

Summarising their specification (see below) it would ideally be:

• visually striking (utilising icons, colours etc.);
• customisable, with user-defined keywords etc. & styling (as above) based on outcomes;
• inter-linked with the logs (under Cylc Review or the "future" integrated version), e.g. with links directly to log lines in question as flagged.

Basic tracking can already be done manually via the command line via cylc cat-log & some filtering commands like grep, but a visual integrated UI utility with lots of crossover & interface to/from an analytics tool is the essence of their feature request.

Full feature request (user's words)

[...] what I’ve got in mind is having a logfile in a format suited to passing to a dashboard application, which can give me an ability to get a top-level overview over lots of past runs, with drill-downable capabilities when I want to investigate failures warnings.

I’ve got in mind something similar to StatusPage’s green bars, but with a few extra tweaks:

• a visual indication of last N runs, with different colours/icons indicating different things (e.g. red for failures, orange for warnings present, green all OK).
• Adding different icons to say indicate whether any warnings/errors present are known/unknown (based on user-provided filters)
• And possibly extending the colour palette / some other visual property to indicate runs which have ended late due to queuing / longer duration.
• Tool-tipping over a given bar can then, like StatusPage, reveal further summary information from the logfile.
• And clicking on that given bar can expose even more information from the logfile.

[oliver-sanders]

Along the lines of the illustration in the design document or something different?

[sadielbartholomew]

Along the lines of the illustration in the design document or something different?

You'll have to point me to the place in / aspect of the design document you are referring to here, as I cannot see anything that resembles this functionality mocked up on there.

[hjoliver]

a visual indication of last N runs ...

What's "runs"? Suites or jobs?

[hjoliver]

a logfile in a format suited to passing to a dashboard application,...

I guess in our context, this would be a UI Server sub-service that scrapes all of the user's suite logs.

[sadielbartholomew]

Maybe I will get back to everyone about this request on Riot, to clarify aspects & get to the crux of what is wanted here...