cylc-uiserver icon indicating copy to clipboard operation
cylc-uiserver copied to clipboard

cat-log: support large files

Open oliver-sanders opened this issue 2 years ago • 7 comments

Currently we cap the log at 5000 lines.

This is to protect the browser, the GUI could have multiple views (incl logs) open simultaneously. Long log files cause issues with Cylc Review (which also hits this problem). Past a hard-coded limit Cylc Review will only offer a "raw" file view which exists outside of the regular UI and scales better.

Two options for the Cylc log view:

  1. Paginate the log files.
    • We can load the logs in X line chunks.
    • E.G. to start reading from line 5000 we could do tail -n +5000.
    • This would involve adding an "offset" argument to the log subscription.
    • Ideally we would be able to query the number of lines in the file so the GUI could display this, however, I'm not sure we can do that without reading the file?
  2. Offer the ability to open long log files in another browser tab ala Cylc Review.
    • This would mean building a light-weight web app to display the log lines in a simple <code> block.
    • This app would need to perform the same authentication as the regular UI (a couple of lines of code which bung the token into a cookie).
    • It would make the problem easier if we added a REST endpoint for accessing the job logs just for this view (no need for a GraphQL client in the new app).

Pull requests welcome!

oliver-sanders avatar Feb 15 '23 10:02 oliver-sanders