st2web icon indicating copy to clipboard operation
st2web copied to clipboard

Better handle large execution results and large datasets

Open Kami opened this issue 4 years ago • 1 comments

We should update st2web to be smarter and better handle executions with large results.

Right now if the execution which is loaded (e.g. either user clicks on it in the history / dashboard page or it's the last execution which result and details are automatically loaded when the page loads) contains large result field, the whole browser window / tab will freeze.

We should handle such scenarios better.

Background, Context

For a long time, working with large executions and result has been StackStorm's weak point.

https://github.com/StackStorm/st2/pull/4846 mostly mitigates and solves that for the server-side, but now the issue will be more pronounced on the client (st2web) side.

Possible Improvements

There are many ways to improve on the status quo.

For one, we should not be automatically loading large results and passing them through the code highlighter widget.

We could do something similar to what various CI systems do (think Circle CI, Github actions, etc.) which display "View raw logs" tab which opens a new window with the raw output or offers a way to download the output as file.

We could add some additional logic to the api to return result_size (or similar field) with each execution object and then inside the st2web we could check if that size is over some threshold (e.g. 100 KB). If it is, instead of trying to display the result and using code highlighting widget on it, we should display something along the lines of "This execution result is too large to be displayed here. Click to view raw execution result or to download raw execution result".

IIRC, we already have "raw action result / attribute value" API endpoint so we could simply use that one for the link target (and if we don't adding one should be straight forward - that endpoint should just return raw result as string without doing any JSON loading).

Kami avatar Mar 14 '21 22:03 Kami

In the light of https://github.com/StackStorm/st2web/issues/867, I decided to also try and tackle server side component of that Web UI issue.

Good news is that with those changes, st2api and st2action runner will breeze through large executions without much sweat, but those large results will still freeze the whole Web UI.

To improve on that situation I implemented two changes (still WIP, need to polish it, add tests, changelog entry, etc.):

  1. Add result_size field to the ActionExecutionDB model. This field contains size for the serialized execution result in number of bytes. It's only available for the new field storage format (which is totally fine) and is populated lazily in to_mongo() so we don't need to serialize data twice to be able to access the size.

  2. Add new API endpoint which allows user to retrieve or download "raw" action execution result (/v1/executions/<id>/result/raw[?download=true])

My JavaScript is very rusty these days, but hopefully implementing WebUI part won't be too hard so I will try to tackle it over weekend.

The plan basically is that instead of trying to render the execution result in the code highlighter widget for executions with large results (we will use same sane default value for definition of "large" based on benchmarking etc, but user will also be able to override it in st2web config), we will display something along the lines of "This execution result is too large to be rendered here. Click to view it raw or to download the result as JSON file".

Kami avatar Mar 15 '21 23:03 Kami