silx-kit
Large mixed type dataset logged to the console can be missleading
Describe the bug
Too large to display mixed type dataset leads to the message: The dataset is too big to be displayed and was logged to the console instead.
This seems missleading to me, since not everyone will look into the dev tools to open the console.
To Reproduce
- Go to https://h5web.panosc.eu/?file=epics.h5
- Click on
arrays->Vdata with mixed types - Message is:
The dataset is too big to be displayed and was logged to the console instead.
Expected behaviour
To avoid missleading message:
- the data could not be displayed at all:
The dataset is too big to be displayed., or - the data could be displayed on-demand:
This dataset is big,display it, if possible with a foldable json viewer.
Context
- H5Web context: demo
The initial problem was that the browser would get stuck while parsing/displaying large JSON outputs. So we could also look for a way to solve this performance issue with an external library that parses JSON asynchronously and can render it in a virtualized way.
After some more research, I couldn't find a library for stringifying JSON asynchronously in the browser. I got decent results using greenlet to move the JSON.stringify call to a web worker, but transfering the output string back and displaying it remains an issue. It basically just shifts the bottleneck... I think that stringifying a very large dataset is just a bad idea, full stop.
Perhaps we should consider removing the RawVis altogether and trying to handle more complex compound types with the MatrixVis instead? I'm thinking that for array fields (regardless of shape), the visualization could display the first few numbers - e.g. [1.2, 34.5, 678.9, ...], and for other non-printable dtypes, we could just show <COMPOUND>, <REGION>, etc. or something along those lines.
Better fix in #1485: instead of logging the JSON string to the browser console, users can now export the dataset to JSON via the toolbar: