redash
redash copied to clipboard
getredash
Filters become slow and inefficient on large datasets
Might be relevant: http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records
I've faced this problem before. ui-grid is Angular-based and can be easily tweaked to scale very well. w2ui also scales well but does not fit in well with the rest of an Angular system.
We're using filtering not only for the data grid, so I prefer to find something that isn't coupled to the grid itself. But I will take a look at the implementation -- maybe the filtering code is separate or even another library.
I don't have an example with a visualization, but here: http://demo.redash.io/queries/430/source try changing the action from "view" to something else and then back to "view".
Hello! Back from vacation, sorry about that.
A few questions:
- is there a non-minified version of demo.redash.io ? It'd make isolating the problem a lot easier.
- What table/grid plugin are you using, if any?
No unminified version, but you can use the Vagrant box (if you do, let me know and I'll send you a data dump to use with it).
We're using an old and modified version of ng-smart-table (https://github.com/EverythingMe/redash/blob/master/rd_ui/app/scripts/ng_smart_table.js).
Ah, that could be part of the problem. Any high-performance Angular project uses ui-grid now (they've done a lot of awesome stuff).
Just switching to ui-grid won't solve the problem, because filters are used with other visualizations too (charts, maps, etc). But I'll check how they implemented filtering in ui-grid (assuming it has good performance for large data sets). And regardless of filters performance, it will be good to replace the custom ng-smart-table with something maintained.
Ah, if you're using filters elsewhere, just ui-grid won't solve the issue.