DatatablesFilterBackend: performance gain by removing call to count()

[matthewhegarty]

I am using a setup similar to the AlbumFilterListView.

I am using the DatatablesFilterBackend on a table with around 20k rows in production. I have been doing some investigation to see if I can improve the load time for the table.

I notice that it is possible to remove the second call to count():

self.set_count_after(view, queryset.count())

I think this call exists because if results have been filtered there will be a new count, however removing this line doesn't cause any issue, because in pagination.py, the check against _datatables_filtered_count has a default call to queryset.count() which appears to return the cached value.

Since count() can be quite expensive, I have noticed that removing it results in a decent performance improvement (removes around 1 second of load time).

I notice that _datatables_filtered_count is also referenced in renderers.py, but this case is not hit when pagination is enabled (I think).

I wanted to ask your thoughts about refactoring this call - are there any issues with doing this? I would be happy to help if you think it is feasible. I have tested on my application and on the example application, and they seem to work fine without this second call.

[izimobil]

Hi @matthewhegarty sure, go ahead, code is 100% covered by tests so you can experiment performance improvements and see how it goes...

[matthewhegarty]

Not much to add here after 1 year, except that I continue to look into it now and then. I did spend some time looking into performance increases but didn't come up with anything I could contribute back to the library.

I use a patched version of this library in production to remove the set_count_before() and set_count_after() calls, because this gives a huge performance increase with large datasets. The ideal solution (I think) would be to allow this to be configurable via the datatables config, so that if pagination is disabled, then there is no need for these calls to happen. I haven't looked into this as yet.

Feel free to close this - at least it can serve as a reference for anyone else trying to improve performance.

[matthewhegarty]

Basically I had limited success in optimising. Some findings noted here