Track-state functionality grinds CIDER to a halt on a project with many namespaces and heavy :refer :all usage

[alexander-yakushev]

We are using CIDER with a really big project (1000+ namespaces). Most of those namespaces have a common "prelude" form that requires and refers many other namespaces. With a project like this, CIDER consistently slows down the application and the Emacs. We were able to pinpoint the problem to track-state middleware, particularly the part that sends the namespaces that were changed since the last time. When the cache is empty, it sends all namespaces from cider-nrepl to cider. This actually creates three problems:

1. Clojure side (on cider-nrepl) spins like crazy and spends a lot of time in cider.nrepl.middleware.track-state/filter-core-and-get-meta function.

2. The huge result (~50Mb) gets transferred to the client. This takes ages when doing any remote development.

3. Emacs caches the received result and keeps it in memory all the time which makes Emacs GC last a few seconds. Most Emacs operations become ridiculously slow.

We were able to reduce this problem a bit by blacklisting :doc and :arglists from the data that gets transferred from cider-nrepl to CIDER. Apparently, it doesn't break anything, so it is not obvious why they are there in the first place.

In any case, it looks like this way of tracking changes (transferring huge maps on every change) doesn't scale too well. I could try to suggest a less strenuous way to do this, but I have to know where else the information from track-state is used. So far, I found only that font-locking relies on it.