RFE: Improve ement's memory footprint

[sergiodj]

OS/platform

GNU/Linux

Emacs version and provenance

GNU Emacs 29.4.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.33, cairo version 1.16.0) of 2024-07-16

I built my own Emacs.

Emacs command

emacs server + client

Emacs frame type

A mix of both (but I only use ement on GUI)

Ement package version and provenance

0.15.1, from ELPA

Actions taken

Like many Emacs users, I leave mine open for several days/weeks/months. Ever since I started using ement, I've been noticing Emacs getting slower and slower over time, to the point where the best "fix" is to restart it.

I haven't tried to tweak any garbage collector settings, but I don't think that's where the problem lies. It seems like all this memory (see below) is actually being used, so there's nothing to be collected.

Observed results

I ran memory-report, which took almost 1 hour to gather all data (!!!). This is what I saw:

Estimated Emacs Memory Usage

   3.2 GiB  Total Buffer Memory Usage
   751 MiB  Overall Object Memory Usage
   549 MiB  Memory Used By Global Variables
    83 MiB  Reserved (But Unused) Object Memory
    11 MiB  Memory Used By Symbol Plists
   812 KiB  Total Image Cache Size

Object Storage

   279 MiB  Strings
   259 MiB  Conses
   148 MiB  Vectors
    58 MiB  Intervals
   7.1 MiB  Symbols
   219 KiB  Buffer-Objects
    39 KiB  Floats

Largest Buffers

   186 MiB  *Ement Notifications*
   186 MiB  *Ement Mentions*
   149 MiB  *Ement Room: REDACTED
   148 MiB  *Ement Room: REDACTED
   148 MiB  *Ement Room: REDACTED
   147 MiB  *Ement Room: REDACTED
   147 MiB  *Ement Room: REDACTED
   146 MiB  *Ement Room: REDACTED
   146 MiB  *Ement Room: REDACTED
   146 MiB   *ement-room--format-message*
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED
   145 MiB  *Ement Room: REDACTED

Largest Variables

   354 MiB  ement-syncs
   145 MiB  ement-sessions
   9.2 MiB  package-archive-contents
   7.1 MiB  gnus-dup-hashtb
   6.9 MiB  gnus-dup-list
     4 MiB  gnus-active-hashtb
   1.7 MiB  load-history
   1.4 MiB  org-fold-core--property-symbol-cache
   1.4 MiB  org-persist--associated-buffer-cache
     1 MiB  ucs-normalize-hangul-translation-alist
   813 KiB  nerd-icons/mdicon-alist
   805 KiB  ement-users
   764 KiB  easy-menu-converted-items-table
   630 KiB  info-lookup-cache
   620 KiB  gnus-newsgroup-data
   558 KiB  emoji--names
   556 KiB  face--new-frame-defaults
   430 KiB  modus-themes-faces
   345 KiB  uni-confusable-table
   327 KiB  package--compatibility-table

Expected results

I'd like to be able to keep using Emacs without having to restart it :-).

Backtrace

No response

Etc.

I compile my own Emacs, but I try to stick to upstream's release branch and that's what I've been using on my two main machines. On both, I see the same symptoms.

[alphapapa]

Ement's design is simple: events come in, and they are stored in memory until the session is disconnected, then references to them are (or should be) removed so they can be GC'ed. If your Matrix sessions remain "connected" (in quotes since Matrix's design means a series of repeated connections, rather than a single, constant connection) for weeks or months, then, sure, the memory usage is going to go up as events are received.

You shouldn't need to restart Emacs to clear Ement's memory usage; just disconnect from the sessions you're connected to. Then you could reconnect and only the events received upon initial sync would be in memory.

Also, if you keep room buffers open indefinitely, then as they grow, I suppose it could begin to take longer to insert new events into the buffers--though that generally shouldn't be the case, since insertion of new events happens starting at the bottom, so if it does, it might indicate a bug.

Regardless, rooms with open buffers have events inserted into them as they arrive, which is more work than just receiving the events and storing them in memory, so if you're not actually looking at a bunch of room buffers, it's probably a good idea to kill them and then reopen them when you need them. That would both reduce the time taken to process new events and reduce memory usage.

Someday we might try to use SQLite to cache received events, and that might allow us to more easily discard some subset of received events from memory. But that would be a long-term goal, not likely to happen anytime soon.

Also, I'd guess that much of the data usage reported in those buffers is actually data structures which are shared among them, since the ement-sessions variable is 145 MB, and most of those room buffers are reported as that size. So the data actually being used by Ement itself within Emacs may not be as high it first appears.

Other than that, I don't know why the ement-syncs variable would be much bigger than ement-sessions, but again, I'm pretty sure that much, if not most, of those reported sizes are common data structures. But it's possible that there are some optimizations that could be done somewhere. Specific suggestions are welcome, but it would likely require some careful investigation.

By the way:

I ran memory-report, which took almost 1 hour to gather all data (!!!).

Yeah, last time I ran it, it seemed quite slow, and I eventually C-g'ed it. That command is fairly new in Emacs and it probably needs optimization.

Ah, yes, memory-report is described:

This report is approximate, and will commonly over-count memory
usage by variables, because shared data structures will usually
by counted more than once.

I'd guess that generating the report itself probably uses a not-insignificant amount of memory, as it probably uses hash tables to count unique objects, or something like that. So it probably causes a lot of GC's, which probably makes it slow.

So I'd guess that, in that case, Ement is probably using 300-400 MB of memory at that time (although since ement-sessions is 145 MB, it might be closer to that), which, if you're leaving it "connected" for weeks or months in a lot of rooms, seems expected to me.

[sergiodj]

Thanks for the detailed explanation.

I don't know why I'm restarting Emacs to "solve" this problem; I believe I might have tried restarting Ement but found that it didn't help. Anyway, I'll make sure to try again to see what happens.

I understand that it may take a long time to implement a better way to store events, but thanks for considering it! I don't plan on stop using Ement anytime soon, so I can wait :-).