Unison does not free memory from sync (but reuses it on repeat, so not quite a leak)

[gdt]

There are multiple reports of excessive memory use, and the situation is confusing.

This bug is limited to the situation where syncing allocates memory, and it is not freed at end of sync, but that memory is reused when the next sync happens.

This bug excludes fswatcher-related issues (see #329) and is probably different from #33

[tleedjarv]

I'm slowly investigating memory consumption issues. I'm still very early in the process but I thought I'd share an update on the progress.

I don't know OCaml or its GC well enough to speculate why memory is not being freed. This seems to be happening though, and in this environment, it is the recursive functions that create a lot of temporary copies of same data that drive the excessive memory use.

So far, I have nailed down one of such functions. Uitext.interact, or more precisely, the selectAction invocation within. https://github.com/bcpierce00/unison/blob/v2.51.3_rc2/src/uitext.ml#L401 I don't know if it is the selectAction function that itself is causing excessive memory usage or if it is the parameters of invocation, but it seems to be the latter. Either way, the list of actions created as one of the parameters is huge.

The thing is, the entire interact function is intended for interactive use, or selectAction the very least. But currently, interact (and thereby selectAction) is executed even in batch mode.

I've run tests with selectAction bypassed in batch mode (still executing interact as this logs all the reconciliation results) and results are as follows (don't pay any attention to the absolute numbers, just the difference):

• number of updates: 100 000
• mem usage, unchanged code: 584M
• mem usage with the tiny change: 128M
• functional impact: none

I also ran the tests with selectAction invocation present, but list of actions parameter limited to only the default action. Memory consumption was slightly higher (135M, if I remember correctly). With every added action to the list, the memory usage increases somewhat.

The change I made is this simple:

diff --git a/src/uitext.ml b/src/uitext.ml
index 811a4a5..dda4f42 100644
--- a/src/uitext.ml
+++ b/src/uitext.ml
@@ -398,6 +398,7 @@ let interact prilist rilist =
                 if not (Prefs.read Trace.terse) then
                   displayDetails ri
               end;
+              if Prefs.read Globals.batch then next() else
               selectAction
                 (if Prefs.read Globals.batch then Some " " else None)
                 [((if (isConflict dir) && not (Prefs.read Globals.batch)

This is not really a fix, more of a workaround, as memory is still not freed as it should be. Just now this is much less noticable because overall much less memory is allocated to begin with.

[tleedjarv]

As I've learned more about the code I now have a better understanding of what is going on.

Actual memory leaks are very few or none (mostly in form of unreleased caches, some fixed now).

Aside from GC characteristics, the real (and maybe the only) issue of not releasing memory seems to be caches. There are several caches in the code that are not released (some intentionally, some due to bugs (some fixed now)). This also explains why the memory is eventually reused. The question of when (and whether) to release various caches is not straightforward.