Gitu slow in large repos

[lyallcooper]

I mainly work in a large git repo (on the order of 1 million commits), and gitu takes ~11 seconds to start on my fast M3 Max MacBook Pro. Any time the main screen needs to be shown it takes about this long to re-render it.

This is likely because of all of the info gitu is fetching by default, fetching untracked files, stashes, recent commits, etc. However 11 seconds is still slower than fetching that info manually would take with git via the command line, so there's probably some additional inefficiency somewhere.

Some of this slowness is unavoidable in a large repo, but some of it can be mitigated through optimizations or configuration options. For example, having an option to not show untracked files could speed things up. For comparison, magit lets one control which sections are inserted on the status screen, and provides a verbose mode that shows how long each section took to insert. This lets one omit especially slow or less useful sections to get a more usable experience in a large repo.

[altsem]

There's likely different things that definitely could be optimized, but I'm surprised just opening city takes such a long time.

I think it should ignore untracked files if configured via git.

Does turning off the file watcher make a difference?

Ideally, I think information should be loaded more lazily.

[lyallcooper]

Does turning off the file watcher make a difference?

How does one do that? Haven't been able to find anything about configuring that in the help or docs

[mWalrus]

The below flamegraph is from a run of gitu with debuginfo on a large project that I've had loading time issues with.

There seems to be two areas of major hang:

1. From starting gitu until the first frame appears
2. From the first frame appearing until input is registered.

You can see the two sections in the below flamegraph and it looks like both of them are pretty similar. They seem to experience slow downs in the closure passed to gitu::screen::status::create. Not sure what is causing it exactly but I wanted to share this at least :)

[altsem]

@lyallcooper this is the option: https://github.com/altsem/gitu/blob/6b8f359db0ba267dbe7e64b8e9036b5b64e33ebf/src/default_config.toml#L11

Config files are loaded from: Linux: ~/.config/gitu/config.toml macOS: ~/.config/gitu/config.toml Windows: %USERPROFILE%\AppData\Roaming\gitu\config.toml

@mWalrus That's interesting, the "similar" lib is only used to produce inline diff highlights. So I would assume there were a lot of diffs going on? Especially since Gitu would compute an internal representation of all of these before rendering anything.

[lyallcooper]

Ah whoops, missed that (was searching for something about the watcher). Just tried with it disabled and didn't seem to make a difference.

[altsem]

I'm not quite able to reproduce the results you are seeing @mWalrus. Is it the result of having a lot of diff on screen? Could it be that your files have really long lines?

It seems that you've found some edge-case in how the lib 'similar' does diffing.

[mWalrus]

@altsem sorry for the late reply! I wont be able to share the repo I'm experiencing issues with since it's for work unfortunately 😅 I'll investigate further today and report back!

[altsem]

I wanted to swap out "similar" for the "imara-diff" lib, which supposedly performs better. But it involves some work.

[wukkuan]

Takes a minute or two to load the monorepo we have at my work. UI is fast once it loads, but any action takes another minute or two. It displays "Running: git apply --cached --reverse" while it's stuck when unstaging a chunk, for example.

[altsem]

Progress being made in https://github.com/altsem/gitu/pull/392 This should speed things up by deferring highlighting of hunks until they are about to be rendered.

[mWalrus]

@altsem, From some preliminary testing on my end, #392 does seem to have removed the hang after the first frame of the UI is rendered, which makes sense as we deferred hunk rendering. However, gitu still hangs for about 2 seconds (on my machine ™ ) from launch to the first frame being rendered. My initial guess is that it might be the diff parsing that is the culprit as I still have some lingering perf.data files that are pretty sizeable laying around the repo from the last time I ran flamegraph and removing those seem to improve load times.

[lyallcooper]

Just tried with the latest version, and it does seem to have improved things a bit in my large repo, but it's still not really in the usably fast range unfortunately—it's about 37% faster than before at around 7 seconds from launch to first render.

[wukkuan]

gitu 0.34.0 took about 27s for the app to become responsive in our monorepo (~500k commits, ~400k files, ~90m lines). Similar amount of time to do any operation (like staging a file).

git status, by comparison, takes 11s cold, but about 1.2s warm.

[altsem]

I found a pretty severe case of too much allocations happening now, thanks to Heaptrack. There's a fix in master with c9775d7a10ea872b4ff668e9cf15127ad0ea0e48. The benchmark is showing a 40x speedup. Would be interesting to hear how this would affect large repos.

[altsem]

There is still the case that the file-watcher is slow and allocating a bunch on startup. Having it enabled adds ~1s for me before the first render, turning it off makes it instant.

(orange/red column being number of allocations when I startup Gitu in its own repo)

[wukkuan]

Cool, thanks for the update! Doesn't seem to have changed anything for our monorepo. Ran it a few times, varied from 28-32s. Here's a log from --log:

❯ gitu --version
gitu fbb5a86
❯ gitu --log # i hit `q` after it rendered the ui
❯ cat gitu.log
[00:00:00.000] (1f2af2140) DEBUG  Initializing terminal backend
[00:00:00.001] (1f2af2140) DEBUG  Starting app
[00:00:00.001] (1f2af2140) DEBUG  Finding git dir
[00:00:00.134] (1f2af2140) DEBUG  Opening repo
[00:00:00.143] (1f2af2140) INFO   No config file at "/Users/blasko/.config/gitu/config.toml"
[00:00:28.215] (1f2af2140) WARN   Couldn't read /Users/blasko/web-code/.git/rebase-merge/onto, due to No such file or directory (os error 2)
[00:00:28.216] (1f2af2140) WARN   Couldn't read /Users/blasko/web-code/.git/MERGE_HEAD, due to No such file or directory (os error 2)
[00:00:28.216] (1f2af2140) WARN   Couldn't read /Users/blasko/web-code/.git/REVERT_HEAD, due to No such file or directory (os error 2)
[00:00:29.435] (1f2af2140) DEBUG  Opening repo
[00:00:29.439] (1f2af2140) INFO   File watcher started (kind: Fsevent)

[altsem]

refresh_on_file_change.enabled = false

Right, @wukkuan do you experience slowness with this option set to false as well? You'd need to refresh manually (g) with it turned off.

[wukkuan]

@altsem the slowness only happens on start (black screen), if I make any change in the app (like staging a file), or if I edit a file. Any of those causes gitu to hang for about 30s.

Setting that option to false gets rid of the last issue with external file edits causing gitu to hang up.

I'm happy to do a debug session or build from a debug branch if it would help. If you want more real time communication than GitHub you can email me and we can figure that out, [email protected].

[altsem]

I cloned down Firefox's repo and experienced some slowness too. I also tried Gitui with similar results. It seems repo.statuses in libgit2 is the culprit (seen in my recent master commit).

A workaround could be to set that config option: git status.showUntrackedFiles false. Gitu did pass this down to libgit2, so it was respected. Yet the function call was still slow.

I'm curious if there's some other git client we could take inspiration from to make it faster.

[wukkuan]

Yep, gitui is slow for me as well.

lazygit (https://github.com/jesseduffield/lazygit) and magit in eMacs are both fast.

[lyallcooper]

I just tried with the latest master (v0.34.0-41-g715b739) in my large repo and yeah it's still slow for me too. And I can confirm both magit and lazygit are much faster.

[altsem]

I noted that lazygit looks to be forking out a new process when calling git. Instead of relying on libgit2. Gitu has a mix of these approaches, but I've been moving towards forking due to other reasons.

I reverted back to doing a similar approach on https://github.com/altsem/gitu/pull/433. Also changed the behaviour of screen refreshes which makes navigation back from nested screens faster.

Time to first render in the Git repo of Firefox (time gitu -k q)

status.showUntrackedFiles	refresh_on_file_change.enabled	Command Time
true	true	2.62 s
true	false	1.00 s
false	true	0.43 s
false	false	0.42 s
git status (baseline)	—	0.51 s

I think this will play better with any optimizations one might already be using too. I doubt libgit2 respects all settings of git.

[wukkuan]

Last night I built with cargo install --git https://github.com/altsem/gitu.git --locked, from f790145.

% time gitu -k q
13.87s user 169.97s system 33% cpu 9:08.07 total

Not sure what happened there to make it take 9m to finish. :-\

[altsem]

@wukkuan damn. Going to need some way to reproduce this or more details in order to fix that. Is there some other public repo where you experience the same? Are you running out of memory? Perhaps it's OS-specific? What kind of hardware do you have?

In case you want to do some profiling, I added make targets to more easily get started. You'd need the tools installed already (cargo install flamegraph / on fedora: dnf install heaptrack)

flamegraph:
        cargo flamegraph --profile profiling --bin gitu

heaptrack:
        cargo build --profile profiling
        heaptrack target/profiling/gitu

e.g.

git clone [email protected]:altsem/gitu.git
cd gitu
make flamegraph && firefox flamegraph.svg
# or
make heaptrack

[wukkuan]

To answer your questions directly: I'm using a shared debian linux instance at work with hundreds of gigs of memory that is very fast. The app seems to sit at around half a gig of memory used while it's running. I tried testing with the linux and chromium repos, but gitu was fast for both. I wasn't able to get the flamegraph/heaptrack tools to run, unfortunately.

However, I think I found the cause. It seems to be related to untracked files. I tried cleaning all my build assets with git clean -xdf and gitu started being around the same speed as other tools. Then I regenerated the assets and it was slow again (minutes to run). Then I disabled status.showUntrackedFiles to false and it was reasonably fast again.

I tried generating a bunch of files in another repo on another machine and I just can't reproduce it. :( Not sure if you have any further ideas?

I'm curious, do you experience this new extra-slowness or did the latest change help you @lyallcooper ?

[lyallcooper]

Sorry I don’t have access to my machine right now so can’t test, but I’ll give it a shot when I’m back in a few days time.