git segment very slow on large repositories

[ghost]

If you cd into a large git repository like https://github.com/freebsd/freebsd-ports it takes 30-+ seconds for the prompt to load on a ssd.

[mexchip]

I cloned the freebsd-ports repo using and old Core 2 laptop, cd into it and ran time git status --porcelain -b, took a bit more than 3 minutes and a half to finish :sweat: (powerline-go and powerline-shell use git status --porcelain -b command to parse git info). Subsequent executions took a lot less.

So, it seems the slowness is because of git itself, not because of the parsing.

[ghost]

Crazy, I'm using the svn checkout now just to avoid this from happening. Not sure theres a good solution to this other than just not using git here.

[breml]

Maybe powerline-go should provide some kind of timeout, how long it waits for a segment to return its value. After this timeout is reached, an alternative segment, which indicated, that the execution took to long, is inserted instead.

[j605]

I am running into a similar problem with a drive mounted with sshfs. A timeout option would be great.

[justjanne]

There have been quite a few performance improvements for the git segment since this issue was opened, could you try this again?

[j605]

I don't know if I can compare but it does feel faster. I can't currently test the sshfs situation but going into the linux repo for the first time was slow (probably due to my HDD) but was fast on the second go. Thanks :)

[klippx]

I have the same issue. I switched from the old powerline shell to this, and for small repos it is very fast 👍

mathias.klippinge  ~  src  SMALL_REPO  master  %  time git status --porcelain -b
## master...origin/master
git status --porcelain -b  0.00s user 0.01s system 77% cpu 0.018 total

But for large repos not so great 😢

 mathias.klippinge  ~  src  LARGE_REPO  master  %  time git status --porcelain -b
## master...origin/master
git status --porcelain -b  0.20s user 28.67s system 829% cpu 3.481 total

My system info:

OS: macOS Catalina 10.15.3 19D76 x86_64
Kernel: 19.3.0
Shell: zsh 5.7.1
Git: 2.62.2

PS: https://github.com/b-ryan/powerline-shell suffers the same issue on large repo so could be a Git issue, like someone suggested.

[justjanne]

I’ve introduced a new solution, you can now set a maximum git index size, if the index is larger than that, powerline-go will avoid checking for untracked files.

The default max is 2MiB, which is a reasonable size (the linux project is at 6.7MiB, and so powerline-go will by default avoid checking for unchanged files there)

Please check if this improves your performance issues :)

[klippx]

I noticed a slight difference, but wasn't sure, so I tried to debug this a bit to present more data.

The code is basically doing git status --porcelain -b --ignore-submodules OR git status --porcelain -b --ignore-submodules -uno

So I tried running bash (without any powerline shell magic) and just getting some statistics by running each command 20 times wrapped around time.

Median of 20 attempts without -uno: 2.62s Median of 20 attempts with -uno: 1.86s

The good news; it is (statistically...?) significantly faster 🎉

Alas, it still means we have a ~2sec between each command entered instead of a ~3sec 🤔

So I wouldn't say it is fixed because for the end user it still feels like Powerline Go is sluggish. Now for the fix, if there is no way for git status to perform <100ms or so I think we might need to throw our hands up and give up completely, or try something completely different.

I checked documentation for git, and decided to try out git config --global core.preloadIndex false

This brings the median(20) down to 0.22s! Without -uno we have 0.37s so your fix is still valuable.

Now we are getting to acceptable performance levels...

There is even more that can be done in userland to improve git status performance: https://github.blog/2018-04-05-git-217-released/#speeding-up-status-with-watchman which is very interesting... But that is beyond the scope of this repo.

Speaking of which (beyond the scope of this repo) in our monorepo I am getting 0.25s with git config change, and 0.75sec without -uno. So I am sure other people working in monorepos will find this discussion interesting.

I will ping back to this thread if I notice the global git config changes I made had unintended side effects.

[AntoineToussaint]

Let me open a can of worms ;) Actually I opened this can of worms myself and it's ugly.

But have you thought of bypassing PS1 completely, forking the shell and display the prompt ncurse style so you could update the prompt in real-time or even async update (so you don't wait for git info), it displays when it can...

Or is there another way to do real-time update on the prompt or async?