Poor server responsivness with big commits

[TomaszSzt]

Recently I did observe a very unpleasant issue. I have a certain repository which was initialized with a single 16'000 files commit with few binary files of total about 1GB checked out (git repo size ~500MB). This repository works fine when it comes to browsing file tree, cloning, pushing and etc. It does however "freeze" the server when a first commit is to be shown, by. ie.

http://xxxx/commit/xxxx.git/71c07836f21bc42b0fc9999c1117be1d6e147307

The First attempt to load commit page after server reset puts server on hold for about 2 minutes. Second attempt (page reload) NEVER (I did await about half an hour) finishes and after few minutes server stops responding on both web and git access channels (web pages not loading, git pull not responding). The only solution I figured out is to kill it and restart.

What I can observe is that CPU at server is running at 100%

Tested it with 1.8.0 and Ubuntu machine (my server) and 1.9.1 on Windows 10 machine (my desktop).

The windows machine have shown in log at page reload the java.lag.outOfMemoryError and returned to functionality after about one minute while the Ubuntu machine just have frozen. The Windows machine has 10 times more memory and it is about ten times faster than Ubuntu machine.

I suppose that the culprit is the moment when commit page is computing "diffs" to show in commit log, which account to 5095165 changes total and includes all the binary file. The "out of memory" error on Windows suggests, that the computation flow is like: "load all models for all repos and THEN extract necessary information" instead of "load model for repo, extract, repeat for next repo". Inspecting source code suggests "getFilesInCommit" in JGitUtils.java.

I will try to work-around it by assigning more memory to Java that it would be otherwise a reasonable guess and try to re-write repo history to not contain such huge commits.

[TomaszSzt]

It is stupid to answer to oneself, but after downloading the source, compiling, adding some detailed logging and fixing most obvious problems I could have pin pointed that:

1. Computation time of such a large commit diff is responsible for about half of the time spent. Remaining time is spent somewhere inside a wicket. I do not know where exactly now.
2. The Wicket page model is build with DataView in CommitPage seems to be retained in memory and due to it's structure is bloated up to tremendous size. The 16k files model uses up 45MB. Amazing requirement for 300kB html page.... Few fast page reloads do overflow the cache it seems. I am surprised it is not held with SoftReference, but this is wicket 1.4 we are talking about while there is version 9 out in the world.
3. After few OutOfMemory errors something breaks, what is not a surprise, and server is getting stuck.

Since it seems to be a conjoined problem with a peculiar commit, wicket ways of building and retaining pages and advertised "lightweightness" of gitblit resulted in my choice of a server it seems that I will just add additional option to hard-limit number of files to be shown in commit page. 16 thousand lines long page is rather not helpful tool in manually finding "what have changed".

[flaix]

Might this be in any way related to #1011 by any chance?

[TomaszSzt]

Might this be in any way related to #1011 by any chance?

Not. 100%. I currently figured out, that CommitPage.java is:

1. Taking quite a lot time to build commits model. That's would be not a problem, server may "pause" for a moment. This is the: List<PathChangeModel> paths = JGitUtils.getFilesInCommit(r, c) line in that file. Computing diffs stats is a main time factor.

2. Then CommitPage constructor finishes immediately.

3. Next, longer pause is related to something else happening. Since the only place related to commit size is: DataView<PathChangeModel> pathsView = new DataView<PathChangeModel>("changedPath", pathsDp) {... in the same file I suspect this is a source of slow response. I did not verified it Yet tough.

4.Then using visualVM I was able to detect, that there are two or three live instances of DataView which do consume a lot of RAM. For some reasons they are held by strong reference.

5.After hitting OutOfMemory once or twice something breaks and sever no longer responds. The way it breaks is unpredictable. Sometimes just web, sometimes at the git server level too. Sometimes it just continuously spits out OutOfMemory even though memory is empty. Sometimes it can be stopped, sometimes it must be killed.

6.The heap limit is set to 256MB, what I expected to be far above requirements and is default for JDK8. Surely moving it up would push the problem away, but this is just a partial solution I do not like.

I will investigate it a bit more (arm code with more logging at TRACE level) and limit the number of files to be shown on commit page. When I will be ready I will try to do a "pull request" or something like that. I still need to learn it. Some hints will be valuable here.

[TomaszSzt]

Ok, I have changed it to my liking.

I created a pull request. Hope it is visible for inspection. I will deploy it to my server soon.

[TomaszSzt]

The same problem still appears on compare commits (http://host/compare/repo) page. This is however acceptable for me, since displaying of this page requires intentional and not easy action from users, while commit page is shown at the moment when user wishes to browse a specified commit in repository history without an intend to check history details.