Clone fails for Postgresql

[cshorler]

For git://git.postgresql.org/git/postgresql.git

When performing a Clone in AGit either from the Device (HTC Desire) or in the Emulator it fails.

Two different errors are reported (but possibly the same cause) - the device raises an IOException in jgit (I don't have a stack trace for this - determined by inspection):

org.eclipse.jgit.transport.PacketLineIn.java readLength() throws this exception:

throw new IOException(MessageFormat.format(JGitText.get().invalidPacketLineHeader,
                    "" + (char) lineBuffer[0] + (char) lineBuffer[1]
                    + (char) lineBuffer[2] + (char) lineBuffer[3]));

And In the Emulator I get a OutOfMemoryError Exception.

Out of memory on a 2323-byte allocation.
 "pool-1-thread-1" prio=5 tid=9 RUNNABLE
   | group="main" sCount=0 dsCount=0 obj=0x405f4ad8 self=0x21d7b0
   | sysTid=986 nice=0 sched=0/0 cgrp=default handle=1998960
   | schedstat=( 7895720724522 2112247922849 217808 )
   at org.eclipse.jgit.transport.PackParser.inflateAndReturn(PackParser.java:~1436)
   at org.eclipse.jgit.transport.PackParser.whole(PackParser.java:945)
   at org.eclipse.jgit.transport.PackParser.indexOneObject(PackParser.java:866)
   at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:469)
   at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:178)
   at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:431)
   at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:672)
   at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:284)
   at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:229)
   at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:225)
   at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:151)
   at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:113)
   at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1062)
   at com.madgag.agit.GitFetchService.fetch(GitFetchService.java:57)
   at com.madgag.agit.operations.Clone.execute(Clone.java:75)
   at com.madgag.agit.operations.GitOperation.executeAndRecordThread(GitOperation.java:38)
   at com.madgag.agit.operations.GitOperationExecutor.call(GitOperationExecutor.java:28)
   at com.madgag.agit.operations.GitAsyncTask.call(GitAsyncTask.java:74)
   at com.madgag.agit.operations.GitAsyncTask.call(GitAsyncTask.java:38)
   at roboguice.util.SafeAsyncTask$Task.doCall(SafeAsyncTask.java:191)
   at roboguice.util.RoboAsyncTask$RoboTask.doCall(RoboAsyncTask.java:54)
   at roboguice.util.SafeAsyncTask$Task.call(SafeAsyncTask.java:165)
   at roboguice.util.SafeAsyncTask$Task.call(SafeAsyncTask.java:153)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:306)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1088)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:581)
   at java.lang.Thread.run(Thread.java:1019)

Given the size of Postgresql - maybe it's all related to this? https://bugs.eclipse.org/bugs/show_bug.cgi?id=312868

I'm yet to run this code on the device, but I expect I'd get similar results. Also - Please note, I used your GitHub JGit branch 'unofficial-roberto-release' in the emulator

[rtyley]

Hi @cshorier - memory issues are obviously a big problem on constrained systems like Android devices - as you can see from my later comment on that JGit bug:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=312868#c4

-I contributed a patch last december which reduced mem usage in certain circumstances - in fact, in the course of developing Agit I've submitted quite a few memory-related patches to JGit! :

http://egit.eclipse.org/r/#dashboard,102

The JGit team have been great about accepting and/or improving on these patches, so both Agit and the JGit project in general have benefited from this work. Unfortunately there are still lots of cases where memory usage is still high, and in some cases (e.g. delta resolution) it's unavoidable, pretty much inherent in the design of Git.

I was able to reproduce the memory error, cloning postgresql on my HTC Desire:

E/dalvikvm-heap( 2006): Out of memory on a 2454-byte allocation. I/dalvikvm( 2006): "pool-1-thread-2" prio=5 tid=10 RUNNABLE JIT I/dalvikvm( 2006): | group="main" sCount=0 dsCount=0 obj=0x4073b3f0 self=0x248bd0 I/dalvikvm( 2006): | sysTid=2151 nice=0 sched=0/0 cgrp=default handle=1927856 I/dalvikvm( 2006): | schedstat=( 138523651283 68830962992 167545 ) I/dalvikvm( 2006): at org.eclipse.jgit.transport.PackParser.inflateAndReturn(PackParser.java:~1436) I/dalvikvm( 2006): at org.eclipse.jgit.transport.PackParser.whole(PackParser.java:945) I/dalvikvm( 2006): at org.eclipse.jgit.transport.PackParser.indexOneObject(PackParser.java:866) I/dalvikvm( 2006): at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:469) I/dalvikvm( 2006): at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:178) I/dalvikvm( 2006): at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:431) I/dalvikvm( 2006): at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:672)

Cloning postgresql is taking too much memory and it's a little interesting where the exception occurred, because it wouldn't have been on a blob object (the usual candidate for a memory hog) - it would have been a commit, tree (ie directory listing) or a tag - a relatively small object, and yet allocating the 2KB byte array to hold it was enough to blow the top off the memory - there must have been a lot of memory already committed.

I've got a couple of ideas about possible avenues for improvement, I'll see if I can take a look over the next few days - thanks for raising the bug!

[cshorler]

I agree! When I stopped the thread after writing my initial post last night - the higher level parse function is iterating over about 470000 objects. Presumably each of these is expanded and some of them require a byte[] allocation.

I suppose there are two things to consider - should the code be refactored to a streaming interface, are all these objects actually required (is this retrieving the entire commit history?)

Chris