Fame gets stuck - pipe blocked?

[rwest]

My RMG job has been frozen for eleven hours waiting for a fame calculation to complete. When I run the fame input stand-alone it takes maybe ten seconds. The output, however, is very long (tens of megabytes). I expect the problem is with the BufferedReader or InputStream used to get the fame output into the java code. If the buffer is not emptied, then the pipe essentially becomes jammed up and fame cannot write to it. I think we read the buffer into a string, appending to the string line by line. Is this stupid? How does java cope with adding a line onto a 12mb string thousands of times in a row? Presumably the string has to keep on being moved around in memory? Would a linked list of strings be better? Or process the results as they come in rather than storing them all up in a string?

[rwest]

Josh tried to help with 36da5c574fa7fca5ffefc0402df15e433a59b189 but it hasn't solved the problem.

[rwest]

I think I may be on to something.... http://www.yoda.arachsys.com/java/strings.html

[rwest]

Use StringBuffer for string concatenation when reading FAME output.

Strings are immutable. Adding a line onto the end of a string is inefficient, and doing it many many thousands of times is a particularly bad idea. Use a StringBuffer instead. See http://www.yoda.arachsys.com/java/strings.html

Hopefully this closed by d012a9130cb683f7629c73bba7c2e8c821ba0bd8 in which a fame result took about a day to be read in to a string.

Edit: StringBuilder is more efficient for single-threaded scenarios

[rwest]

only thing to be wary of with StringBuffers is that toString() can waste memory http://fishbowl.pastiche.org/2005/04/27/the_string_memory_gotcha/ (but no more than the original StringBuffer)

[rwest]

In da2ca3d8761807a32c3e2e95a1461330b4160373 I changed it further, so that we process the results directly from the buffer, instead of storing it in a StringBuilder first.

[rwest]

I have just re-opened this bug. Although the recent changes have helped, it still gets stuck somewhere. Previously I got to 19 core species, this time I got to 45. However, it still seems to be stuck on fame. The biggest fame input is this one http://drop.io/u8kaelj and the output is some 41 megabytes. It takes less than a minute for fame to run, but the RMG job got stuck for 16 hours (before being killed by the job scheduler on pharos)