SearchEngine Error - "Too many open files in system"

[dev-bloke]

I'm not sure if what I'm doing with OpenGrok is supported but I thought I'd report this anyway. :-)

I wrote a quick utility class to make repeated calls to SearchEngine.search() and I discovered that after 32 calls or so, an IOException was thrown as follows (edited for brevity)

java.io.FileNotFoundException: /usr/local/var/opengrok/data/index/api/_1.si (Too many open files in system)
    at java.io.RandomAccessFile.open(Native Method)
        ...
    at org.opensolaris.opengrok.search.SearchEngine.searchMultiDatabase(SearchEngine.java:194)
        ...

Trawling through the source distribution, it looks like the IndexReaders opened at the top of this method aren't closed at the end of the search. In my own local build I added a loop at the bottom of the method to close the IndexReaders and this seemed to fix the problem.

[vladak]

great, feel free to submit pull request.

[vladak]

If I am not mistaken the readers are created in searchMultiDatabase() in this loop:

203        for (Project project : root) {
204            IndexReader ireader = (DirectoryReader.open(FSDirectory.open(new File(droot, project.getPath()).toPath())));
205            subreaders[ii++] = ireader;
206        }

and subreaders is local to the method so should be automatically reaped at the end of the call. Maybe GC just was not quick enough.

[czguang]

I also meet this problem when use opengrok(docker) with aosp code. And when increase the value RLIMIT_NOFILE from 1024 to 102400 in file docker/sync.yml, it is ok.

code diff:

➜  ~/sourcecode/github/opengrok (master) ✗ git diff
diff --git a/docker/sync.yml b/docker/sync.yml
index 51fb6de632c..88894f496d2 100644
--- a/docker/sync.yml
+++ b/docker/sync.yml
@@ -20,7 +20,7 @@ commands:
       --connectTimeout, '%API_TIMEOUT%',
       -r, dirbased, -G, -m, '256', --leadingWildCards, 'on',
       -c, /usr/local/bin/ctags, -U, '%URL%', -H, '%PROJECT%']
-    limits: {RLIMIT_NOFILE: 1024}
+    limits: {RLIMIT_NOFILE: 102400}
     args_subst: {"%API_TIMEOUT%": "$API_TIMEOUT"}
 - call:
     uri: '%URL%api/v1/messages?tag=%PROJECT%'