Periodically close and reopen an IndexWriter to avoid memory leaks

[Shazwazza]

This is a known issue in Lucene that keeping an IndexWriter open forever with a ton of writes will eventually cause memory to leak/grow.

We need to periodically close and reopen an IndexWriter. RavenDb does this by tracking number of writes and when it reaches a threshold it closes and reopens. We can probably do something similar. See comment https://github.com/Shazwazza/Examine/pull/174#issuecomment-678013901

[nzdev]

Might need to track the reads as well.

Class SearcherManager
Utility class to safely share IndexSearcher instances across multiple threads, while periodically reopening. This class ensures each searcher is disposed only once all threads have finished using it.

Use Acquire() to obtain the current searcher, and Release(G) to release it, like this:

IndexSearcher s = manager.Acquire();
try 
{
    // Do searching, doc retrieval, etc. with s
} 
finally 
{
    manager.Release(s);
    // Do not use s after this!
    s = null;
}
In addition you should periodically call MaybeRefresh(). While it's possible to call this just before running each query, this is discouraged since it penalizes the unlucky queries that do the reopen. It's better to use a separate background thread, that periodically calls MaybeRefresh(). Finally, be sure to call Dispose() once you are done.

[berendhaan]

@Shazwazza Do you have a link to the source of RavenDb where they implement this?

We are running to some memory issues using AzureDirectory and writing a lot of times to the index while keeping the writer open for the lifetime of the application.

[Shazwazza]

@berendhaan are you running a fork of AzureDirectory or is this for Examine 0.1.x? Are you sure this memory issue is because of this? Sorry I don't have the link to ravendb source. I haven't seen any specific examples of this issue in real life.

[berendhaan]

We are using a fork, Lucene.Net.Store.Azure, I guess your implementation is based on this? I was triggered because we do a lot of writes and keeping the index open.

[nzdev]

I believe this can now be closed. With Lucene.NET 4 this is implemented in Examine 3.