LUCENE-9379: Encrypting Directory - functional but to optimize

[bruno-roustant]

Performance issue because of too many new Ciphers when slicing IndexInput. javax.crypto.Cipher is heavy weight to create and is stateful. I tried a CipherPool, but actually there are many cases where we need to get lots of slices of the IndexInput so we have to create lots of new stateful Cipher. The pool turns out to be a no-go, there are too many Cipher in it.

[Update] Now there is a lighter alternative to Cipher: LightAesCtrEncrypter. It has light constructor, initialization and cloning. And all tests pass.

[rmuir]

I'm -1 to this approach because it breaks the use of the OS cache (I stated this on the JIRA issue, it was ignored, I don't want anyone to forget).

Because we use the os filesystem cache, any encryption must also be done by os (filesystem, block level, etc) so that page caching works correctly.

[dsmiley]

Lets use JIRA ( LUCENE-9379 ) to discuss higher level concerns and GitHub code reviews for code details.

[martin-huber]

@bruno-roustant we tried to integrate your FileEncryptingDirectory in our use case. We before already did own attempts for a similar goal and in order to test it thoroughly we subclassed the org.apache.lucene.store.BaseDirectoryTestCase and gave an overwritten getDirectory() to instantiate the FileEncryptingDirectory.

Doing this, 2 tests were failing (not all the time, but most of the time):

1. testThreadSafetyInListAll
2. testSliceOfSlice

Maybe you have an idea how to fix this ?