Deserialization concurrent issue caused by `cachedBuffer`

[jealous]

Hi there,

I found a concurrent issue in FSTInputStream. Here is my scenario: Basically, I am trying to do a merge sort from 4 streams. And I am doing the merge with a single thread. The reason is simple, only one thread is responsible for the sorting and writing output. In this task, in order to read input from those streams, I need to open 4 input streams. Note those 4 input streams stay in the same thread so they share a critical member in FSTInputStream:

    public static ThreadLocal<byte[]> cachedBuffer = new ThreadLocal<byte[]>();

During the merge sort, I am reading from those streams in a random order multiple times(depends on my buffer size). And as you notice, the buffer used for input is polluted when you switch the stream to read in the same thread. When the deserailizer got confused, it reports this error:

2019-05-18 17:09:35.789 INFO  [dag-scheduler-event-loop] o.a.s.s.DAGScheduler : ShuffleMapStage 1 
(map at SplashUnsafeSorterTest.scala:93) failed in 0.396 s due to Job aborted due to stage failure: Task 0 
in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): 
java.io.EOFException: read FST stream failed: java.io.IOException: Failed to read the next byte
    at org.apache.spark.serializer.FstDeserializationStream.readObject(FstSerializer.scala:79)
    at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:158)

It took me days to find out the root cause... Is there any way to work around this issue?

[chrisco484]

I'm seeing a similar "Failed to read the next byte" issue under highly concurrent load testing (and using G1GC with String deduplication.

It might be that FST is getting "deeply involved" with String internals and under String Deduplication the JVM does some behind the scenes manipulation of the String internals to perform the "dedupe" magic.

Perhaps FST assumes the internals are immutable, which they are when "dedupe" is off but not when "dedupe" is on.

Maybe there needs to be an option that causes FST to deal with Strings at only a higher level so that it avoids issues when internals of Strings are changed behind its back by the JVM.