Minion Keeping segments on disk after RealtimeToOfflineSegmentsTask

[TT1103]

We noticed high disk usage on the minion servers and it appears that the RealtimeToOfflineSegmentsTask sometimes leaves the segments on disk.

We also noticed that Pinot is keeping a large amount deleted files open, which might be a memory management bug. Example lsof -n | grep deleted output:

java      106586   8183 TaskState        username  634w      REG              252,0   7401250816             5887067 /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-6fe9e9c9-238c-4b99-8018-094f32bbad38/workingDir/mapper_output/0/record.offset (deleted)
java      106586   8183 TaskState        username  635w      REG              252,0 128791597056             5887068 /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-6fe9e9c9-238c-4b99-8018-094f32bbad38/workingDir/mapper_output/0/record.data (deleted)
java      106586   8184 HelixTask        username  634w      REG              252,0   7401250816             5887067 /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-6fe9e9c9-238c-4b99-8018-094f32bbad38/workingDir/mapper_output/0/record.offset (deleted)

see slack thread: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1663342366477189

[Jackie-Jiang]

@klsince Can you help take a look at this issue?

[klsince]

some quick thoughts but need verify: although we do delete the files in a finally{} upon task completion, but we may not release all the mmap over those files, which is done by

// in SegmentProcessorFramework.process(): L145, under the hood, the record.offset/data files are mmap'ed 
GenericRowFileReader fileReader = fileManager.getFileReader();
...
// in SegmentProcessorFramework.process(): L166, but this is not called in finally block
fileManager.cleanUp();

[snleee]

Good catch 👍