Out of Memory Error when `mmap`ing local block file to read

[dbw9580]

Alluxio Version: current master

Describe the bug Alluxio Worker crashes with a core dump saying:

Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.

It appears JVM was trying to create a new thread but mmap limit has been hit. The vm.max_map_count (link to doc) kernel parameter is set to 100000, and the worker process has already created about 100,000 mappings, so further attempts to create a new mapping fails.

This is caused by call to the FileChannel.map method in LocalFileBlockReader:

https://github.com/Alluxio/alluxio/blob/8f74e6584b6d2d71be3a3737057808b6f77c44ae/core/common/src/main/java/alluxio/worker/block/io/LocalFileBlockReader.java#L96-L101

To Reproduce

1. set up worker storage with tiered block store
2. set alluxio.worker.network.reader.buffer.pooled to false, so that alluxio.worker.block.io.LocalFileBlockReader#read is used when reading data from block file
3. run StressWorkerBench with enough stress, say 4x256 threads.

Expected behavior The worker should not crash.

Urgency Medium

Are you planning to fix it Yes

Additional context Worker JVM error dump: FileNum-30-HeapSize-16.log.zip