support using hadoop native methods in local file system

[Jassy1994]

What changes were proposed in this pull request?

support using hadoop native methods while choosing local file system to store sessionMetadata and session nextId, using native methods prevent to create new process. which costs time and may lead to nextId method retain object lock of sessionManager for long time and may cause throughput decreased or create/delete thread hanged

and also a param is suggested to set in livy-env.sh： export LD_LIBRARY_PATH=/usr/local/hadoop-current/lib/native

How was this patch tested?

manual tests and debug

[jerryshao]

using native methods prevent to create new process

@Jassy1994 would you please explain more about this? I was thinking should it be the behavior of Hadoop, not Livy, why do we need to be aware of this in Livy side?

[Jassy1994]

using native methods prevent to create new process

@Jassy1994 would you please explain more about this? I was thinking should it be the behavior of Hadoop, not Livy, why do we need to be aware of this in Livy side?

@jerryshao I find Livy FileSystemStateStore use RawLocalFileSystem when store recoveryMetadata in local. RawLocalFileSystem default use deprecated file status which won't call hadoop native libs, so we need call this static method explicitly to forbid using deprecated file status. of course this issue has relation about hadoop and hadoop native libs should be supplied and can be quoted by livy server.

[jerryshao]

So this happens only when you store metadata in local filesystem, not hdfs, right?