FLUME-3190: flume shutdown hook issue when both hbase and hdfs sink a…

[mcsanady]

When both hdfs and hbase sink are in use, during shutdown (KILL SIGTERM), the hdfs sink won't be able to rename/close the .tmp hdfs file because the underlying filesystem could be closed earlier when shutting down the other component.

This change registers a new ShutdownHook in the hadoop's ShutdownHookManager, which will prevent other hooks to run until Flume stops itself.

Tested on a cluster which could reproduce the error before the change, but eliminated after.

[majorendre]

Hi @mcsanady,

Thanks for your contribution. I have some questions: Why does it matter if we use HBaseSink? How could we test this the simplest way?

Thanks, Endre

[mcsanady]

Please ask @adenes, he continues all my works at Cloudera.

[asfgit]

Can one of the admins verify this patch?