FLUME-3217 . Creates empty files when quota

[ffernandez92]

As it can be seen in FLUME-3217, Flume creates empty files when HDFS quota has been reached.

This is a first approach to the solution. It works although a better approach it could be implemented. The idea is to capture the quota Exception in order to delete this annoying empty files generated by flume.

[asfgit]

Can one of the admins verify this patch?

[ffernandez92]

Hi @szaboferee I have reproduced the environment and you are right. I could happen that if the quota is reached in the middle of a “transaction”, the all file dumps.

It is necessary to check if the file weight is zero before doing any delete operation. Moreover it will be necessary to implement some sort of “stopper” in the “append” function of the BucketWriter class in order to achieve the expected behavior.

I will try to do the modifications asap. Thank you and sorry for the stupid mistake.

[szaboferee]

@ffernandez92 can you add a test with a minicluster? It would make the review much easier

[ffernandez92]

Hi @szaboferee sure, I will add a test with minicluster. Now I am uploading a new version as I am having some trouble with test part in local.

[majorendre]

@ffernandez92 thanks for the contribution.

I guess this problem can be quite painful. For performance reasons I would go another way to fix this problem. I would add a check to the close() function. It would check after successful close if the file length is 0. That is simple to implement and would happen only after a close. If the close was not called from a catch block, we could omit this for further speed. I guess, still there could be a problem when the quota is exceeded, Flume can get into a sort of infinite loop as it tries to write the same event to HDFS again and again. Some sort off backoff should be implemented to reduce load. Needs to be checked if it is true.

Let me know please, what you think.

[ffernandez92]

Hi!

Sorry for the late answer I’ve been a little bit under a rush lately. I think you are right @szaboferee about the performance degradation. I think that the solution you are proposing @majorendre it was also my first approach. But it was also problematic because of what you were saying about the infinite loop when quota was reached in the middle of a write.

I will try to reformulate the solution trying both approaches to see which one would be better.