logging-flume icon indicating copy to clipboard operation
logging-flume copied to clipboard
verified publisher icon apache

FLUME-3325 fixed TaildirSource oom issue

Open belveyf opened this issue 7 years ago • 4 comments

when tailing a lot of files where some file is writing frequently but the others are idle can cause the idleInodes List increasing infinitely, change it to set(list backend) can fix this problem.

belveyf avatar Mar 26 '19 15:03 belveyf

Hi @oldbelvey,

Thank you for this PR! The change looks good, but can you please explain why the change resolves the issue described in the title?

bessbd avatar Dec 07 '20 09:12 bessbd

Hi @oldbelvey,

Thank you for this PR! The change looks good, but can you please explain why the change resolves the issue described in the title?

Hello bessbd, the oom problem is caused by repeatedly add same value into the list. changing list to set just preventing from adding repeated values. And it is described more specificly at https://issues.apache.org/jira/browse/FLUME-3325

belveyf avatar Dec 11 '20 09:12 belveyf

@oldbelvey : can you please include some directions to prove this change resolves the issue? Ie. how can I reproduce the original issue and see if it happens before applying this change and that it doesn't after applying it?

bessbd avatar Dec 15 '20 08:12 bessbd

@bessbd I don't think it's easy to reproduce. Tho OOM happens at our production environment, where one flume agent monitored more than fifteen app's log. And as it is described at the JIRA, with one file frequently write while they others are idle, the oom will happen right after the Arraylist boom.

belveyf avatar Dec 17 '20 08:12 belveyf