dfs-datastores icon indicating copy to clipboard operation
dfs-datastores copied to clipboard

PailRecordWriter should bound number of open files

Open derrickburns opened this issue 11 years ago • 0 comments

Right now, a PailRecordWriter can open an unlimited number of files. Instead of using a HashMap to contain the mapping of attributes to open files, PailRecordWriter should use a LinkedHashMap and close the eldestEntry when the number of open files exceeds a constant, say 128.

Otherwise, for some bad cases, the number of open files can exceed the per process limit, resulting in an exception.

derrickburns avatar Aug 07 '13 04:08 derrickburns