elephant-bird icon indicating copy to clipboard operation
elephant-bird copied to clipboard

uses FileSystem's listStatus rather than FileInputFormat's in LzoInputFormat

Open nellore opened this issue 9 years ago • 3 comments

Proposed resolution to #426, which describes how FileInputFormat's listStatus is slow on S3 for input paths spanning many files.

nellore avatar Dec 13 '14 20:12 nellore

@rangadi , @isnotinvain this seems like it's been around a while, discussed on its issue a bit. Any thoughts? should it get merged?

ianoc avatar Mar 03 '15 22:03 ianoc

@gerashegalov commend on #426 that this can break compatibility since does not handle globs.

Also, I am not sure why EB traverse the directory tree again, since super.listStatus is supposed to have done that already.

rangadi avatar Mar 04 '15 08:03 rangadi

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Jul 18 '19 15:07 CLAassistant