spatial-framework-for-hadoop icon indicating copy to clipboard operation
spatial-framework-for-hadoop copied to clipboard

Split Enclosed JSON

Open randallwhitman opened this issue 11 years ago • 1 comments

Currently, Enclosed JSON - i.e. a file of complete valid Esri REST JSON - is treated as non-splittable; in other words, if used as an input to a MapReduce job, the whole file will be processed by a single Mapper.

It would be possible to skip the opening enclosing brace and the metadata section outside the "features" section, and otherwise the RecordReader could be similar to UnenclosedJsonRecordReader. In fact, if the Enclosed JSON is restricted to the convention of the metadata preceding the "features", likely only small differences from UnenclosedJsonRecordReader would suffice.

Collaborators do not have information as to how much Enclosed JSON is used as MapReduce input. The idea will be set aside, and can potentially pursued if use cases are presented.

randallwhitman avatar Nov 18 '14 19:11 randallwhitman

@komamitsu FYI, in case this is of interest.

randallwhitman avatar Aug 11 '15 18:08 randallwhitman