loaders.gl icon indicating copy to clipboard operation
loaders.gl copied to clipboard

return dbfHeader from shapefile parser

Open nreese opened this issue 3 years ago • 3 comments

This PR updates shapefile parser to return dbfHeader. For my usecase, it is useful to return dbfHeader from parseShapefileInBatches so that dbfHeader.nRecords is available. I am using dbfHeader.nRecords to track stream processing progress and want to avoid having to read header of dbf file in a separate call.

nreese avatar Jan 28 '22 17:01 nreese

@nreese Thanks for the PR. Being able to read the number of records in advance during streaming makes total sense when the loader has that information.

However, I'd prefer to make that a more standardized field that any streaming format that knows the length a priori can use.

Separately I am a little hesitant to expose the DBFHeader directly. A general idea of loaders.gl is that loaders to return standardized data so that applications can be written independently of specific loaders, and thus we generally avoid "slapping" loader specific things onto the output.

In conclusion I'm positive to landing this but give me a day or two to consider alternatives.

ibgreen avatar Jan 28 '22 17:01 ibgreen

Separately I am a little hesitant to expose the DBFHeader directly. A general idea of loaders.gl is that loaders to return standardized data so that applications can be written independently of specific loaders

In this case, the shapefile loader is already returning header with is very specific to shapefiles.

Thanks for taking a look and thinking over the implementation. Loaders is a very useful library. I really appreciate it.

nreese avatar Jan 28 '22 18:01 nreese

Just noting that as @nreese says, the current return format of Shapefiles includes shapefile-specific metadata here: https://github.com/visgl/loaders.gl/blob/39f6ee8f62b130107def832c44a3f9cf4a347b62/modules/shapefile/src/lib/parsers/parse-shapefile.ts#L136-L142

kylebarron avatar Mar 07 '22 23:03 kylebarron

closing, stalled

nreese avatar Jun 12 '23 21:06 nreese