dfs-datastores icon indicating copy to clipboard operation
dfs-datastores copied to clipboard

Allow custom file formats (aside from SequenceFileFormat) to pass args

Open baconz opened this issue 11 years ago • 6 comments

I'm working on an AvroFileFormat, and I'd like to be able to pass it args -- this seemed like the least bad way to make it happen. Open to suggestions for other approaches.

baconz avatar Jun 24 '13 02:06 baconz

@sorenmacbeth, all you!

sritchie avatar Jun 24 '13 02:06 sritchie

Looks good at first glance to me. I'll give a more thorough going over soon. Thanks!

sorenmacbeth avatar Jun 26 '13 05:06 sorenmacbeth

I ran into this same issue when trying to create a custom Avro file format that keeps track of the schema in args. +1 for this!

cmerrick avatar Jul 08 '13 03:07 cmerrick

:+1: for the well thought change. @baconz @cmerrick how do you get the arguments inside the serialize and deserialize methods in your implemented PailStructure? I am not able to figure this out.

kul avatar Sep 15 '13 05:09 kul

@baconz Did you get AvroFileFormat working?

I've made a pail structure that serializes and deserializes to from avro but it's stored in a SequenceFile and I'm getting poor performance from the pail tap this way. Thanks

stanfea avatar Apr 09 '15 15:04 stanfea

@steanbacon we never did. More, or less abandoned this project.

baconz avatar Apr 11 '15 13:04 baconz