Andy Grove

Thanks for the ping @paddyhoran and I'm excited to see some of the amazing work that @jorgecarleitao is doing here and I would very much like to see this upstreamed...

DataFusion has code for loading parquet into arrow ... might be worth looking at On Thu, Nov 8, 2018 at 4:47 AM Ivan wrote: > I suggest adding an item...

I am very interested in this. I am wondering if we can add a generic reader trait to the main arrow project and then have an implementation in parquet-rs. I...

Actually, maybe this is as simple as implementing `Iterator`

Here is a cleaned up version: ```rust let mut builder : Builder = Builder::with_capacity(batch_size); match r.read_batch(1024, None, None, builder.slice_mut(0, batch_size)) { Ok((count,_)) => { builder.set_len(count); let arrow_array = Array::from(builder.finish()); match...

I've made good progress with integrating parquet-rs with datafusion .. I have examples like this working ```rust let mut ctx = ExecutionContext::local(); let df = ctx.load_parquet("test/data/uk_cities.parquet/part-00000-bdf0c245-d300-4b28-bfdd-0f1f9cb898c4-c000.snappy.parquet").unwrap(); ctx.register("uk_cities", df); // define...

I have merged the current Parquet support to master in DataFusion. I have added examples for both DataFrame and SQL API. https://github.com/datafusion-rs/datafusion-rs/tree/master/examples This is very rough code and I won't...

Thank you @djKooks that would be great

Yes, I think that makes sense.

I would be interested to hear what others think though. @alamb @Dandandan @jorgecarleitao @houqp do you have an opinion on this?