Matthew Turner
Matthew Turner
@yahoNanJing fyi @seddonm1 and I have been working on https://github.com/datafusion-contrib/datafusion-objectstore-s3. Still early stages but would be great to have more use cases that we could steer development with.
@yahoNanJing there actually already is a hdfs extension here https://github.com/datafusion-contrib/datafusion-hdfs-native. Understood on your point - in that case what would be your ideal API for accessing S3? I would have...
ill play with these flags locally and keep you posted on impact
@Dandandan I've done the following to build the wheel: ``` export RUSTFLAGS='-C target-feature=+fxsr,+sse,+sse2,+sse3,+ssse3,+sse4.1+sse4.2,+popcnt,+aes,+avx,+avx2' && maturin build --release ``` Then i just reinstalled the wheel and reran the benchmark which produced...
@realno FYI
below is with native and `sn-malloc` - some faster, some slower. roughly in line. ``` q1: 0.05099512500000003 q2: 0.3307659999999999 q3: 1.228696541 q4: 0.062102542000000316 q5: 1.2268319589999996 q6: 1.2571589580000002 q7: 1.1611415420000002 q8:...
Cool - that was what it looked like to me as well from my scan of the code. IMHO in the medium term it would be nice to have pyarrow...
@andygrove only concern that i have with this is if in the future you were unable to contribute as much to the python bindings that the maintenance burden would fall...
Asides from repr, do you see any other advantages?
@seddonm1 @yjshen @houqp FYI - in case you have thoughts on this.