Andrew Lamb
Andrew Lamb
Thnks @a-agmon -- maybe this example would help: https://docs.rs/datafusion/latest/datafusion/catalog/trait.AsyncSchemaProvider.html I agree the trick will be figuring out how to async calls.
> I can see that ListingTableUrl::parse supports glob strings, so does it make sense to simply implement this as a listing table? Yes this is what I would expect --...
> Hi [@comphead](https://github.com/comphead) and [@alamb](https://github.com/alamb) I thought it might be a good idea to split this issue to several PRs 1 - add the support to use `CREATE TABLE` syntax...
> Regarding the location of the code, if it is in datafusion proper rather than the CLI, it would be available in datafusion python, and any other projects that want...
Thanks @rtyler -- this sounds like we need to do some rounding in the tests when comparing floating point results We do this in other places, for example: https://github.com/apache/datafusion/blob/78055fe13b64cdfb4accd131e9a462770c21a934/datafusion/sqllogictest/test_files/scalar.slt#L104-L107 My...
This may now be feasible after - https://github.com/apache/datafusion/issues/13519
> I've been trying to measure the impact of these changes, but for the life of me I can't get useful results out of the benchmark comparison script. The sample...
🤖 `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing task_budget (5bfff9798b2815377ef8d40dfaffd26947c51385) to 06631c25316ed2fc20ab8114d0dcc801f353fbad [diff](https://github.com/apache/datafusion/compare/06631c25316ed2fc20ab8114d0dcc801f353fbad..5bfff9798b2815377ef8d40dfaffd26947c51385) Benchmarks: tpch_mem clickbench_partitioned clickbench_extended...
🤖: Benchmark completed Details ``` Comparing HEAD and task_budget -------------------- Benchmark clickbench_extended.json -------------------- ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Query ┃ HEAD ┃ task_budget ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ QQuery 0 │ 1938.69 ms...
🤖 `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing task_budget (a749e8f60cbcdf5542b4b69a0a2f94c7790fab45) to 06631c25316ed2fc20ab8114d0dcc801f353fbad [diff](https://github.com/apache/datafusion/compare/06631c25316ed2fc20ab8114d0dcc801f353fbad..a749e8f60cbcdf5542b4b69a0a2f94c7790fab45) Benchmarks: clickbench_1 Results will...