r-polars icon indicating copy to clipboard operation
r-polars copied to clipboard

Published 20 hours ago • verified publisher icon pola-rs

Scan R data.frame

Open eitsupi opened this issue 1 year ago • 3 comments

For now, in as_polars_lf, we convert data.frame to RPolarsDataFrame (high cost) and then scan that. Can we scan data.frame directly and do pushing down? I think duckdb does that.

eitsupi avatar Jan 06 '24 03:01 eitsupi

That could also improve pl$LazyFrame(), which creates a DataFrame and then calls $lazy()

etiennebacher avatar Jan 06 '24 21:01 etiennebacher

It seems that this requires something like these:

  • https://github.com/pola-rs/polars/blob/7749371169d62ba0a170520baa10834073c5d51f/crates/polars-lazy/src/frame/python.rs#L6-L18
  • https://github.com/pola-rs/polars/blob/7749371169d62ba0a170520baa10834073c5d51f/crates/polars-lazy/src/physical_plan/executors/python_scan.rs#L10-L54

...and update here.

https://github.com/pola-rs/polars/blob/7cdd4f81af679a337d279259211f8d8264019141/crates/polars-lazy/src/physical_plan/planner/lp.rs#L143-L571

eitsupi avatar Feb 04 '24 04:02 eitsupi

AnonymousScan (pola-rs/polars#3561) seems exist for usages like this.

eitsupi avatar Feb 05 '24 13:02 eitsupi