buckaroo icon indicating copy to clipboard operation
buckaroo copied to clipboard

Figure out how to join polars JLisp commands into a single Lazy block

Open paddymul opened this issue 2 years ago • 0 comments

Checks

  • [X] I have checked that this enhancement has not already been requested

How would you categorize this request. You can select multiple if not sure

Developer ergonomics (defaults, error messages), Performance, Developer Experience/CI (feature to make it easier to devlop on Buckaroo), Low Code UI

Enhancement Description

Polars is blazing fast if all operations are collected into a single lazy query. Pandas doesn't get this speed increase and it was simpler to write commands as returning a new dataframe, even if this was slow.

Pseudo Code Implementation

instead of emitting code like

def clean(df):
    df = df.with_columns(OP_A)
    df = df.with_columns(OP_B)
    df = df.with_columns(OP_C)
   return df

emit code like this

def clean(df):
    return df.with_columns(OP_A, OP_B, OP_C)

Prior Art

n/a

paddymul avatar Nov 20 '23 15:11 paddymul