graphchain icon indicating copy to clipboard operation
graphchain copied to clipboard

Optimisation hook for array/bag/dataframe

Open multimeric opened this issue 3 years ago • 2 comments

I just realised that the current hook that we use to insert graphchain into dask is delayed_optimize, which only actually applies for "raw" Delayed instances, but doesn't actually apply to e.g. the DataFrame API, which is a very popular part of dask. As described here, these types have their own optimisation hooks that we would need to implement. I will look in to writing one for the DataFrame API. I think we should be able to integrate it fairly easily into the current codebase.

  • [ ] array_optimize
  • [ ] dataframe_optimize
  • [ ] bag_optimize

multimeric avatar Aug 11 '22 05:08 multimeric

That would be great, thank you @multimeric!

lsorber avatar Aug 11 '22 05:08 lsorber

Okay so upon investigation, it seems that all the hooks use the same API, so we're all good in that regard, and it was just an unrelated bug that I was hitting.

multimeric avatar Aug 11 '22 11:08 multimeric