Han Wang
Auto persist is done, auto unpersist is not
I guess you want to have another level of distribution inside `filter_rm_duplicates`, right? I think this is disallowed by dask see similar https://stackoverflow.com/questions/6974695/python-process-pool-non-daemonic
@Matthieu-Tinycoaching do you mind to share the `filter_rm_duplicates`? This should be a very common case and of course Fugue supports. Also this error means dask doesn't allow you to do...
@Matthieu-Tinycoaching if you don't have further question, do you mind I close this issue?
Closing, feel free to reopen. Thanks
That will be fantastic!! @rdmolony can you join the [slack channel](https://join.slack.com/t/fugue-project/shared_invite/zt-jl0pcahu-KdlSOgi~fP50TZWmNxdWYQ) let's chat about the details? I started a little bit on the conda side long time ago, it would...
This is great. I will start working on that next week.
Hi @FedericoGarza sorry about the delay. I will try to start later this week. We were busy with databricks integrations in the last few weeks.
You can do ```python df1.partition_by("a").zip(df2).transform(scd2_merge).show() ``` or ```python dag.zip(df1, df2, partition={"by":"a"}).transform(scd2_merge).show() ```