jsleight

I also encountered this. I'm pretty sure it is caused by #6887. So you should set the `XDG_RUNTIME_DIR` env variable to control where the socket gets created. I just set...

Don't think there's been any updates w.r.t. simplifying the process of making custom transformers.

In my experience, the `TypeError: 'JavaPackage' object is not callable` error has always indicated that spark's jvm didn't initialize correctly or doesn't have the correct packages installed. If you add...

#860 Is working on it.

Don't know, could ask on that PR.

#864 is merged so in theory someone can make the project cross compile now without needing to worry about the dependencies being compatible. If anyone wants to start a PR,...

Not sure, but once it has a clean build I'm happy to merge it. I see you've commented on that PR as well, which is good.

SQLTransformer isn't supported, and tbh is hard to support. We'd essentially need to replicate Spark's sql engine without Spark. I would advise to filter the rows before calling the pipeline.

I might have some time to make a PR. I can envision a couple of approaches so interested to see which you'd prefer. 1. Just change [the line in Dataset](https://github.com/stanfordnlp/dspy/blob/303c66991899a7fc5123ee5642f4ff4d7f76560f/dspy/datasets/dataset.py#L76)...

releasing a new mleap version sgtm. I might have time to do this next week.