MrPowers
DataFrame#transform method added in Spark 3
The from quinn.extensions import * import statement will run this code.
Should we add a PySpark version check to only run this code if the PySpark version is less than 3?
@nvander1 @afranzi - let me know your thoughts!
I say we just delete this file. This would be something good to tackle for the 1.0 release.
@NikhilGupta178 - yea, there is a suggested solution here: https://github.com/MrPowers/quinn/issues/35#issuecomment-1460146628. Do you want to grab this one?
cc: @SemyonSinchenko
@MrPowers Yeah, I would like to do this. Suggested solution looks like a good fit.
@NikhilGupta178 - assigned you to the issue. Let us know when you have a PR that's ready for us to review/merge. Thank you!
I think we should do it for all the extensions, not only transform.
P.S. @MrPowers we already discussed this topic that such a pattern is not fir with Python Zen; we should avoid it and maybe delete it in the future versions. What do you think about wrap all the extensions into syntax sugar with DeprecationWarning inside? Like this:
import warnings
def _ext_function(self, f):
warnings.warn(
"Extensions may be removed in the future versions of quinn. Please use explicit functions instead",
category=DeprecationWarning,
stacklevel=2
)
return transform(self, f)
DataFrame.transform = getattr(DataFrame, "transform", _ext_function)
@SemyonSinchenko Isn't the above code snippet, a complete solution for this particular issue ?
@SemyonSinchenko Isn't the above code snippet, a complete solution for this particular issue ?
It is just one of ways how to do it. I didn't run it to ne homest and there may be a typo or syntax error.
@SemyonSinchenko - can you open a new issue for the broader discussion?
@NikhilGupta178 - for purposes of this issue, you can just submit a PR that addresses the DataFrame transform extension. Loading it for PySpark 2 and ignoring it for PySpark 3 would be ideal. Thank you!
@MrPowers I created a discussion here #74 I guess we can continue there.