Better support for UDF when using Dotnet Spark.NET

[GeorgeS2019]

When using UDF or User-Defined Functions with Spark.NET, there are problems 4. User-Defined Functions

Describe the solution you'd like Now, there is a new initiative to continue support for Spark.NET, we need more compatibility with Dotnet Interativ and Dotnet Spark.NET, especially when working with UDF.

Describe alternatives you've considered Coordinate with SPARK.NET user community to identify ways to make UDF workable using Interativ. https://github.com/dotnet/spark/discussions/1179#discussioncomment-11427251

https://github.com/dotnet/spark/issues/1199

[GeorgeS2019]

We now have a new Spark.NET 2.3.0 version that support .NET 8

@wudanzy We shall test if there is still UDF problem with .NET interactive

[grazy27]

@GeorgeS2019 we wtill have this bug, path handling logic needs to be updated. I shared details in https://github.com/dotnet/spark/discussions/1179

[jonsequitur]

I'm open to a call to discuss the state of this and provide any guidance I can. Spark in .NET Interactive has been out of support for a few years because of the deprecation of BinaryFormatter but I think the fundamentals should still be in place for this to work if the appropriate fixes are made in .NET for Spark.

[grazy27]

@jonsequitur thanks! I'm on a vacation next week, but available afterwards, what is your availability? Help with dotnet.interactive APIs would be invaluable, currently I'm stuck on this DIsposableDirectory: https://github.com/dotnet/spark/discussions/1179#discussioncomment-11883846