MegaSparkDiff
MegaSparkDiff copied to clipboard
Return results of compareAppleTables in a wrapper class
have the compareAppleTable method return the results in a wrapper class that will further provide more APIs.
So the code should look something like this
result = compareAppleTables(x,y).getLeft().sortByCols(“a”,“b”) OR result = compareAppleTables(x,y).getJoinedByCols(“a”,“b”).sortBy(“a”,“b”)
@aosama I thought about this a little after we met today. Apparently Pair
is just an abstract class. So if we made the new wrapper class a decorator for Pair<DataFrame, DataFrame>
I believe it would be a non-breaking change, unless Scala has quirks about covariant return types that I don't know about.
Since we've apparently decided to do breaking changes anyways, it might not be worth it to go with the suggestion I made (extending Pair
is a little weird).
was thinking of a custom result class like so
class ComparisonResults { inLeftNotInRight :DataFrame inRightNotInLeft: DataFrame def joinedResults(keys :Seq[String]) // this will return a full outer join between the differences } @mmlinford what do you think?