MegaSparkDiff icon indicating copy to clipboard operation
MegaSparkDiff copied to clipboard

Return results of compareAppleTables in a wrapper class

Open aosama opened this issue 7 years ago • 3 comments

have the compareAppleTable method return the results in a wrapper class that will further provide more APIs.

So the code should look something like this

result = compareAppleTables(x,y).getLeft().sortByCols(“a”,“b”) OR result = compareAppleTables(x,y).getJoinedByCols(“a”,“b”).sortBy(“a”,“b”)

aosama avatar Feb 16 '18 04:02 aosama

@aosama I thought about this a little after we met today. Apparently Pair is just an abstract class. So if we made the new wrapper class a decorator for Pair<DataFrame, DataFrame> I believe it would be a non-breaking change, unless Scala has quirks about covariant return types that I don't know about.

mmlinford avatar May 31 '18 21:05 mmlinford

Since we've apparently decided to do breaking changes anyways, it might not be worth it to go with the suggestion I made (extending Pair is a little weird).

mmlinford avatar Jun 04 '18 14:06 mmlinford

was thinking of a custom result class like so

class ComparisonResults { inLeftNotInRight :DataFrame inRightNotInLeft: DataFrame def joinedResults(keys :Seq[String]) // this will return a full outer join between the differences } @mmlinford what do you think?

aosama avatar Oct 04 '18 19:10 aosama