spark-fast-tests
spark-fast-tests copied to clipboard
MrPowers
Feature: add ignore Column Ordering in Dataset and Dataframe comparison
Adding the option to ignore column ordering. This is done by sorting the the dataset by column names. Consequently, this option is incompatible with ignoreColumnNames.
I have a working solution but i would like feed back before I write the remaining unit tests. Dataframe but not Dataset coverage. Additionally, for the dataset coverage I may need to create Datasets a little differently than all the other Dataset unit tests.
TODOs
- dataset unit tests
- schema compare unit test
@Nestor10 Thank you for the contribution. Unfortunately @MrPowers do not have time for supporting of the project and delegated the maintenance to me and @alfonsorr. As you may see, we are trying to bring the project back to life. At the moment I'm revisiting the old issues and open PRs. It is sad to say it, but what do you think about closing this PR in favor of #132 that brings the same functionality? Your contribution is appreciated anyway!
#132 was merged and the functionality is already in spark-fast-tests. I'm closing this PR now, @Nestor10 sorry about that. Your contributions is appreciated anyway!