spark-daria
spark-daria copied to clipboard
MrPowers
Essential Spark extensions and helper methods ✨😲
I am trying to write some test cases to validate the data between a .parquet file in s3 and target (hive table). I have loaded the .parquet data into one...
Try to create a better error messages when the row sizes aren't equal to the number of columns with the `createDF` method.
Some users don't want to download this library. They just want to download a wheel file and attach it to their cluster. We should make this workflow easier for people...
Hi. I would like to know if there is a way to turn off case sensitivity on **requiredColumns** and **addedColumns**? Even if I have spark.sql.caseSensitive set to false my unit...
Could you please provide an option to pass options value where we can pass to DariaWriters.writeSingleFile(). For example, I want to pass the header value.
- [x] run tests, formatting check on pull request #160 - [x] update outdated packages and plugins - [ ] Run CI only on project affected by changes
Close #155 - Add rand_range to Samples from ∼ U ( α , β ) - Add randn to Samples from ∼ N ( α , β ) - Rename...
This would be similar to how flaten_dataframe work in quinn
This feature would be similar to either https://github.com/mrpowers-io/quinn/issues/88 or how sparklyr method to generate dataframe with certain distribution E.x https://spark.posit.co/packages/sparklyr/latest/reference/sdf_rexp.html Random integers generation: - [ ] Samples from ∼ U...