spark-compaction
spark-compaction copied to clipboard
KeithSSmith
File compaction tool that runs on top of the Spark framework.
Hi Keith, In line no 255 Cannot resolve symbol 'Option' . ### am using scala 2.11 & spark 2.2.0.  Thanks and Regards Venkat
Made changes to work with spark 2.2 and hadoop 2.7.2 Build passes successfully and works as expected.
A common approach of implementing Spark has become utilizing AWS EMR and using S3 for file storage in lieu of HDFS. Please implement S3 support for implementing a compaction algorithm.
This error occured while Iwas trying to comapact all snappy.parquet files which were generated in Spark2.1 with DataFrames. Is there any work around? Maybe to try with RDDs but how...
Metadata
Owner
Metadata
File compaction tool that runs on top of the Spark framework.