Michel Hua

Thank you very much. The script works on my side. I found [Heather Miller's tutorial on Flintrock + S3](https://heather.miller.am/blog/launching-a-spark-cluster-part-2.html) quite cool if one day you write a tutorial from the...

I get this error message when trying to read Parquet on S3 : ```scala @ spark.read.parquet("s3a://bucket/path/to/parquet") java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154) org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2580) org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2593) org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) ``` This import wasn't...

I added it after, will try this afternoon to add them before or use `sync` thanks.

After adding imports at the correct place, ```scala @ import $ivy.`com.sun.jersey:jersey-client:1.9.1`, $ivy.`org.apache.spark::spark-sql:2.3.1`, $ivy.`sh.almond::ammonite-spark:0.1.1` @ import $ivy.`org.apache.hadoop:hadoop-aws:2.8.4`, $ivy.`com.amazonaws:aws-java-sdk-s3:1.11.336`, $ivy.`com.amazonaws:aws-java-sdk-emr:1.11.336` @ val spark = { AmmoniteSparkSession.builder() .progressBars() .master("yarn") .config("spark.executor.instances", "4") .config("spark.executor.memory", "2g")...

To solve the `Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found`, two related issues that tells us to add these to the classpath ``` /usr/share/aws/emr/emrfs/lib/* /usr/share/aws/emr/emrfs/auxlib/* /usr/share/aws/emr/emr-metrics/lib/* /usr/share/aws/emr/emrfs/conf or [ { "classification":"spark-defaults", "properties": {...

Well the link didn't work before my change, so it is a simple but big improvement haha !

Thanks I actually deleted the broken link validator found and regenerated the TOC completely using this online tool. https://luciopaiva.com/markdown-toc/ The TOC now looks ok, there were plenty of links missing...

(up) this is ready

Because it is not here anymore, I regenerated the summary based on what is on the document right now.

Thanks, I have just indented UI and Utilities as groups as you recommended. For the Utilities group it doesn't seem organized enough for me. It currently contains Files, Functional Programming,...