amazon-emr-cli icon indicating copy to clipboard operation
amazon-emr-cli copied to clipboard

Add support for `--show-logs` in cluster mode on EMR on EC2

Open dacort opened this issue 2 years ago • 0 comments

With the recent --show-logs flag, we switch the deploy mode to client so that EMR steps can capture the driver stdout.

Unfortunately, --client mode doesn't work with additional archives provided via the --archives flag or --conf spark.archives parameter. See https://issues.apache.org/jira/browse/SPARK-36088 for more a related issue.

In order to support this for cluster mode, we'd need to parse the step stderr logs to retrieve the Yarn application ID, then fetch the Yarn application logs from S3.

dacort avatar Apr 06 '23 23:04 dacort