Pass aws credentials to spark executors as environment variables

[88manpreet]

[solarkennedy]

@huadongliu can you review this one? Doesn't "--aws-credentials-yaml" do this for users already?

[lydian]

@huadongliu can you review this one? Doesn't "--aws-credentials-yaml" do this for users already?

We only do it for spark driver, the change is for spark executor. Normally spark executor doesn't need this env. This env is only helpful when people trying to access AWS resources inside the executor. For native spark script, we don't recommend people access AWS resource inside executor, because it is higher chance to hit the AWS api limit. However, it looks like this is the quick fix for mrjob harness script, so maybe go with this fix first, and if people do see the s3 limit error, we can discuss with the team for the workaround.

[mattmb]

Cleaning up and closing some very old PRs. Please re-open or nudge me if you’re still planning to work on this.