MLCOMPUTE-967 | add default iam role for Spark drivers on k8s

[CaptainSame]

• Add newly created iam role to system paasta configs
• Add methods to fetch the iam role
• Use the iam role for creating the service account to start the Spark drivers with
• Mount kubeconfig file to Spark driver pods
• Point to the mounted kubeconfig using env variable

[88manpreet]

Can you add description, testing, any impact on existing infra, rollout and rollback plan if needed?

[edingroot]

The Spark ui port is currently picked from system_paasta_confg with a default value of 33000.

=> It should be picked by a pre-defined port range from spark_constants in srv-configs. Which the ranged is currently used by our Jupyter spark metrics aggregator.

---

EDIT:

1. The logic above is included in service_configuration_lib.get_spark_conf(), which should also be called by the spark driver on k8s implementation (might need to be slightly modified).

2. Some other logic such as _parse_user_spark_args(), where the executor pod template (to set dnsPolicy) is assigned, also need to be called by the driver on k8s implementation.

[edingroot]

Cherry-picked https://github.com/Yelp/paasta/pull/3749

[88manpreet]

Please manually test most common cases before pushing. Preferable to push after EST morning timezone.

[edingroot]

We'll need to upgrade service_configuration_lib to the latest version which includes https://github.com/Yelp/service_configuration_lib/pull/139

[88manpreet]

--mrjob option integration is missing. It needs to be added as part of list TronConfig