mrjob icon indicating copy to clipboard operation
mrjob copied to clipboard

Support --conf-path when using mrjob spark-submit

Open mj3c opened this issue 5 years ago • 0 comments

Am I doing something wrong or is the --conf-path / -c option not supported when trying to run a PySpark script using mrjob spark-submit? For example, I have an EMR runner configuration specified in the custom.conf file, and I try to run the script with:

mrjob spark-submit -r emr -c custom.conf main.py

In the logs, it clearly says:

Looking for configs in /home/<my_home>/.mrjob.conf Looking for configs in /etc/mrjob.conf No configs found; falling back on auto-configuration

Why does it only check those two locations, and not consider the passed argument?

mj3c avatar Feb 11 '20 10:02 mj3c