Support for ARM machine

[bhavaniravi]

Describe the bug The spark script and the quarto script fails in pipeline since i'm running docker-compose on my M1 mac

To Reproduce

1. Run the make up command
2. Run the airflow pipeline
3. If everything goes well the spark script will fail saying "no such file or directly"
4. If you change the Java to arm instead of amd in tar, it fixes it
5. Then the quarto fails. Similarly updating the bash script to arm fixes it

Expected behavior Spark should run without any issues

Screenshots

Desktop (please complete the following information):

• OS: Mac M1 Sonoma
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

[MounirHader]

I am facing the same error here: the airflow DAG fails on the spark script with the following error logs:

[2024-09-18, 20:00:18 UTC] {subprocess.py:75} INFO - Running command: ['/usr/bin/bash', '-c', 'python /opt/***/dags/scripts/spark/random_text_classification.py']
[2024-09-18, 20:00:18 UTC] {subprocess.py:86} INFO - Output:
[2024-09-18, 20:00:18 UTC] {subprocess.py:93} INFO - /opt/spark/bin/load-spark-env.sh: line 68: ps: command not found
[2024-09-18, 20:00:18 UTC] {subprocess.py:93} INFO - /opt/spark/bin/spark-class: line 71: /usr/lib/jvm/java-17-openjdk-amd64/bin/java: No such file or directory

@bhavaniravi what exactly did you have to change to make it work on ARM machine?