how to set proxy for astro cli

[lrcuwicRSG1]

Hi all, when I am starting astro-runtime via astro dev start I ran into a timeout when checking for new astronomer runtime. How can I set my proxy globally when building the image via astro? Operating system: Ubuntu 22.04 via WSL in Windows

[2024-02-21 18:02:12 +0000] [358] [INFO] Booting worker with pid: 358 [2024-02-21T18:02:23.371+0000] {update_checks.py:186} INFO - Checking for new version of Astronomer Runtime, previous check was performed at None [2024-02-21T18:02:44.477+0000] {update_checks.py:144} ERROR - Update check died with an exception, trying again in one hour Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request raise new_e File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn conn.connect() File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 611, in connect self.sock = sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f1e508dd110>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='updates.astronomer.io', port=443): Max retries exceeded with url: /astronomer-runtime?site=http%3A%2F%2Flocalhost%3A8080 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1e508dd110>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/astronomer/airflow/version_check/update_checks.py", line 139, in run update_available, wake_up_in = self.check_for_update() ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/astronomer/airflow/version_check/update_checks.py", line 199, in check_for_update for release in self._process_update_json(self._get_update_json()): ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/astronomer/airflow/version_check/update_checks.py", line 314, in _get_update_json r = requests.get( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='updates.astronomer.io', port=443): Max retries exceeded with url: /astronomer-runtime?site=http%3A%2F%2Flocalhost%3A8080 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1e508dd110>: Failed to establish a new connection: [Errno 111] Connection refused'))

[dosubot[bot]]

🤖

Hey there, @lrcuwicRSG1! Long time no see in the GitHub universe. Hope you've been sailing smooth in the code seas!

To set a global proxy when building the image via astro, you can use the ENV instruction in your Dockerfile to define the proxy environment variables. Here's how you can do it:

FROM quay.io/astronomer/astro-runtime:10.0.0-base

USER root

# Set the proxy environment variables
ENV http_proxy=http://proxy-server:proxy-port
ENV https_proxy=http://proxy-server:proxy-port

COPY ./pyproject.toml  ${AIRFLOW_HOME}/astronomer_cosmos/
COPY ./README.rst  ${AIRFLOW_HOME}/astronomer_cosmos/
COPY ./cosmos/  ${AIRFLOW_HOME}/astronomer_cosmos/cosmos/

# install the package in editable mode
RUN pip install -e "${AIRFLOW_HOME}/astronomer_cosmos"[dbt-postgres,dbt-databricks]

# make sure astro user owns the package
RUN chown -R astro:astro ${AIRFLOW_HOME}/astronomer_cosmos

USER astro

# add a connection to the airflow db for testing
ENV AIRFLOW_CONN_AIRFLOW_DB=postgres://airflow:pg_password@postgres:5432/airflow
ENV DBT_ROOT_PATH=/usr/local/airflow/dags/dbt
ENV DBT_DOCS_PATH=/usr/local/airflow/dbt-docs

Replace proxy-server and proxy-port with your proxy server's address and port. This will set the proxy for all the subsequent RUN instructions in the Dockerfile.

However, it's important to note that the provided Dockerfile in the Cosmos repository does not include any instructions for setting a global proxy. The Dockerfile mainly focuses on setting up the environment for the Cosmos application, including copying necessary files, installing the package, and setting environment variables.

To set a global proxy, you would typically need to add instructions to the Dockerfile to set environment variables such as HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. However, these instructions are not present in the provided Dockerfile. You can refer to the Dockerfile in the Cosmos repository for more details.

Sources

• dev/Dockerfile

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[lrcuwicRSG1]

There is an open thread affecting this topic --> https://forum.astronomer.io/t/astro-cli-astro-dev-start-failing-where-to-setup-proxy/2918/1 I posted my experiences there. I am looking forward to your suggestions Thanks Chris

[tatiana]

@lrcuwicRSG1, this issue seems related to Astro CLI. Please, reopen the ticket if it has any relation to Cosmos itself.

Please follow up with Astro CLI issues in the project: https://github.com/astronomer/astro-cli

[lrcuwicRSG1]

@tatiana Thanks for your reply. It is related to ASTRO CLI.