airflow-tutorial
airflow-tutorial copied to clipboard
docker-compose up -d error
OS: Arcolinux
I have installed the prerequisites, not sure why I'm getting this error. I don't have Postgres installed, maybe it has to do with this?
➜ docker-compose up -d
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.9/http/client.py", line 1257, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1252, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1012, in _send_output
self.send(msg)
File "/usr/lib/python3.9/http/client.py", line 952, in send
self.connect()
File "/usr/lib/python3.9/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.9/http/client.py", line 1257, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1252, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1012, in _send_output
self.send(msg)
File "/usr/lib/python3.9/http/client.py", line 952, in send
self.connect()
File "/usr/lib/python3.9/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/usr/lib/python3.9/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/lib/python3.9/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 33, in <module>
sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
File "/usr/lib/python3.9/site-packages/compose/cli/main.py", line 81, in main
command_func()
File "/usr/lib/python3.9/site-packages/compose/cli/main.py", line 200, in perform_command
project = project_from_options('.', options)
File "/usr/lib/python3.9/site-packages/compose/cli/command.py", line 60, in project_from_options
return get_project(
File "/usr/lib/python3.9/site-packages/compose/cli/command.py", line 152, in get_project
client = get_client(
File "/usr/lib/python3.9/site-packages/compose/cli/docker_client.py", line 41, in get_client
client = docker_client(
File "/usr/lib/python3.9/site-packages/compose/cli/docker_client.py", line 170, in docker_client
client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
I just cloned the repo a moment ago and ran into some errors when trying to run "docker-compose up" too. For my case, the error message was complaining about some missing mysql file towards the end..
I updated the docker image for puckel/docker-airflow to version 1.10.9 instead of 1.10.1 at the docker-compose.yml file. And also the sqlalchemy dependency to version 1.3.0.
It works after that on my machine. I'm on Windows 10 with Docker Desktop v3.5.2(66501). Hope this helps!
p.s. Also, I happened to miss out the 3rd prerequisite initially, that it to install the Apache Airflow python package. Don't miss it out.
My working version of the docker-compose.yml:
version: '3'
services:
postgres:
image: postgres:9.6
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- "5432:5432"
webserver:
image: puckel/docker-airflow:1.10.9
build:
context: https://github.com/puckel/docker-airflow.git#1.10.9
dockerfile: Dockerfile
args:
AIRFLOW_DEPS: gcp_api,s3
PYTHON_DEPS: sqlalchemy==1.3.0
restart: always
depends_on:
- postgres
environment:
- LOAD_EX=n
- EXECUTOR=Local
- FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo=
volumes:
- ./examples/intro-example/dags:/usr/local/airflow/dags
# Uncomment to include custom plugins
# - ./plugins:/usr/local/airflow/plugins
ports:
- "8080:8080"
command: webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
I applied your suggestion but I keep getting the same error with _mysql. Docker engine v20.10.8 Docker desktop v 3.6.0
webserver_1 |
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/bin/airflow", line 21, in
I just cloned the repo a moment ago and ran into some errors when trying to run "docker-compose up" too. For my case, the error message was complaining about some missing mysql file towards the end..
I updated the docker image for puckel/docker-airflow to version 1.10.9 instead of 1.10.1 at the docker-compose.yml file. And also the sqlalchemy dependency to version 1.3.0.
It works after that on my machine. I'm on Windows 10 with Docker Desktop v3.5.2(66501). Hope this helps!
p.s. Also, I happened to miss out the 3rd prerequisite initially, that it to install the Apache Airflow python package. Don't miss it out.
My working version of the docker-compose.yml:
version: '3' services: postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow ports: - "5432:5432" webserver: image: puckel/docker-airflow:1.10.9 build: context: https://github.com/puckel/docker-airflow.git#1.10.9 dockerfile: Dockerfile args: AIRFLOW_DEPS: gcp_api,s3 PYTHON_DEPS: sqlalchemy==1.3.0 restart: always depends_on: - postgres environment: - LOAD_EX=n - EXECUTOR=Local - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo= volumes: - ./examples/intro-example/dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins ports: - "8080:8080" command: webserver healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3
This worked 😄
I applied your suggestion but I keep getting the same error with _mysql. Docker engine v20.10.8 Docker desktop v 3.6.0
webserver_1 | webserver_1 | Traceback (most recent call last): webserver_1 | File "/usr/local/bin/airflow", line 21, in webserver_1 | from airflow import configuration webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/init.py", line 36, in webserver_1 | from airflow import settings webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 261, in webserver_1 | configure_adapters() webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 218, in configure_adapters webserver_1 | import MySQLdb.converters webserver_1 | File "/usr/local/lib/python3.6/site-packages/MySQLdb/init.py", line 24, in webserver_1 | version_info, _mysql.version_info, _mysql.file webserver_1 | NameError: name '_mysql' is not defined
Having the same issue here. Did you find a way to fix it?
@zishen1229 Have you fulfilled the third prerequisite yet? That is to install the Python package for Apache Airflow here .
@zishen1229 Have you fulfilled the third prerequisite yet? That is to install the Python package for Apache Airflow here .
Yes, my versions are as below: docker: Client: Cloud integration: 1.0.17 Version: 20.10.8 API version: 1.41 Go version: go1.16.6 Git commit: 3967b7d Built: Fri Jul 30 19:55:20 2021 OS/Arch: darwin/amd64 Context: default Experimental: true docker-compose: docker-py version: 4.4.4 CPython version: 3.8.5 OpenSSL version: OpenSSL 1.1.1h 22 Sep 2020 airflow version: 2.1.4
@zishen1229
Hi. Have you fixed the error?
After a few tries, I ran it successfully by following the guide from @DriLLFreAK100 above.
Please notice that you have to modify the context to 1.10.9 as well.
context: https://github.com/puckel/docker-airflow.git#1.10.9
For me Manual installation steps for older versions of WSL worked.