docker-airflow
docker-airflow copied to clipboard
unable to edit postgres_default connection
i'm getting an invalidToken
error when attempting to edit the default postgres_default
connection. i suspect this is related to setting FERNET_KEY
but i can't seem to figure out what i'm doing wrong. i have FERNET_KEY
and AIRFLOW__CORE__FERNET_KEY
set to the same values for the webserver, scheduler, and worker containers.
i've also tried docker stop
to destroy all containers and rebuilding them with FERNET_KEY
and AIRFLOW__CORE__FERNET_KEY
set as environment variables in my docker-compose.yml
. i'm using CeleryExecutor
btw.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 2125, in edit_view
form = self.edit_form(obj=model)
File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 1340, in edit_form
return self._edit_form_class(get_form_data(), obj=obj)
File "/usr/local/lib/python3.7/site-packages/wtforms/form.py", line 212, in __call__
return type.__call__(cls, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_admin/form/__init__.py", line 16, in __init__
super(BaseForm, self).__init__(formdata=formdata, obj=obj, prefix=prefix, **kwargs)
File "/usr/local/lib/python3.7/site-packages/wtforms/form.py", line 278, in __init__
self.process(formdata, obj, data=data, **kwargs)
File "/usr/local/lib/python3.7/site-packages/wtforms/form.py", line 127, in process
if obj is not None and hasattr(obj, name):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py", line 353, in __get__
retval = self.descriptor.__get__(instance, owner)
File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 155, in get_password
return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
File "/usr/local/lib/python3.7/site-packages/cryptography/fernet.py", line 171, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
may be related to #320 and #387
additionally, i'm able to create new connections with encrypted fields and view/edit them. but i can't view any of the *_default
connections that include encrypted fields.
Just as a reference for future travellers who google their way here as I did: #320 does seem to have answer, which is setting the FERNET_KEY
env var... I'd created a stack based on the example docker-compose-LocalExecutor.yml
, without explicitly defining FERNET_KEY
; I didn't try to edit the postgres_default
connection until I'd spun the stack down and up a few times, and I guess that meant a stored (and autogenerated) fernet key didn't match the one (also autogenerated, but different) currently in use by the container. Even when I explicitly set FERNET_KEY
in my docker-compose.yml
, it still didn't seem to work, so just nuked my DB and started again, and then all was well: no key was autogenerated, it just used my key from the start, and the error went away. Hope this helps someone else.
@gimbo - When you say "Nuked my DB", what do you mean by this in the context of your Docker services? Did you execute a complete restart of the DB directly on the container, or did you just swap out for a new container? Did you nuke the Persistent Data volume? It might be helpful for the Googler's like me to get more specifics to your solution, since FERNET_KEY
explicit setting hasn't been resolved in either of the open issues in #320 and #387.
@mmain10 Sorry for the delayed reply... it was a while ago now but I think the answer is that yes, I deleted the persistent data volume.
This might work for you...
1) Before building/setting up the image based on puckel's docker file, generate a fernet key using your CLI with the line below:
docker run puckel/docker-airflow python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"
That will pull puckel's image, spin up a container, and generate a random fernet key from with in the container.
2) Copy the Fernet Key Generated and insert in your docker-compose file as an environment variable.
webserver:
build:
context: https://github.com/puckel/docker-airflow.git#1.10.10
dockerfile: Dockerfile
restart: always
environment:
- LOAD_EX=n
- EXECUTOR=Local
- AIRFLOW__CORE__FERNET_KEY=<INSERT THE FERNET KEY GENERATED HERE>
...
3) Remove the container generated on the step 2 (Optional).
4) Run the docker-compose file and enjoy.
Hope that works for you! Be safe!