Alembic error when using airflow-migrations

[chengzi0103]

Checks

• [X] I have checked for existing issues.
• [X] This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"clean", GoVersion:"go1.16.5"}

Description

When I configure additional packages, there is alembic in it, and the version is the same, both are 1.7.7 But when using airflow-db-migrations, there seems to be a version compatibility problem of alembic

I observed code.tpl that may be in db-migrations But I don't know how should I modify this code

Relevant Logs

/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.

   option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)

 /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.

   option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)

 /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.

 /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_pool_size option in [core] has been moved to the sql_alchemy_pool_size option in [database] - the old setting has been used, but please update your config.

 /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.

 /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.

 Traceback (most recent call last):

   File "/home/airflow/.local/bin/airflow", line 8, in <module>

     sys.exit(main())

   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 38, in main

     args.func(args)

   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command

     return func(*args, **kwargs)

   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 136, in check_migrations

     db.check_migrations(timeout=args.migration_wait_timeout)

   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 696, in check_migrations

     with _configured_alembic_environment() as env:

   File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__

     return next(self.gen)

   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 717, in _configured_alembic_environment

     config = _get_alembic_config()

   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 661, in _get_alembic_config

     from alembic.config import Config

   File "/home/airflow/.local/lib/python3.8/site-packages/alembic/config.py", line 12, in <module>

     from . import __version__

 ImportError: cannot import name '__version__' from 'alembic' (/home/airflow/.local/lib/python3.8/site-packages/alembic/__init__.py)

Custom Helm Values

extraPipPackages:
    - "apache-airflow-providers-docker"
    - "apache-airflow-providers-cncf-kubernetes"
    - "iconfig" # This is my private package

[chengzi0103]

repository: apache/airflow tag: 2.2.5-python3.8 pullPolicy: IfNotPresent

[thesuperzapper]

@chengzi0103 the problem is that something is causing apache-airflow==2.3.2 to be installed (the current latest), are you requiring an apache-airflow version in your iconfig package?

(Test it yourself by running pip install iconfig in an apache/airflow:2.2.5-python3.8 docker container, and see if it tries to install apache-airflow==2.3.2)

---

Also, the base airflow images already have all the providers installed like apache-airflow-providers-docker and apache-airflow-providers-cncf-kubernetes, so there is no need to specify them in extraPipPackages.

---

Finally, you should ALWAYS pin a version of packages like xxxxxx==1.0.0, otherwise:

• Pods will pick up whatever the latest version was at the time it started (which will inevitably result in an inconsistent version being installed across the cluster)
• you will run into errors like the above where the version of airflow that is "required" by your package changes and breaks everything

---

On a unrelated note, while looking at your issue, I found a problem with the way we implemented extraPipPackages in some edge cases see https://github.com/airflow-helm/charts/issues/598.

[chengzi0103]

the problem is that something is causing apache-airflow==2.3.2 to be installed (the current latest), are you requiring an apache-airflow version in your iconfig package?

(Test it yourself by running pip install iconfig in an apache/airflow:2.2.5-python3.8 docker container, and see if it tries to install apache-airflow==2.3.2)

thank you for your reply: My iconfig will not be associated with any airflow related packages, only the alembic package, I suspect that this package of alembic made some operations during initialization, which caused some problems

[thesuperzapper]

@chengzi0103 to clarify what are the install_requires in your iconfig package, are you saying its alembic, if so what version?

[chengzi0103]

@chengzi0103 to clarify what are the install_requires in your iconfig package, are you saying its alembic, if so what version?

i have checked . my iconfig pkg list:

click twine setuptools_rust bump2version psycopg2-binary pytest==7.0.1 setuptools==59.6.0 alembic==1.7.7

[thesuperzapper]

@chengzi0103 are you 100% sure that you are using apache/airflow:2.2.5-python3.8 as your base image?

Because you must be using airflow 2.3.X to get the error you list: /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config..

Something must be causing you to end up with airflow 2.3.X (and its probably related to your error), can you confirm if you are:

1. Actually using the apache/airflow:2.2.5-python3.8 (by kubectl describing one of your Pods)
2. Mounting any extraVolumes (especially over the /home/airflow/.local)
3. Installing ANY other packages with extraPipPackages

[chengzi0103]

@chengzi0103 are you 100% sure that you are using apache/airflow:2.2.5-python3.8 as your base image?

Because you must be using airflow 2.3.X to get the error you list: /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:525: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config..

Something must be causing you to end up with airflow 2.3.X (and its probably related to your error), can you confirm if you are:

1. Actually using the apache/airflow:2.2.5-python3.8 (by kubectl describing one of your Pods)
2. Mounting any extraVolumes (especially over the /home/airflow/.local)
3. Installing ANY other packages with extraPipPackages

1，yes i am 100% sure im using apache/airflow:2.2.5-python3.8 2, iconfig files tree ├── alembic │   ├── env.py │   ├── init.py │   ├── README │   ├── script.py.mako │   └── versions │   ├── 9c3e2908f8b8_message.py │   └── init.py ├── alembic.ini ├── CONTRIBUTING.rst ├── Dockerfile ├── HISTORY.rst ├── ibodb │   └── util.py ├── MANIFEST.in ├── README.md ├── README.rst ├── requirements.txt ├── setup.cfg ├── setup.py └── tests └── init.py

I think it has something to do with alembic.ini or the version folder put

[thesuperzapper]

@chengzi0103 I still believe your extraPipPackages are resulting in pip upgrading airflow to 2.3.X in an attempt to satisfy dependencies.

There is a new feature in chart version 8.6.1, which will cause the pods to fail at startup if your extraPipPackages try and cause an airflow upgrade/downgrade (see PR https://github.com/airflow-helm/charts/pull/610)

I imagine you will see this failure if you update, so you should figure out what you need to change about your extraPipPackages before updating.

Can you report back if you figure it out?

[chengzi0103]

@thesuperzapper Okay, what I'm sure about is that the problem is on alembic. We have our own set of package versions, which conflict with some packages that come with airflow. My temporary solution is to temporarily give up my own package control and use airflow to automatically control the package. with the package and version, so I have skipped this question, thank you for your reply, hope to do better

[thesuperzapper]

@chengzi0103 I will close this issue for now, feel free to reopen it if you think it's still a problem.