schireson
AttributeError: 'async_generator' object has no attribute 'dialect'
I have configured my project following the setup instruction. In my project I use asyncpg, sqlalchemy and alembic.
Running command pytest --test-alembic produces this output:
FAILED tests::pytest_alembic::test_model_definitions_match_ddl - AttributeError: 'async_generator' object has no attribute 'dialect'
FAILED tests::pytest_alembic::test_up_down_consistency
FAILED tests::pytest_alembic::test_upgrade - AttributeError: 'async_generator' object has no attribute 'dialect'
my env.py:
import asyncio
from alembic import context
from asyncpg import pool
from sqlalchemy import engine_from_config
from sqlalchemy.ext.asyncio.engine import AsyncEngine
from bot.db import Base
def run_migrations_online():
connectable = context.config.attributes.get("connection", None)
if connectable is None:
connectable = AsyncEngine(
engine_from_config(
context.config.get_section(context.config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
future=True,
)
)
# Note, we decide whether to run asynchronously based on the kind of engine we're dealing with.
if isinstance(connectable, AsyncEngine):
asyncio.run(run_async_migrations(connectable))
else:
do_run_migrations(connectable)
# Then use their setup for async connection/running of the migration
async def run_async_migrations(connectable):
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def do_run_migrations(connection):
context.configure(connection=connection, target_metadata=Base.metadata)
with context.begin_transaction():
context.run_migrations()
# But the outer layer still allows sychronous execution also.
run_migrations_online()
More traceback:
cls = <class 'alembic.runtime.migration.MigrationContext'>, connection = <async_generator object create_postgres_fixture.<locals>._async at 0x105e3e9c0>, url = None, dialect_name = None
dialect = None, environment_context = None, dialect_opts = {}, opts = {}
@classmethod
def configure(
cls,
connection: Optional["Connection"] = None,
url: Optional[str] = None,
dialect_name: Optional[str] = None,
dialect: Optional["Dialect"] = None,
environment_context: Optional["EnvironmentContext"] = None,
dialect_opts: Optional[Dict[str, str]] = None,
opts: Optional[Any] = None,
) -> "MigrationContext":
"""Create a new :class:`.MigrationContext`.
This is a factory method usually called
by :meth:`.EnvironmentContext.configure`.
:param connection: a :class:`~sqlalchemy.engine.Connection`
to use for SQL execution in "online" mode. When present,
is also used to determine the type of dialect in use.
:param url: a string database url, or a
:class:`sqlalchemy.engine.url.URL` object.
The type of dialect to be used will be derived from this if
``connection`` is not passed.
:param dialect_name: string name of a dialect, such as
"postgresql", "mssql", etc. The type of dialect to be used will be
derived from this if ``connection`` and ``url`` are not passed.
:param opts: dictionary of options. Most other options
accepted by :meth:`.EnvironmentContext.configure` are passed via
this dictionary.
"""
if opts is None:
opts = {}
if dialect_opts is None:
dialect_opts = {}
if connection:
if isinstance(connection, Engine):
raise util.CommandError(
"'connection' argument to configure() is expected "
"to be a sqlalchemy.engine.Connection instance, "
"got %r" % connection,
)
> dialect = connection.dialect
E AttributeError: 'async_generator' object has no attribute 'dialect'
What does your alembic_engine fixture look like? From the traceback, it seems like connectable is an async generator, which seems like it'd indicate your alembic_engine` isn't resolving into a real engine and is just the generator produced by an incorrectly set up fixture. My perhaps wild guess based on that traceback is perhaps your fixture looks like this:
@pytest.fixture
async def alembic_engine(...):
...
yield create_async_engine(URL(...))
...
If so:
-
create_async_engineisn't async so you shouldn't need to have an async fixture unless you're doing other stuff, which you may well be. - if you do need the async fixture, you'd need
@pytest.mark.asynciowith thepytest-asynciopackage to get pytest to resolve that into a proper fixture.
With all that said, given that your env.py is already set up to run in either sync or async mode, perhaps you'd be better off just running it synchronously? Seems like it implies that you have both the sync/async driver; and avoiding needing the sync driver in the first place, seems like the primary reason you'd gain anything from async alembic.
So, my fixtures:
import pytest
from pytest_alembic.config import Config
from pytest_mock_resources import create_postgres_fixture
@pytest.fixture
def alembic_config():
"""Override this fixture to configure the exact alembic context setup required.
"""
return Config(
config_options={
'sqlalchemy.url': 'url', # url to test database
'script_location': 'app/db/migrations'
}
)
alembic_engine = create_postgres_fixture(async_=True)
Do I have to use async_=False instead ?
With all that said, given that your env.py is already set up to run in either sync or async mode, perhaps you'd be better off just running it synchronously? Seems like it implies that you have both the sync/async driver; and avoiding needing the sync driver in the first place, seems like the primary reason you'd gain anything from async alembic.
So, right now I only have the asyncpg driver, but I've already tried to do this with the psycopg2 driver and got a few more errors
you should be able to use async_=True, we have a test for that: https://github.com/schireson/pytest-alembic/blob/main/examples/test_async_sqlalchemy_native/conftest.py
I wouldn't expect you to need to set the sqlalchemy.url config_option, as alembic_engine should be being set as connectable, which means you wont read that value out of the ini config; just fyi.
Do you have pytest-asyncio installed?
you should be able to use
async_=True, we have a test for that: https://github.com/schireson/pytest-alembic/blob/main/examples/test_async_sqlalchemy_native/conftest.pyI wouldn't expect you to need to set the
sqlalchemy.urlconfig_option, asalembic_engineshould be being set asconnectable, which means you wont read that value out of the ini config; just fyi.Do you have
pytest-asyncioinstalled?
I thought that Alembic_engine will interact with my database and will change DDL, what I don't want to do.
So, this is my poetry dependencies tree:
aiogram 3.0.0-beta.4 Modern and fully asynchronous framework for Telegram Bot API
├── aiofiles >=0.8.0,<0.9.0
├── aiohttp >=3.8.1,<4.0.0
│ ├── aiosignal >=1.1.2
│ │ └── frozenlist >=1.1.0
│ ├── async-timeout >=4.0.0a3,<5.0
│ ├── attrs >=17.3.0
│ ├── charset-normalizer >=2.0,<3.0
│ ├── frozenlist >=1.1.1 (circular dependency aborted here)
│ ├── multidict >=4.5,<7.0
│ └── yarl >=1.0,<2.0
│ ├── idna >=2.0
│ └── multidict >=4.0 (circular dependency aborted here)
├── magic-filter >=1.0.8,<2.0.0
├── pydantic >=1.9.2,<2.0.0
│ └── typing-extensions >=3.7.4.3
└── pygments >=2.12.0,<3.0.0
aioredis 2.0.1 asyncio (PEP 3156) Redis support
├── async-timeout *
└── typing-extensions *
alembic 1.8.1 A database migration tool for SQLAlchemy.
├── mako *
│ └── markupsafe >=0.9.2
└── sqlalchemy >=1.3.0
└── greenlet !=0.4.17
asyncpg 0.25.0 An asyncio PostgreSQL driver
greenlet 1.1.2 Lightweight in-process concurrent programming
mypy 0.971 Optional static typing for Python
├── mypy-extensions >=0.4.3
├── tomli >=1.1.0
└── typing-extensions >=3.10
pytest 7.1.2 pytest: simple powerful testing with Python
├── atomicwrites >=1.0
├── attrs >=19.2.0
├── colorama *
├── iniconfig *
├── packaging *
│ └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
├── pluggy >=0.12,<2.0
├── py >=1.8.2
└── tomli >=1.0.0
pytest-alembic 0.8.4 A pytest plugin for verifying alembic migrations.
├── alembic *
│ ├── mako *
│ │ └── markupsafe >=0.9.2
│ └── sqlalchemy >=1.3.0
│ └── greenlet !=0.4.17
├── pytest >=1.0
│ ├── atomicwrites >=1.0
│ ├── attrs >=19.2.0
│ ├── colorama *
│ ├── iniconfig *
│ ├── packaging *
│ │ └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
│ ├── pluggy >=0.12,<2.0
│ ├── py >=1.8.2
│ └── tomli >=1.0.0
└── sqlalchemy *
└── greenlet !=0.4.17
pytest-asyncio 0.19.0 Pytest support for asyncio
└── pytest >=6.1.0
├── atomicwrites >=1.0
├── attrs >=19.2.0
├── colorama *
├── iniconfig *
├── packaging *
│ └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
├── pluggy >=0.12,<2.0
├── py >=1.8.2
└── tomli >=1.0.0
pytest-mock-resources 2.4.4 A pytest plugin for easily instantiating reproducible mock resources.
├── filelock *
├── psycopg2 *
├── pytest >=1.0
│ ├── atomicwrites >=1.0
│ ├── attrs >=19.2.0
│ ├── colorama *
│ ├── iniconfig *
│ ├── packaging *
│ │ └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
│ ├── pluggy >=0.12,<2.0
│ ├── py >=1.8.2
│ └── tomli >=1.0.0
├── python-on-whales >=0.22.0
│ ├── pydantic *
│ │ └── typing-extensions >=3.7.4.3
│ ├── requests *
│ │ ├── certifi >=2017.4.17
│ │ ├── charset-normalizer >=2,<3
│ │ ├── idna >=2.5,<4
│ │ └── urllib3 >=1.21.1,<1.27
│ ├── tqdm *
│ │ └── colorama *
│ ├── typer >=0.4.1
│ │ └── click >=7.1.1,<9.0.0
│ │ └── colorama * (circular dependency aborted here)
│ └── typing-extensions * (circular dependency aborted here)
├── responses *
│ ├── requests >=2.0,<3.0
│ │ ├── certifi >=2017.4.17
│ │ ├── charset-normalizer >=2,<3
│ │ ├── idna >=2.5,<4
│ │ └── urllib3 >=1.21.1,<1.27
│ └── urllib3 >=1.25.10 (circular dependency aborted here)
└── sqlalchemy >1.0,<1.4.0 || >1.4.0,<1.4.1 || >1.4.1,<1.4.2 || >1.4.2,<1.4.3 || >1.4.3,<1.4.4 || >1.4.4,<1.4.5 || >1.4.5,<1.4.6 || >1.4.6,<1.4.7 || >1.4.7,<1.4.8 || >1.4.8,<1.4.9 || >1.4.9,<1.4.10 || >1.4.10,<1.4.11 || >1.4.11,<1.4.12 || >1.4.12,<1.4.13 || >1.4.13,<1.4.14 || >1.4.14,<1.4.15 || >1.4.15,<1.4.16 || >1.4.16,<1.4.17 || >1.4.17,<1.4.18 || >1.4.18,<1.4.19 || >1.4.19,<1.4.20 || >1.4.20,<1.4.21 || >1.4.21,<1.4.22 || >1.4.22,<1.4.23 || >1.4.23
└── greenlet !=0.4.17
python-dotenv 0.20.0 Read key-value pairs from a .env file and set them as environment variables
redis 4.3.4 Python client for Redis database and key-value store
├── async-timeout >=4.0.2
├── deprecated >=1.2.3
│ └── wrapt >=1.10,<2
└── packaging >=20.4
└── pyparsing >=2.0.2,<3.0.5 || >3.0.5
sqlalchemy 1.4.40 Database Abstraction Library
└── greenlet !=0.4.17
the if connectable is None: branch of your env.py should be being skipped because pytest-alembic ought to be setting the connectable for you, by subbing in the result of alembic_engine.
I think i found the problem, pytest-asyncio appears to have a set of breaking changes above version 0.16.0. You start seeing issues at 0.17.0 to 0.19.0. I can see if there's an easy fix in either pytest-alembic or pytest-mock-resources.
the
if connectable is None:branch of yourenv.pyshould be being skipped because pytest-alembic ought to be setting theconnectablefor you, by subbing in the result ofalembic_engine.I think i found the problem,
pytest-asyncioappears to have a set of breaking changes above version 0.16.0. You start seeing issues at 0.17.0 to 0.19.0. I can see if there's an easy fix in either pytest-alembic or pytest-mock-resources.
Am I have to downgrade pytest-asyncio to 0.16.0? I think this isn't the best solution, but...
I has done fully prohibited change. I edited package pytest_mock_resources and changed in function create_postgres_fixture decorator for inner function _async from @pytest.fixture to @pytest_async.fixture. And now, it works fine, but suddenly I found some errors related with alembic migrations, which I need to fix.
that's interesting, I had attempted to create a standalone package using pytest-alembic and pytest-mock-resources, while making similar in-place modifications create_postgres_fixture but i was still seeing the originally reported issue.
Like I said, for a right-now fix for yourself, you could downgrade pytest-asyncio to 0.16.
I can probably address the compatibility issue in pytest-mock-resources and release a new version later today/tomorrow; if I can reproduce the fixed behavior change you're seeing with your suggested change.
there's a new version of pytest-mock-resources released (2.5.0) which addresses this issue, as far as the issue applies to it.
If you're using the --test-alembic flag, i seem to still see the issue; versus if you directly from pytest_alembic.tests import test_upgrade, ..., then it appears to work that way.
Okay, turns out there was an additional incompatibility with pytest-mock-resources versions >=2.4 which would fail with pytest-asyncio versions above 0.19, specifically with async under certain circumstances.
So 2.5.1 should close the loop on this and will likely be released tomorrow. and I'm willing to call this issue done, since both issues turned out to be external to pytest-alembic.