ensure version_locations can be modified in env.py, memoizations are reset, etc.

[alexykot]

Right now the dirname of the migrations path is hardcoded to versions. Would be great to have it configurable via the EnvironmentContext instead. It would allow to keep migration histories of several applications alongside each other in folder structure like migrations/app1/, migrations/app2/ etc. This would massively simplify migrations management in monorepo system architecture.

[zzzeek]

hi there -

can you check out:

https://alembic.sqlalchemy.org/en/latest/tutorial.html?highlight=version_locations

https://alembic.sqlalchemy.org/en/latest/branches.html?highlight=version_locations#setting-up-multiple-version-directories

and see if that solves your problem? you can probably set version_locations from env.py as well as long as you add "revision_environment=true" to your alembic.ini

# version location specification; this defaults
# to alembic/versions.  When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions

[alexykot]

That works from alembic.ini, but it does not work from env.py. So I have my migrations in model/migrations/app1/ and alembic.ini at the root level.
In alembic.ini I have:

[alembic]
script_location = model/migrations/
version_locations = %(here)s/model/migrations/app1/

This works fine.

But if I try to remove the above from alembic.ini and put in the env.py to control it programmatically - it's no longer visible to alembic. In env.py I tried adding:

from alembic import context

config = context.config
config.set_main_option("script_location", f"model/migrations/")
config.set_main_option("version_locations", "model/migrations/app1/")

This fails on trying to run or generate migrations with: FAILED: No 'script_location' key found in configuration.

I have revision_environment=true in alembic.ini, but it doesn't make a difference.

This can be workarounded by setting version_locations = %(here)s/model/migrations/app1/ %(here)s/model/migrations/app2/ ... in alembic.ini, but that will mean I'll have to set it up manually on creating every new app.

[zzzeek]

what is your full front-to-back workflow with setting a different version location within env.py ? will the env.py be invoked and set a single version_location each time ? how is it being told that, are you using -x or a custom front-end ?

[zzzeek]

because note there is also a workflow where a single alembic.ini can serve multple version locations if you have a separate env.py in each one:

https://alembic.sqlalchemy.org/en/latest/cookbook.html#run-multiple-alembic-environments-from-one-ini-file

in additition, you can have each app-specific env.py import a central "env.py" that has your full set of workflow within it, so you only need a short "stub" env.py in each one.

[alexykot]

what is your full front-to-back workflow

I'm figuring it out now, so I don't really have a working version yet. But here is what I have. Dir structure:

py/
- alembic.ini
- model/
  - app1.py   # app1 models
  - app2.py   # app2 models
  - migrations/
    - env.py
    - script.py.mako
    - app1/
      - ..app1 migrations files
    - app2/
      - ..app2 migrations files

alembic.ini contents:

[alembic]
script_location = model/migrations/
revision_environment=true
... standard logging config...

env.py contents:

import os

from alembic import context
from sqlalchemy import engine_from_config, pool

app_name = os.environ.get("APP_NAME")
db_url = os.environ.get("DB_URL")
app_model = import_module(f'model.{app_name}')
target_metadata = app_model.Base.metadata

config.set_main_option("version_locations", f"model/migrations/{app_name}/")
...
def run_migrations_online():
    connectable = engine_from_config(
        config.get_section(config.config_ini_section),
        url=db_url,
        prefix='sqlalchemy.',
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            version_locations=f"migrations/{app_name}/",
            target_metadata=target_metadata,
            include_schemas=True,
            version_table_schema=app_name,
            version_table="db_migrations",
        )

        with context.begin_transaction():
            context.run_migrations()

With all the above I'm just setting the env vars for the app I work with right now using dotenv and issue alembic upgrade head or alembic revision --autogenerate from the py/ dir.

[zzzeek]

OK so the first thing is, when I mentioned "you can probably set this from env.py", I didn't tell you how. Can you try:

context.script.version_locations = "/your/path"

Because script.version_locations has already been set up by the time env.py is being invoked, however, it looks like it hasn't been used yet , so subsequent operations that look for versions should use the updated path. See if that works.

As for the recipe I linked, that can also work here but it would look a little different than what you planned. Your workflow is sort of like:

APP_NAME=<name> alembic <command>    # <--- env.py sets version_location using template

and the one we have in https://alembic.sqlalchemy.org/en/latest/cookbook.html#run-multiple-alembic-environments-from-one-ini-file is like:

alembic --name <name> <command>   # alembic sets version_location using section in .ini file

so you can get effectively the same idea with what's available but it wouldn't look exactly the same. each of the environments would also have its own "env.py", but as mentioned before each one can just have "from myapp import my_alembic_env", which is where you put all the normal context.configure() stuff. An advantage of the --name approach is that you can put all kinds of per-app configuration in your alembic.ini and it's declaratively spelled out rather than implemented in imperative code inside env.py.

but try the first part above and maybe you can do it the way you want anyway.

[zzzeek]

please reopen if you have more activity on this.

[alexykot]

Yes, will do. Python is not the priority stack for me at the moment, but it may become used again.

[Ivorforce]

This doesn't work since (at least) the config is using cache (memoized_property), as far as I can see.

[zzzeek]

This doesn't work since (at least) the config is using cache (memoized_property), as far as I can see.

just in the interests of coming up with a feature can you for now do a "script.__dict__.pop('_version_locations', None)" after setting the attribute, and confirm that's the only thing that holds this back ?

[zzzeek]

would include renaming _version_locations to _version_locations_absolute and creating a descriptor for .version_locations that would reset the memoization when it is updated