Cannot import local packages in the migration modules

[shahriarshm]

My project structure is something like this.

migrations/
   20220116125514_initial.py
src/
    models/
        __init__.py
        company.py
    __init__.py

My models:

# src/models/company.py

from beanie import Document

class OldCompany(Document):
    title: str

class Company(Document):
    name: str

My initial migration:

# migrations/20220116125514_initial.py

from beanie import iterative_migration

from src.models.company import Company, OldCompany

class Forward:
    @iterative_migration()
    async def title_to_name(self, input_document: OldCompany, output_document: Company):
        output_document.name = input_document.title
 
class Backward:
    @iterative_migration()
    async def name_to_title(self, input_document: Company, output_document: OldCompany):
        output_document.title = input_document.name

When I run migrate command: beanie migrate --uri 'db_uri' --db my_db --p home/shahriar/project/migrations

I get this error:

Building migration list
Traceback (most recent call last):
  File "/home/shahriar/project/entities/.venv/bin/beanie", line 8, in <module>
    sys.exit(migrations())
  File /home/shahriar/project/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahriar/project/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/shahriar/project.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/shahriar/project/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahriar/project/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/shahriar/project/.venv/lib/python3.8/site-packages/beanie/executors/migrate.py", line 153, in migrate
    asyncio.run(run_migrate(settings))
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/shahriar/project/.venv/lib/python3.8/site-packages/beanie/executors/migrate.py", line 77, in run_migrate
    root = await MigrationNode.build(settings.path)
  File "/home/shahriar/project/.venv/lib/python3.8/site-packages/beanie/migrations/runner.py", line 183, in build
    module = SourceFileLoader(
  File "<frozen importlib._bootstrap_external>", line 522, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1027, in load_module
  File "<frozen importlib._bootstrap_external>", line 852, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/shahriar/project/migrations/20220116125514_initial.py", line 5, in <module>
    from src.models.company import Company, OldCompany
ModuleNotFoundError: No module named 'src'

What's the problem?

[okaminoseishin]

Hi, @shahriarshm. This is because of how entire execution flow works. beanie command is actually a script in the .venv/bin directory, and it's invocation is the same as running python .venv/bin/beanie. In this form python does not add current directory to the sys.path (as opposite to -m and -c options) resulting in not importable local modules.

When you have desired package (src in your case) "installed" into your user site-packages, that module will be importable from migration file.

There are two solutions for your problem:

• modify sys.path and run migrations programmatically (mimic this method in your code)
• install desired package into your environment either with poetry or directly with pip install --editable

Project contributors don't encounter this problem because they effectively use second approach implicitly with poetry, but I suggest them to prepend sys.path with current working directory by the package itself, at least when aforementioned method is called through a CLI entrypoint.

EDIT: here is a sample of how migrations work in my project. No comments though, but it's pretty simple, and supports many versions in a nice way (eliminates the Model -> OldModel -> OldOldModel stuff). Maybe you will find some ideas useful.

beanie-migrations.zip

[sshghoult]

@okaminoseishin's answer is entirely correct, but I'd like to suggest another alternative. If you run the python interpreter from the directory of your repository that contains the package you need to import, it starts up with the correct place to search for the mentioned package already in the sys.path. Therefore, you can run python3 -m beanie.executors.migrate migrate from the directory containing your src package (most likely the root of your repo) to run the migrations. It's especially convenient in dockerized setups - both local and demo/prod.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity.