aiidateam
Migration error from old develop branch (dec 2021) to most recent develop (2.0a1)
I get this error when I run verdi -p default storage migrate:
Report: Migrating to the head of the legacy django branch
Traceback (most recent call last):
File "/Users/pizzi/.virtualenvs/aiida-dev/bin/verdi", line 8, in <module>
sys.exit(verdi())
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/Users/pizzi/git/aiida-core/aiida/cmdline/commands/cmd_storage.py", line 79, in storage_migrate
storage_cls.migrate(profile)
File "/Users/pizzi/git/aiida-core/aiida/storage/psql_dos/backend.py", line 63, in migrate
cls.migrator(profile).migrate()
File "/Users/pizzi/git/aiida-core/aiida/storage/psql_dos/migrator.py", line 241, in migrate
self.migrate_up('django@head')
File "/Users/pizzi/git/aiida-core/aiida/storage/psql_dos/migrator.py", line 262, in migrate_up
upgrade(config, version)
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/command.py", line 298, in upgrade
script.run_env()
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/script/base.py", line 489, in run_env
util.load_python_file(self.dir, "env.py")
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/pizzi/git/aiida-core/aiida/storage/psql_dos/migrations/env.py", line 49, in <module>
run_migrations_online()
File "/Users/pizzi/git/aiida-core/aiida/storage/psql_dos/migrations/env.py", line 42, in run_migrations_online
context.run_migrations() # pylint: disable=no-member
File "<string>", line 8, in run_migrations
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
self.get_context().run_migrations(**kw)
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/runtime/migration.py", line 510, in run_migrations
with self.begin_transaction(_per_migration=True):
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/alembic/runtime/migration.py", line 367, in begin_transaction
self._transaction = self.bind.begin()
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/sqlalchemy/future/engine.py", line 144, in begin
return super(Connection, self).begin()
File "/Users/pizzi/.virtualenvs/aiida-dev/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 766, in begin
raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: This connection has already initialized a SQLAlchemy Transaction() object via begin() or autobegin; can't call begin() here unless rollback() or commit() is called first.
Did anyone see this? (I'm at commit 1d36c0c47cd74c8423995c4e825cf57ae4363f93) @sphuber @chrisjsewell
Can't say that I have. It doesn't seem to be an error that might be related to a database that was previously migrated with the develop branch, but rather just a problem in the code. It's weird that we wouldn't have encountered it before though, but it is really @chrisjsewell who wrote the new migration code, so maybe he has an idea.
Yeh nothing comes immediately to mind I'm afraid, and seems strange that this would follow a different code path than any we have previously tested
One way to check, might be to manually add echo=True in the aiida-core code, to the create_engine function (https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine.params.echo).
Then you should get a print out of exactly what sqlalchemy is doing during verdi -p default storage migrate
