"sqlite3.OperationalError: database is locked" when running "cellphonedb database generate"

[chenxing-zhang]

Hello !

I got an error ”sqlite3.OperationalError: database is locked‘, when running "cellphonedb database generate --user-interactions interaction_input.csv --user-interactions-only".

I tried to run it on python3.7 and 3.8 on two linux computers(Ubuntu 16 and 18), and installed different versions of cellphonedb (from v2.1.7 to v2.0.0), but it couldn’t solve this problem. It has bothered me for two days, Do you have any suggestions?

p.s.1 the interaction_input.csv is downloaded from https://github.com/Teichlab/cellphonedb-data/tree/master/data. p.s.2 The error seems to appear in “app.database_manager.database.create_all()” in the “collect_database” function.

############################################################################################

(cpdb) gaolin@gaolin://192.168.0.201/e/desktop/cellphonedb_example_data/example_data$ cellphonedb database generate --user-interactions interaction_input.csv -- user-interactions-only /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.cluster.k_means_ module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.cluster. Anything that cannot be imported from sklearn.cluster is now part of the private API. warnings.warn(message, FutureWarning) read local uniprot file /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/tools_terminal_api_endpoints/tools_terminal_commands.py:243: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead 'secreted_desc': pd.np.nan, /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/tools_terminal_api_endpoints/tools_terminal_commands.py:246: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead 'receptor_desc': pd.np.nan, /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/tools_terminal_api_endpoints/tools_terminal_commands.py:249: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead 'other_desc': pd.np.nan, /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/tools_terminal_api_endpoints/tools_terminal_commands.py:251: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead 'tags_reason': pd.np.nan, /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/tools_terminal_api_endpoints/tools_terminal_commands.py:252: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead 'tags_description': pd.np.nan, /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/tools_terminal_api_endpoints/tools_terminal_commands.py:253: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead 'pfam': pd.np.nan, [ ][APP][26/06/21-12:57:36][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results [ ][APP][26/06/21-12:57:36][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results read local ensembl file read local uniprot file /home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/core/generators/gene_generator.py:31: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy ensembl_db_filtered.dropna(inplace=True) [ ][APP][26/06/21-12:57:51][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results [ ][APP][26/06/21-12:57:51][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results [ ][APP][26/06/21-12:57:51][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results [ ][APP][26/06/21-12:57:51][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results [ ][APP][26/06/21-12:57:52][WARNING] Output directory (/192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out) exist and is not empty. Result can overwrite old results [ ][CORE][26/06/21-12:57:52][INFO] Initializing SqlAlchemy CellPhoneDB Core [ ][CORE][26/06/21-12:57:52][INFO] Using custom database at /192.168.0.201/e/desktop/cellphonedb_example_data/example_data/out/cellphonedb_user_2021-06-26-12_57.db Traceback (most recent call last): File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/gaolin/anaconda3/envs/cpdb/bin/cellphonedb", line 8, in sys.exit(cli()) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/core.py", line 722, in call return self.main(*args, **kwargs) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/api_endpoints/terminal_api/database_terminal_api_endpoints/database_terminal_commands.py", line 114, in generate data_path=output_path) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/database/manager/DatabaseVersionManager.py", line 92, in collect_database app.database_manager.database.create_all() File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/cellphonedb/src/core/database/Database.py", line 25, in create_all self.base_model.metadata.create_all(self.engine) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4665, in create_all ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2095, in _run_visitor conn._run_visitor(visitorcallable, element, **kwargs) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1656, in _run_visitor visitorcallable(self.dialect, self, **kwargs).traverse_single(element) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 145, in traverse_single return meth(obj, **kw) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 787, in visit_metadata _is_metadata_operation=True, File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 145, in traverse_single return meth(obj, **kw) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 832, in visit_table include_foreign_key_constraints, # noqa File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection return connection._execute_ddl(self, multiparams, params) File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1073, in _execute_ddl compiled, File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in execute_context e, statement, parameters, cursor, context File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from=e File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise raise exception File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/home/gaolin/anaconda3/envs/cpdb/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: CREATE TABLE multidata_table ( id_multidata INTEGER NOT NULL, name VARCHAR NOT NULL, receptor BOOLEAN, receptor_desc VARCHAR, other BOOLEAN, other_desc VARCHAR, secreted_highlight BOOLEAN, secreted_desc VARCHAR, transmembrane BOOLEAN, secreted BOOLEAN, peripheral BOOLEAN, integrin BOOLEAN, is_complex BOOLEAN, PRIMARY KEY (id_multidata), UNIQUE (name), CHECK (receptor IN (0, 1)), CHECK (other IN (0, 1)), CHECK (secreted_highlight IN (0, 1)), CHECK (transmembrane IN (0, 1)), CHECK (secreted IN (0, 1)), CHECK (peripheral IN (0, 1)), CHECK (integrin IN (0, 1)), CHECK (is_complex IN (0, 1)) )

] (Background on this error at: http://sqlalche.me/e/13/e3q8)