paperless
paperless copied to clipboard
Published
20 hours ago •
the-paperless-project
the-paperless-project
Failing to detect / edit documents (Database is Locked - docker)
I am running Paperless on docker (webserver and consumer containers). I can reach the WEB UI. Now I'm at the stage of trying to import some documents. From the documentation I seem to understand that I have to run document_consumer, but whenever I run
docker-compose run --rm paperless-consumer document_consumer
I get
Operations to perform:
Apply all migrations: admin, auth, contenttypes, documents, reminders, sessions
Running migrations:
No migrations to apply.
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tesseract-ocr-data-ita (4.1.0-r0)
OK: 305 MiB in 128 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tesseract-ocr-data-swe (4.1.0-r0)
OK: 318 MiB in 129 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tesseract-ocr-data-nld (4.1.0-r0)
OK: 340 MiB in 130 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tesseract-ocr-data-bul (4.1.0-r0)
OK: 348 MiB in 131 packages
Starting document consumer at /consume with inotify
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 305, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/paperless/src/manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python3.8/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python3.8/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/usr/src/paperless/src/documents/management/commands/document_consumer.py", line 85, in handle
logging.getLogger(__name__).info(
File "/usr/lib/python3.8/logging/__init__.py", line 1434, in info
self._log(INFO, msg, args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1577, in _log
self.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1587, in handle
self.callHandlers(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1649, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 950, in handle
self.emit(record)
File "/usr/src/paperless/src/documents/loggers.py", line 23, in emit
Log.objects.create(**kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 417, in create
obj.save(force_insert=True, using=self.db)
File "/usr/src/paperless/src/documents/models.py", line 353, in save
models.Model.save(self, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/base.py", line 728, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/lib/python3.8/site-packages/django/db/models/base.py", line 759, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/lib/python3.8/site-packages/django/db/models/base.py", line 842, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/lib/python3.8/site-packages/django/db/models/base.py", line 879, in _do_insert
return manager._insert([self], fields=fields, return_id=update_pk,
File "/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1128, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1285, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 305, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked
Note that I also get an error whenever I try and edit a document from the UI
Environment:
Request Method: POST
Request URL: http://192.168.1.200:4444/admin/documents/document/2/change/
Django Version: 2.0.13
Python Version: 3.8.2
Installed Applications:
['whitenoise.runserver_nostatic',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'corsheaders',
'django_extensions',
'paperless',
'documents.apps.DocumentsConfig',
'reminders.apps.RemindersConfig',
'paperless_tesseract.apps.PaperlessTesseractConfig',
'paperless_text.apps.PaperlessTextConfig',
'django.contrib.admin',
'rest_framework',
'crispy_forms',
'django_filters',
'djangoql']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py" in _execute
85. return self.cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py" in execute
305. return Database.Cursor.execute(self, query, params)
The above exception (database is locked) was the direct cause of the following exception:
File "/usr/lib/python3.8/site-packages/django/core/handlers/exception.py" in inner
35. response = get_response(request)
File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py" in _get_response
128. response = self.process_exception_by_middleware(e, request)
File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py" in _get_response
126. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3.8/site-packages/django/contrib/admin/options.py" in wrapper
575. return self.admin_site.admin_view(view)(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/utils/decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/contrib/admin/sites.py" in inner
223. return view(request, *args, **kwargs)
File "/usr/src/paperless/src/documents/admin.py" in change_view
238. return super(DocumentAdmin, self).change_view(
File "/usr/lib/python3.8/site-packages/django/contrib/admin/options.py" in change_view
1557. return self.changeform_view(request, object_id, form_url, extra_context)
File "/usr/lib/python3.8/site-packages/django/utils/decorators.py" in _wrapper
62. return bound_func(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/utils/decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/utils/decorators.py" in bound_func
58. return func.__get__(self, type(self))(*args2, **kwargs2)
File "/usr/lib/python3.8/site-packages/django/contrib/admin/options.py" in changeform_view
1451. return self._changeform_view(request, object_id, form_url, extra_context)
File "/usr/lib/python3.8/site-packages/django/contrib/admin/options.py" in _changeform_view
1491. self.save_model(request, new_object, form, not add)
File "/usr/lib/python3.8/site-packages/django/contrib/admin/options.py" in save_model
1027. obj.save()
File "/usr/lib/python3.8/site-packages/django/db/models/base.py" in save
728. self.save_base(using=using, force_insert=force_insert,
File "/usr/lib/python3.8/site-packages/django/db/models/base.py" in save_base
759. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/lib/python3.8/site-packages/django/db/models/base.py" in _save_table
822. updated = self._do_update(base_qs, using, pk_val, values, update_fields,
File "/usr/lib/python3.8/site-packages/django/db/models/base.py" in _do_update
872. return filtered._update(values) > 0
File "/usr/lib/python3.8/site-packages/django/db/models/query.py" in _update
712. return query.get_compiler(self.db).execute_sql(CURSOR)
File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py" in execute_sql
1379. cursor = super().execute_sql(result_type)
File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py" in execute_sql
1068. cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py" in execute
100. return super().execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py" in execute
68. return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py" in _execute_with_wrappers
77. return executor(sql, params, many, context)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py" in _execute
85. return self.cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/utils.py" in __exit__
89. raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py" in _execute
85. return self.cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py" in execute
305. return Database.Cursor.execute(self, query, params)
Exception Type: OperationalError at /admin/documents/document/2/change/
Exception Value: database is locked
Any idea what's causing this issue?
So far it seems that if I restart the entire container, then there is a chance some new docs get picked up, but it always ends with a "database is locked" situation.
If I check the consumer log, it seems to be stuck here
Processing sheet #1: /tmp/paperless/paperless-j8xqi972/convert-0001.pnm -> /tmp/paperless/paperless-j8xqi972/convert-0001.unpaper.pnm
Processing sheet #1: /tmp/paperless/paperless-j8xqi972/convert-0003.pnm -> /tmp/paperless/paperless-j8xqi972/convert-0003.unpaper.pnm
Processing sheet #1: /tmp/paperless/paperless-j8xqi972/convert-0002.pnm -> /tmp/paperless/paperless-j8xqi972/convert-0002.unpaper.pnm
Processing sheet #1: /tmp/paperless/paperless-j8xqi972/convert-0000.pnm -> /tmp/paperless/paperless-j8xqi972/convert-0000.unpaper.pnm
[pgm_pipe @ 0x561f845d7cc0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[pgm_pipe @ 0x55f85ec59cc0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[pgm_pipe @ 0x5578b197fcc0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[pgm_pipe @ 0x55912a95fcc0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[image2 @ 0x561f845d9100] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[image2 @ 0x561f845d9100] Encoder did not produce proper pts, making some up.
[image2 @ 0x55f85ec5b100] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[image2 @ 0x55f85ec5b100] Encoder did not produce proper pts, making some up.
out of deviation range - NO ROTATING
[image2 @ 0x5578b1981100] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[image2 @ 0x5578b1981100] Encoder did not produce proper pts, making some up.
[image2 @ 0x55912a961100] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[image2 @ 0x55912a961100] Encoder did not produce proper pts, making some up.
OCRing the document
Parsing for eng
Parsing for nld
Hey,
sorry if it's a dumb question but are you sure that there was no other consumer process running when you started the consumer manually?
Especially, are you sure, you didn't start your docker-compose file with docker-compose up ? Because that would automatically start to consume (and lock the db)
@bauerj not a dumb question, this is literally the first time I run this and the documentation has not been super clear for me. The answer is "I don't know, but probably yes". This is basically connected to @totti4ever 's post. I did use docker-compose up, I did not read in the documentation that putting the container up would automatically start to consume (and lock the db).
Tried to restart the containers but that didn't help. In the end I just removed the container and created new ones, and everything seemed to work just fine.
I am assuming from now on I should just dump the documents in the consumer folder and Paperless should pick them up automatically (with consume running all the time in the background) ?
(with consume running all the time in the background) ?
That is what is configured in the docker-compose.yaml.example: one container is started as consumer and one for the web ui
Im glad you figured it out!
I took a look at the documentation and could not immediately find the command (docker-compose run --rm paperless-consumer document_consumer) you ran. Where exactly did this occur to you?
Pointers on how to further improve the documentation would be great.
I thought it was just missing information in the documentation, so I sorta took the command from the Exporter, and adapted it for the consumer.
I suggest adding a Docker paragraph (like in the Exporter and the Importer) and just mention that after starting the consumer container, consumption will automatically start in the background.
Thanks for the help, I'll see if I stumble on more issue and maybe I'll try and contribute to the documentation to make the onboarding a bit clearer