Document consumer crashes when mail authentication fails

[CkuT]

Hey,

Sometimes the document consumer is crashing when an error occurs in the mail consumption. I don't have the time to investigate at the moment :(

I think we need to catch this error and skip the mail consumption (or retry several seconds later ?) instead of leaving the consumer down.

The secret wasn't in the body
The secret wasn't in the body
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/.venvs/paperless/lib/python3.5/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/opt/.venvs/paperless/lib/python3.5/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/.venvs/paperless/lib/python3.5/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/.venvs/paperless/lib/python3.5/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/opt/paperless/src/documents/management/commands/document_consumer.py", line 96, in handle
    self.loop_inotify(mail_delta)
  File "/opt/paperless/src/documents/management/commands/document_consumer.py", line 144, in loop_inotify
    self.mail_fetcher.pull()
  File "/opt/paperless/src/documents/mail.py", line 185, in pull
    for message in self._get_messages():
  File "/opt/paperless/src/documents/mail.py", line 203, in _get_messages
    self._login()
  File "/opt/paperless/src/documents/mail.py", line 227, in _login
    login = self._connection.login(self._username, self._password)
  File "/usr/lib/python3.5/imaplib.py", line 582, in login
    raise self.error(dat[-1])
imaplib.error: b'authentication failed'

[ovv]

I think we need to catch this error and skip the mail consumption (or retry several seconds later ?) instead of leaving the consumer down.

Crashing the customer makes it obvious that there is something wrong, if it just skip mail consumption will user notice it when it's failing ?

[syntonym]

Crashing the consumer does not necessarily mean that the user notices it failing, only when the user is searching for a recently scanned document and it's not present. I think the consumer should stay up no matter what and log an error. Maybe the UI could be improved so that critical errors are displayed very dominantly? I think currently you have to look into the logs (or spot it under "recent actions") to notice a critical error?

[CkuT]

Maybe we could check at consumer start if the credentials are wrong ? My credentials are good, but I think my mail provider is not stable and is down sometimes, occuring this crash, forcing me to restart the consumer by hand.

I think whatever external network service is linked should not be crashing paperless if an error occurs.

[ovv]

I see two options:

1. Let the consumer crash on error and rely on docker/systemd/else to restart it when necessary
2. Ensure the consumer can recover from any crash

2 requires more work but would be the best solution IMO. As @syntonym said that would also require critical errors be more clearly visible in the UI

[jvstein]

This showed up in my logs this morning during a Gmail authentication outage, restarting the background worker 23 times in ~100 minutes. I wasn't using paperless at the time, I just got my alerts about the restarted containers and looked into it afterward.

For me, the email consumer is very much a secondary, nice to have, feature. I do most of my scanning directly into a folder consumed by paperless.

I could see this error being annoying if it happened when I was trying to scan documents using my normal process, though. I wouldn't expect even an internet outage to affect my normal process, but it would.