Exception thrown when placing outbox email in a directory outside inbox

[samdmarshall]

I have an alot config that looks like this:

[accounts]
    [[default]]
        sent_box = maildir:///home/samantha/eMail/Outbox/
        draft_box = maildir:///home/samantha/eMail/Drafts/

I store my email in a directory structure that looks like this:

o /home/samantha/eMail
|
+-o Inbox
| +-o new
| +-o cur
| +-o tmp
|
+-o Outbox
| +-o new
| +-o cur
| +-o tmp
|
+-o Drafts
  +-o new
  +-o cur
  +-o tmp

All three Inbox, Outbox, and Draft directories are setup in maildir format, and configured such that none overlap in contents.

• Inbox is for all my inbound email, which is fetched by getmail
• Outbox is for all outgoing email, sent by msmtp via alot
• Drafts is where email that is being composed in alot lives in temporary form

When I compose and then send an email via alot it seems an exception gets thrown and then handled (no crash) due to my Outbox email box not being a subdirectory of the Inbox. So far this has caused no issues for me but it would be nice to not have this exception get thrown this way.

I don't know if i can capture the backtrace properly because it gets drawn over rather quickly after getting printed to screen. Let me know if you need any more info.

[pazz]

Hi samantha, welcome to alot and thanks for the bug report :) Yes, further info from the logfile would be helpful here. You can start alot, for instance with

alot -d debug -l logfile

and trigger the bug. Please post the (interesting suffix of the) logfile here.

[samdmarshall]

Ok, i had alot create the log file like that but it doesn't contain anything about the exception; i was luckily able to capture the output this time:

Unhandled error in Deferred:


Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_baseprocess.py", line 64, in maybeCallProcessEnded
    proto.processEnded(Failure(reason))
  File "/usr/share/alot/alot/helper.py", line 352, in processEnded
    self.deferred.callback(out)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 393, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/share/alot/alot/commands/envelope.py", line 252, in afterwards
    ui.dbman.add_message(path, account.sent_tags + initial_tags)
  File "/usr/share/alot/alot/db/manager.py", line 428, in add_message
    raise DatabaseError(msg)
alot.db.errors.DatabaseError: message path /home/samantha/eMail/Outbox/new/1501076831.M362598P14326Q1.queersorceress.email:2,S  is not below notmuchs root path (/home/samantha/eMail/Inbox/)

[pazz]

right: So the fact that this does not show up in the logfile should be a bug.

What happens here is that alot tries to index the newly send mail after sending, which fails because notmuch does not know about that folder.

Please make sure that your draft-mailbox is below the notmuch database root directory: check notmuch config get database.path..

[samdmarshall]

the result of that command is: /home/samantha/eMail/Inbox/, i don't want a draft or outgoing email to be indexed against my inbox database, which is why they are not under the /Inbox/ directory. Alot does the correct thing when writing out draft and outgoing emails. If i were to change that, then those emails could be incorrectly categorized by my tagging rules.

[pazz]

I'm not sure about your tagging rules, but most people would index all their mails, including the ones they sent themselves (otherwise threads wont show your replies for instance).

Remember that in notmuch, the "inbox" tag is completely independent of where the mail file lives, so you don't need separate notmuch databases for incoming and outgoing mails. Your tagging script could add the "inbox" tag only to mails in the Inbox directory for example, and you then simply search you database for "tag:inbox" to get that view on it.

All of this is independent of the bug here: of course alot should properly report this exception. I do think however, that a warning is appropriate, because most people will want to index their outbox and might have misconfigured notmuch.

[samdmarshall]

that is fair; i don't want that because of my unique tagging rules. I just want to not have this exception dumped to screen each time i send an email :)

[pazz]

would you mind checking if this is still an issue on current master? I mean that this exception appears in the logs but does not trigger a notification? We've been restructuring alot to use asyncio instead of twisted's deferreds, so your issue might have been solved accidentally in the process.. thanks!

[samdmarshall]

I haven't been able to update to 0.8 yet, so i'll update this again once I can do that.

[samdmarshall]

@pazz hey so i'm trying to check out the latest version from HEAD and i am having difficulties with getting it to run. it seems to build okay but when i go to run the 0.8 executable i get the following error:

Traceback (most recent call last): 
  File "/home/demi/.local/bin/alot", line 11, in <module>
    load_entry_point('alot', 'console_scripts', 'alot')()
  File "/home/linuxbrew/.linuxbrew/lib/python3.7/site-packages/pkg_resources/__init__.py", line 487, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name) 
  File "/home/linuxbrew/.linuxbrew/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2728, in load_entry_point
    return ep.load() 
  File "/home/linuxbrew/.linuxbrew/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2346, in load 
    return self.resolve() 
  File "/home/linuxbrew/.linuxbrew/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2352, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0) 
  File "/home/demi/Projects/alot/alot/__main__.py", line 10, in <module>
    from alot.settings.const import settings 
  File "/home/demi/Projects/alot/alot/settings/const.py", line 4, in <module>
    from .manager import SettingsManager 
  File "/home/demi/Projects/alot/alot/settings/manager.py", line 17, in <module>
    from ..utils import configobj as checks 
  File "/home/demi/Projects/alot/alot/utils/configobj.py", line 13, in <module>
    from .. import crypto 
  File "/home/demi/Projects/alot/alot/crypto.py", line 5, in <module>
    import gpg 
  File "/home/demi/.local/lib/python3.7/site-packages/gpg-1.10.0-py3.7-linux-x86_64.egg/gpg/__init__.py", line 101, in <module>
    from . import core
  File "/home/demi/.local/lib/python3.7/site-packages/gpg-1.10.0-py3.7-linux-x86_64.egg/gpg/core.py", line 34, in <module>
    from . import gpgme 
ImportError: cannot import name 'gpgme' from 'gpg' (/home/demi/.local/lib/python3.7/site-packages/gpg-1.10.0-py3.7-linux-x86_64.egg/gpg/__init__.py)

I don't know if this has to do with conflicts on my system with alot 0.7 installed with python 2 via linuxbrew or what but i won't be able to check on the sending-mail crash until i get past this. if you want me making a separate ticket for this then i can do that.

[pazz]

Thanks for the bug report. It seems that you are running pygpgme, and not the (relatively new) lib python-gpg from the official gnupg project. See https://github.com/pazz/alot/pull/1086 and https://github.com/pazz/alot/issues/1334 It also looks like you are running all this on a mac, and I have no expertise about packaging the above there..

[samdmarshall]

i use to be running this on a mac; currently on a surface tablet in the WSL environment. i’ll try to take a look at the linked issues and see if i can resolve the 0.8 install problem and get an answer about the exception thrown with sending mail.

[samdmarshall]

hey, i’m trying to follow up after a break on this and i’m not sure what i need to do to move forward. the linked issues seem to indicate that alot wants to use the python gpg library instead of gpgme, and my install seems to be interfacing with gpgme instead of the library it should?? is that what seems to be going on?

[pazz]

why don't you try removing python-gpgme from your system? it's deprecated anyway if I remember correctly..

[samdmarshall]

alright so I just sent a test email to myself and enabled debug logging to see what happens. After composing and sending the message I am brought back to my default screen of mail tagged as "unread" and I see this:

however when I check the log file:

debug log contents

INFO:utils:Unknown values were found in `/home/demi/.config/alot/config`. Please check for typos if a specified setting does not seem to work:
bindings: bufferlist
bindings: search
bindings: envelope
bindings: taglist
bindings: thread
DEBUG:manager:unable to load hooks file:/home/demi/.config/alot/hooks.py
DEBUG:manager:template directory: `/home/demi/.config/alot/templates`
DEBUG:manager:themes directory: `/home/demi/.config/alot/themes`
INFO:utils:Unknown values were found in `/home/demi/.config/alot/themes/tomorrow`. Please check for typos if a specified setting does not seem to work:
error
16_text_fg
16_text_bg
256_text_fg
256_text_bg
16_alternate
256_alternate
16_gray_bg
256_gray_bg
16_normal_fg
16_normal_bg
256_normal_fg
256_normal_bg
16_focus_fg
16_focus_bg
256_focus_fg
256_focus_bg
16_unread_fg
16_unread_bg
256_unread_fg
256_unread_bg
16_tags_fg
16_tags_bg
256_tags_fg
256_tags_bg
16_author_fg
16_author_bg
256_author_fg
256_author_bg
16_date_fg
16_date_bg
256_date_fg
256_date_bg
16_mail_fg
16_mail_bg
256_mail_fg
256_mail_bg
16_header_fg
16_header_bg
256_header_fg
256_header_bg
DEBUG:manager:abook defined: {u'type': u'shellcommand', u'command': u'khard email --parsable', u'regexp': u'^(?P[^@]+@[^\\t]+)\\t+(?P[^\\t]+)', u'ignorecase': True, 'abook_contacts_file': '/home/demi/.abook/addressbook', 'shellcommand_external_filtering': True}
INFO:ui:setup gui in 256 colours
DEBUG:ui:fire first command
DEBUG:ui:global command string: "search tag:unread"
DEBUG:__init__:mode:global got commandline "search tag:unread"
DEBUG:__init__:ARGS: [u'search', u'tag:unread']
DEBUG:__init__:cmd parms {'sort': None, 'query': [u'tag:unread']}
DEBUG:manager:Worker process 5989 spawned
DEBUG:manager:Worker process 5989 exited successfully
DEBUG:utils:unquoted header: |Craig Maloney |
DEBUG:utils:unquoted header: |Lyft Ride Receipt |
DEBUG:utils:unquoted header: |Hugo Locurcio |
DEBUG:utils:unquoted header: |[email protected]|
DEBUG:utils:unquoted header: |Andreas Rumpf |
DEBUG:utils:unquoted header: |Zireael07 |
DEBUG:utils:unquoted header: |Zireael07 |
DEBUG:utils:unquoted header: |EaseUS |
DEBUG:walker:EMPTY PIPE
DEBUG:ui:Got key ([':'], [58])
DEBUG:ui:cmdline: 'prompt'
DEBUG:ui:search command string: "prompt"
DEBUG:__init__:mode:search got commandline "prompt"
DEBUG:__init__:ARGS: [u'prompt']
DEBUG:__init__:cmd parms {'startwith': ''}
INFO:globals:open command shell
DEBUG:ui:Got key (['c'], [99])
DEBUG:ui:Got key (['o'], [111])
DEBUG:ui:Got key (['m'], [109])
DEBUG:ui:Got key (['p'], [112])
DEBUG:ui:Got key (['o', 's'], [111, 115])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['enter'], [10])
DEBUG:globals:CMDLINE: compose
DEBUG:ui:search command string: "compose"
DEBUG:__init__:mode:search got commandline "compose"
DEBUG:__init__:ARGS: [u'compose']
DEBUG:__init__:cmd parms {'spawn': None, 'sender': None, 'tags': None, 'cc': None, 'omit_signature': False, 'rest': [], 'bcc': None, 'to': None, 'template': None, 'attach': None, 'subject': None}
DEBUG:envelope:TEMPLATE: None
DEBUG:globals:True
DEBUG:globals:[]
DEBUG:ui:Got key (['m'], [109])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['@'], [64])
DEBUG:ui:Got key (['d'], [100])
DEBUG:ui:Got key (['a'], [97])
DEBUG:ui:Got key (['m'], [109])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['m'], [109])
DEBUG:ui:Got key (['a'], [97])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['backspace'], [127])
DEBUG:ui:Got key (['h'], [104])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['l'], [108])
DEBUG:ui:Got key (['l'], [108])
DEBUG:ui:Got key (['o'], [111])
DEBUG:ui:Got key (['@'], [64])
DEBUG:ui:Got key (['p'], [112])
DEBUG:ui:Got key (['e', 'w'], [101, 119])
DEBUG:ui:Got key (['p'], [112])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['w'], [119])
DEBUG:ui:Got key (['t'], [116])
DEBUG:ui:Got key (['h', 'e'], [104, 101])
DEBUG:ui:Got key (['s'], [115])
DEBUG:ui:Got key (['p'], [112])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['l'], [108])
DEBUG:ui:Got key (['l'], [108])
DEBUG:ui:Got key (['s'], [115])
DEBUG:ui:Got key (['.'], [46])
DEBUG:ui:Got key (['c', 'o'], [99, 111])
DEBUG:ui:Got key (['m'], [109])
DEBUG:ui:Got key (['enter'], [10])
DEBUG:ui:Got key (['t'], [116])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['s'], [115])
DEBUG:ui:Got key (['t'], [116])
DEBUG:ui:Got key (['i'], [105])
DEBUG:ui:Got key (['n'], [110])
DEBUG:ui:Got key (['g'], [103])
DEBUG:ui:Got key (['enter'], [10])
DEBUG:globals:SUBJECT: "testing"
DEBUG:globals:No encryption by default, encrypt_by_default=none
INFO:envelope:editable headers: set(['To', 'From', 'Subject'])
DEBUG:globals:using editor_cmd: emacs
DEBUG:globals:{'spawn: ': False, 'in_thread': False}
DEBUG:globals:{'spawn': False}
DEBUG:globals:cmdlist: ['emacs', '/tmp/alot.uD9RBo.eml']
INFO:globals:calling external command: ['emacs', '/tmp/alot.uD9RBo.eml']
DEBUG:envelope:GoT: """
To: [email protected]
From: Samantha Demi 
Subject: testing
this is an email to test if alot throws and exception on sending a message and placing it in another directory not under the notmuch DB
"""
DEBUG:ui:Got key ([':'], [58])
DEBUG:ui:cmdline: 'prompt'
DEBUG:ui:envelope command string: "prompt"
DEBUG:init:mode:envelope got commandline "prompt"
DEBUG:init:ARGS: [u'prompt']
DEBUG:init:cmd parms {'startwith': ''}
INFO:globals:open command shell
DEBUG:ui:Got key (['s'], [115])
DEBUG:ui:Got key (['e'], [101])
DEBUG:ui:Got key (['n'], [110])
DEBUG:ui:Got key (['d'], [100])
DEBUG:ui:Got key (['enter'], [10])
DEBUG:globals:CMDLINE: send
DEBUG:ui:envelope command string: "send"
DEBUG:init:mode:envelope got commandline "send"
DEBUG:init:ARGS: [u'send']
DEBUG:init:cmd parms {}
DEBUG:envelope:signing plaintext: Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
this is an email to test if alot throws and exception on sending a message =
and placing it in another directory not under the notmuch DB
DEBUG:envelope:FROM: "Samantha Demi [email protected]"
DEBUG:envelope:ACCOUNT: "[email protected]"
DEBUG:helper:ENV = {'FZF_DEFAULT_COMMAND': "pt --hidden --home-ptignore -g=''", 'KITTY_WINDOW_ID': '6', 'COMPOSE_CONVERT_WINDOWS_PATHS': 'true', 'XCURSOR_SIZE': '48', 'DOCKER_HOST': 'tcp://127.0.0.1:2375', 'NOTMUCH_CONFIG': '/home/demi/.config/notmuch/notmuch-config', 'LOGNAME': 'demi', 'USER': 'demi', 'NVM_DIR': '/home/demi/.nvm', 'HOME': '/home/demi', 'PATH': '/home/demi/.pyenv/shims:/home/demi/.local/bin:/home/demi/.nimble/bin:/home/linuxbrew/.linuxbrew/sbin:/home/linuxbrew/.linuxbrew/bin:/home/demi/.config/scripts:/home/demi/.gem/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'DISPLAY': '127.0.0.1:0', 'LANG': 'en_US.utf8', 'TERM': 'xterm', 'SHELL': '/home/linuxbrew/.linuxbrew/bin/fish', 'PYENV_SHELL': 'fish', 'SHLVL': '2', 'FZF_DEFAULT_OPTS': '--preview="preview --metadata {} "', 'GIT_CONFIG': '/home/demi/.config/git/config', 'PLATFORM_NAME': 'Linux+WSL', 'WINDOWID': '8388621', 'EDITOR': 'emacs', 'TERMINFO': '/home/demi/Projects/kitty/terminfo', 'XDG_CONFIG_HOME': '/home/demi/.config', 'PLATFORM_OS': 'GNU/Linux', 'DBUS_SESSION_BUS_PID': '5421', 'PULSE_SERVER': 'tcp://127.0.0.1', 'HOSTTYPE': 'x86_64', 'WSLENV': '', 'GPG_TTY': '/dev/pts/5', 'DBUS_SESSION_BUS_ADDRESS': 'unix:abstract=/tmp/dbus-n7Ch84KKfE,guid=55ff84b067374e02b9cfbb535c8687aa', 'PLATFORM_ARCH': 'x86_64', 'NAME': 'Tolva', 'PWD': '/home/demi', 'COLORTERM': 'truecolor', 'PAGER': 'w3m', 'GEM_HOME': '/home/demi/.gem', 'DBUS_SESSION_BUS_WINDOWID': '8388609'}
DEBUG:helper:CMD = ['msmtp', '--file', '/home/demi/.config/msmtp/msmtprc', '--read-recipients', '--read-envelope-from']
DEBUG:helper:writing to stdin
INFO:account:sent mail successfully
INFO:account:
DEBUG:envelope:mail sent successfully
INFO:ui:closing current buffer [envelope] to: [email protected]
DEBUG:account:Maildir
DEBUG:account:got mailbox msg id : 1552320695.M805393P5966Q1.Tolva
DEBUG:account:path of saved msg: /home/demi/eMail/Outbox/new/1552320695.M805393P5966Q1.Tolva:2,S
DEBUG:envelope:adding new mail to index

it contains no hint of a problem.

also as requested, I did check my system for gpgme and I think I have found the problem to be coming from the install formula used by the package manager "brew". what I am thinking of doing now is to again try to install from source and try to use that going forward instead of relying on a more convenient distribution and update method. I will update this ticket again once I've been able to do that and see if the problem still persists.