[mu4e bug] Garbled headers when editing a saved msg with no body

[jman-schief]

When I edit a previously saved message that has no body, the email headers gets mixed up.

How to Reproduce

• Create a new email (with M-x mu4e-compose-new)
• Add something into the To: and Subject: fields. Don't add anything else.
• Save the email anche close the buffer
• Retrieve that email from the drafts and edit it again (M-x mu4e-compose-edit)
• Email header Message-ID appears mixed up with the text --text follows this line--

The text --text follows this line-- is somehow attached when the buffer with the editable email is created because it is not present in the saved draft.

If I add something in the body the issue does not reproduce.

Variant of this bug

• Create a new email (with M-x mu4e-compose-new)
• Save the draft without changing anything
• Retrieve that email from the drafts and edit it again (M-x mu4e-compose-edit)
• An error message will appear: Search failed: "^--text follows this line--$"

Environment

• Doom emacs
• mu/mu4e 1.12.5 (Debian package)
• Emacs 29.1

Checklist

• [x] you are running either an 1.10.x/1.12.x release or master (otherwise please upgrade)
• [ ] you can reproduce the problem without 3rd party extensions (including Doom/Evil, various extensions etc.)
• [x] you have read all of the above

Thank you!

[jman-schief]

I think also related to #2661

[djcb]

Unable to reproduce. Are you using Doom's special mu4e version?

[jman-schief]

Unable to reproduce. Are you using Doom's special mu4e version?

Thanks for investigating. I'm pretty sure that Doom Emacs does not customize mu4e (not anymore in any case). I install the mu4e Debian package and the mu4e configuration lives completely in my own elisp files.

When composing an email I don't see any major or minor mode that could be the culprit of interfering, here's the output of describe-mode:

Minor modes enabled in this buffer: Abbrev Apheleia Auto-Fill Auto-Save Better-Jumper-Local Company
Display-Line-Numbers Dtrt-Indent Eldoc Flymake Flymake-Popon Font-Lock Hl-Line Mml Mu4e-Context
Smartparens Undo-Fu-Session Vi-Tilde-Fringe Visual-Line Whitespace Ws-Butler Yas

The major mode is mu4e:compose mode defined in mu4e-compose.el

Unsure if it helps: I enabled toggle-debug-on-error and when I try the "Variant of this bug" (explained above) I see:

Debugger entered--Lisp error: (search-failed "^--text follows this line--$")
   message-position-on-field("Subject")
   message-goto-subject()
   mu4e--jump-to-a-reasonable-place()
   mu4e--prepare-draft-buffer(edit nil)
   mu4e--draft(edit #f(compiled-function () #<bytecode 0xc61a197c12512c4>))
   mu4e-compose-edit()
   funcall-interactively(mu4e-compose-edit)
   command-execute(mu4e-compose-edit)

If I have time, I'll try to install mu4e on a vanilla Emacs ...

[djcb]

I tried with emacs-29.4 as well as with 28.2 and no further customizations, and I can't reproduce the misbehavior. I.e.,

/usr/local/bin/emacs-29.4 -Q --eval "(progn (add-to-list 'load-path \"~/Sources/mu/build/mu4e\") (setq mu4e-mu-binary \"~/Sources/mu/build/mu/mu\") (require 'mu4e) (mu4e))"

can you try that as well? (change the paths as needed). Thanks.

[tsdh]

I think, I see the same bug (vanilla emacs master, mu4e master). Since updating to mbsync 1.5.0 from 1.4.x I get some sync errors (not fatal) where local messages cannot be uploaded to the far side IMAP server. Debugging showed that those are all caused by drafts where the --text follows this line-- separator hasn't been removed and thus the message appears to consist of only headers with invalid format and no body.

I don't usually save drafts, and I certainly didn't with the messages mbsync complains about. Oh, and the draft messages have filenames like these (note the #es):

.mail/Fastmail/Drafts/cur/#1725020107.a5c8b0748f9453b6.thinkpad-t440p,U=3:2,DS#
.mail/Fastmail/Drafts/cur/.#1725087352.7e237ef11c9863bc.thinkpad-t440p:2,DS
.mail/Fastmail/Drafts/cur/#1725087352.7e237ef11c9863bc.thinkpad-t440p:2,DS#
.mail/Fastmail/Drafts/cur/#1725020107.a5c8b0748f9453b6.thinkpad-t440p,U=2:2,DS#

I think they are auto-save files. I can create them just by starting a new mail and not sending or saving (maybe I need to switch buffers).

When I save the draft explicitly with C-x C-f, it will be saved in a similarly named file without the hashes and in that case the separator line --text follows this line-- is replaced with an empty line dividing headers and body, too, i.e. it becomes a valid email.

[djcb]

@tsdh those are indeed auto-save files, they shouldn't be written to the draft directory https://www.gnu.org/software/emacs/manual/html_node/elisp/Auto_002dSaving.html

The problem in this ticket is slightly different though, you re-load a draft and somewhere between loading it (correctly) and the user seeing it, something mangles the text.

[jman-schief]

ok @djcb I have a stable repro and I can't figure out what's going on.

I compose a new email using the latest master, then save. The *mu4e-article* buffer shows the following fields:

From: <something>
Subject: <something>
To: <something>
Date: Sun, 01 Sep 2024 18:30:41 +0200 (15 minutes, 42 seconds ago)
Fcc: /home/me/.local/mail/sent/cur/1725208225.3430cdb1e0538bc9.localhost:2,S
Maildir: /drafts

Editing this email is just fine.

I compose an email using 1.12.6 with my config, then save. The *mu4e-article* buffer shows up like this:

From: <something>
Subject: <something>
To: <something>
Date: Sun, 01 Sep 2024 18:34:13 +0200 (12 minutes, 52 seconds ago)
Maildir: /<email-domain>/Drafts
Fcc: /home/me/.local/mail/email-domain/Sent/cur/1725208441.214a25597516462e.localhost:2,S

If I edit this one, I can reproduce my issue. Seems that the Fcc: field is confusing but only when it's the last. In addition I don't understand why it has "sent" in the path.

Both files on the filesystem are almost identical (you just added a User-Agent field in the meanwhile). I can send them to you for inspection.

Is it enough for you to reproduce? Can you spot something wrong in my configuration? I've tried some bisection but I was unsuccessful.

thanks

[djcb]

Hmm, thing to check, /Drafts and /Sent must come from somewhere, the defaults are /drafts and /sent; so seems some other configuration is active too.

And then copy settings from your repro config into the bare-bones one until it reproduces (or comment out from the repro config until it no longer reproduces). That should hopefully get us closer.

[tsdh]

@djcb Alright, I have fixed my slightly different problem by evaluating (setq-local buffer-auto-save-file-name nil) in message-mode-hook.

[jman-schief]

Ok, after a fun bisection session, here's our culprit:

(add-hook 'before-save-hook 'whitespace-cleanup)

If I automatically cleanup whitespaces when saving buffers, this breaks something in mu4e. Unsure if it happens when saving the email or when reopening for edit :man_shrugging:

[djcb]

@tsdh: I pushed a change ot explicitly ignore emacs autosaves, which avoid parts of the problem at least.

[djcb]

@jman-schief: I could imagine that interferes with some of the mu4e handling. So, better don't do that. I'll add a note to the manual.

Good that you found the culprit, but it's time-consuming / impossible for me to investigate such problems, so in the future please start with the minimal setup, then add things until you can reproduce and then file a ticket. Saves time for all. Thanks.

Note that I did push a fix for #2661 mentioned here, but I think it's unrelated.

[jman-schief]

First of all @djcb thank you for patiently strong-arming me into finding the root of the problem. I am sorry for wasting more time than necessary.

I wish there was another reply other than "better don't do that". I use whitespace-cleanup a lot and I'll have to add an exception when composing emails but I don't have the guts (or the time) to dive into the mu4e source code, so I guess that is all for this issue.

Closing as WONTFIX.

[djcb]

@jman-schief: If you want to dig deeper, there mu4e--compose-before-save and mu4e--compose-after-save that are relevant to this. Start with a minimal setup and your own hooks and see what happens -- good luck!

[tsdh]

@jman-schief Using whitespace-cleanup in email messages sounds a bit dangerous. For example, the signature separator is literally -- \n, i.e., two dashes and a space on a separate line. I'm pretty sure whitespace-cleanup will eat that space. That's probably not very problematic. The receiver will just see the signature as normal text.

Looking at what whitespace-cleanup does, I think the problem is that it's documented to delete empty lines at the beginning and end of the buffer. In case where you haven't written any text in the mail but only filled the headers, the buffer probably ends with an empty line (where the ---text follows this line--- is shown). That's probably a significant trailing empty line.

Additionally, it seems whitespace-cleanup seems to adjust leading spaces in certain circumstances, too. That could also invalidate email headers where some RFC specifies how long headers have to be wrapped into multiple lines.

I guess that's essentially the same category as untabifying a Makefile. You can do that but then it's broken.

Of course, one could think about enhancing whitespace-cleanup such that it only performs safe cleanups in modes deriving from message-mode or mail-mode. You suggest that to the Emacs devs using M-x report-emacs-bug.