Writing emails with non ascii header values mangles them

[JCapucho]

Hi, when testing out himalaya I've noticed that when sending emails that contain non ascii data on the headers it gets mangled, for example the following email

Content-Type: text/plain; charset=utf-8
From: "ã" <[email protected]>
To: 
Subject: 

will have the From header value changed to "ã" <[email protected]>

[soywod]

Really strange, I was not able to reproduce. I have such non ascii char in my name é and it works well. Could you give a bit more context (how you sent, which version, which OS, env etc)?

[JCapucho]

The first mail I sent was to myself and I noticed my name was displaying weirdly in the non ascii characters, after looking at the trace log I noticed this line

[2022-05-28T21:59:07Z TRACE himalaya::msg::msg_entity] header value: "\"João Capucho\" <[email protected]>"

João Capucho should have been João Capucho and editing again displayed the name as normal, only sending it seems to keep it mangled.

I'm using the latest master commit, freshly compiled with cargo, my OS is NixOS 22.05 and my $LANG is en_US.UTF-8

[soywod]

Could you share your config and the email your sent? So I can try to reproduce the bug locally.

[JCapucho]

config

name = ""
notify-cmd = "/nix/store/vdywxsw4j0ar1293j6jb5fl2hbjy5w27-libnotify-0.7.12/bin/notify-send"
notify-query = "not seen"

[main]
default = true
draft-folder = "Drafts"
email = "[email protected]"
imap-host = "imap.gmail.com"
imap-login = "[email protected]"
imap-passwd-cmd = "'kwallet-query' '-f' 'email' '-r' 'gmail-password' 'kdewallet'"
imap-port = 993
imap-starttls = false
inbox-folder = "Inbox"
name = "João Capucho"
sent-folder = "Sent"
smtp-host = "smtp.gmail.com"
smtp-login = "[email protected]"
smtp-passwd-cmd = "'kwallet-query' '-f' 'email' '-r' 'gmail-password' 'kdewallet'"
smtp-port = 465
smtp-starttls = false

email

Content-Type: text/plain; charset=utf-8
From: "João Capucho" <[email protected]>
To: [email protected]
Subject: test

Log

[2022-05-29T14:21:49Z INFO  himalaya::compl::compl_args] entering completion command matcher
[2022-05-29T14:21:49Z INFO  himalaya::config::deserialized_config] begin: try to parse config from path
[2022-05-29T14:21:49Z DEBUG himalaya::config::deserialized_config] path: None
[2022-05-29T14:21:49Z INFO  himalaya::config::deserialized_config] end: try to parse config from path
[2022-05-29T14:21:49Z TRACE himalaya::config::deserialized_config] config: DeserializedConfig { name: "", downloads_dir: None, signature: None, signature_delimiter: None, default_page_size: None, notify_cmd: Some("/nix/store/vdywxsw4j0ar1293j6jb5fl2hbjy5w27-libnotify-0.7.12/bin/notify-send"), notify_query: Some("not seen"), watch_cmds: None, accounts: {"main": Imap(DeserializedImapAccountConfig { name: Some("João Capucho"), downloads_dir: None, signature: None, signature_delimiter: None, default_page_size: None, notify_cmd: None, notify_query: None, watch_cmds: None, format: None, read_headers: [], default: Some(true), email: "[email protected]", smtp_host: "smtp.gmail.com", smtp_port: 465, smtp_starttls: Some(false), smtp_insecure: None, smtp_login: "[email protected]", smtp_passwd_cmd: "'kwallet-query' '-f' 'email' '-r' 'gmail-password' 'kdewallet'", pgp_encrypt_cmd: None, pgp_decrypt_cmd: None, mailboxes: {}, hooks: None, imap_host: "imap.gmail.com", imap_port: 993, imap_starttls: Some(false), imap_insecure: None, imap_login: "[email protected]", imap_passwd_cmd: "'kwallet-query' '-f' 'email' '-r' 'gmail-password' 'kdewallet'" })} }
[2022-05-29T14:21:49Z INFO  himalaya::config::account_config] begin: parsing account and backend configs from config and account name
[2022-05-29T14:21:49Z DEBUG himalaya::config::account_config] account name: "default"
[2022-05-29T14:21:49Z TRACE himalaya::config::account_config] account config: AccountConfig { name: "main", default: true, display_name: "João Capucho", email: "[email protected]", downloads_dir: "/tmp", sig: None, default_page_size: 10, notify_cmd: None, notify_query: "not seen", watch_cmds: [], format: Auto, read_headers: [], mailboxes: {}, hooks: Hooks { pre_send: None }, smtp_host: "smtp.gmail.com", smtp_port: 465, smtp_starttls: false, smtp_insecure: false, smtp_login: "[email protected]", smtp_passwd_cmd: "'kwallet-query' '-f' 'email' '-r' 'gmail-password' 'kdewallet'", pgp_encrypt_cmd: None, pgp_decrypt_cmd: None }
[2022-05-29T14:21:49Z TRACE himalaya::config::account_config] backend config: Imap(ImapBackendConfig { imap_host: "imap.gmail.com", imap_port: 993, imap_starttls: false, imap_insecure: false, imap_login: "[email protected]", imap_passwd_cmd: "'kwallet-query' '-f' 'email' '-r' 'gmail-password' 'kdewallet'" })
[2022-05-29T14:21:49Z INFO  himalaya::config::account_config] end: parsing account and backend configs from config and account name
[2022-05-29T14:21:49Z INFO  himalaya::backends::imap::imap_args] entering imap command matcher
[2022-05-29T14:21:49Z INFO  himalaya::config::account_args] >> account command matcher
[2022-05-29T14:21:49Z INFO  himalaya::config::account_args] << account command matcher
[2022-05-29T14:21:49Z INFO  himalaya::mbox::mbox_args] entering mailbox command matcher
[2022-05-29T14:21:49Z INFO  himalaya::msg::msg_args] entering message command matcher
[2022-05-29T14:21:49Z INFO  himalaya::msg::msg_args] write command matched
[2022-05-29T14:21:49Z DEBUG himalaya::msg::msg_args] attachments paths: []
[2022-05-29T14:21:49Z DEBUG himalaya::msg::msg_args] encrypt: false
[2022-05-29T14:21:49Z INFO  himalaya::msg::msg_entity] start editing with editor
[2022-05-29T14:21:49Z TRACE himalaya::msg::msg_utils] local draft path: "/tmp/himalaya-draft.eml"
[2022-05-29T14:21:49Z DEBUG html5ever::tokenizer] processing in state Data
[2022-05-29T14:21:49Z DEBUG html5ever::tokenizer] got characters None
[2022-05-29T14:21:49Z DEBUG html5ever::tokenizer] processing EOF in state Data
[2022-05-29T14:21:49Z DEBUG html5ever::tree_builder] processing EOFToken in insertion mode InBody
[2022-05-29T14:21:49Z TRACE himalaya::msg::msg_entity] template: "Content-Type: text/plain; charset=utf-8\nFrom: \"João Capucho\" <[email protected]>\nTo: \nSubject: \n\n\n"
[2022-05-29T14:21:49Z TRACE himalaya::msg::msg_utils] local draft path: "/tmp/himalaya-draft.eml"
[2022-05-29T14:21:49Z DEBUG himalaya::ui::editor] create draft
[2022-05-29T14:21:49Z DEBUG himalaya::ui::editor] open editor
[2022-05-29T14:22:47Z DEBUG himalaya::ui::editor] read draft
[2022-05-29T14:22:47Z INFO  himalaya::msg::msg_entity] begin: building message from template
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] template: "Content-Type: text/plain; charset=utf-8\r\nFrom: \"João Capucho\" <[email protected]>\r\nTo: [email protected]\r\nSubject: test\r\n\r\n\r\n"
[2022-05-29T14:22:47Z INFO  himalaya::msg::msg_entity] end: building message from template
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] >> build message from parsed mail
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] parsed mail: ParsedMail { raw_bytes: [67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 58, 32, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 59, 32, 99, 104, 97, 114, 115, 101, 116, 61, 117, 116, 102, 45, 56, 13, 10, 70, 114, 111, 109, 58, 32, 34, 74, 111, 195, 163, 111, 32, 67, 97, 112, 117, 99, 104, 111, 34, 32, 60, 106, 99, 97, 112, 117, 99, 104, 111, 55, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 62, 13, 10, 84, 111, 58, 32, 106, 99, 97, 112, 117, 99, 104, 111, 55, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 13, 10, 83, 117, 98, 106, 101, 99, 116, 58, 32, 116, 101, 115, 116, 13, 10, 13, 10, 13, 10], header_bytes: [67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 58, 32, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 59, 32, 99, 104, 97, 114, 115, 101, 116, 61, 117, 116, 102, 45, 56, 13, 10, 70, 114, 111, 109, 58, 32, 34, 74, 111, 195, 163, 111, 32, 67, 97, 112, 117, 99, 104, 111, 34, 32, 60, 106, 99, 97, 112, 117, 99, 104, 111, 55, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 62, 13, 10, 84, 111, 58, 32, 106, 99, 97, 112, 117, 99, 104, 111, 55, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 13, 10, 83, 117, 98, 106, 101, 99, 116, 58, 32, 116, 101, 115, 116, 13, 10, 13, 10], headers: [MailHeader { key: "Content-Type", value: "text/plain; charset=utf-8" }, MailHeader { key: "From", value: "\"João Capucho\" <[email protected]>" }, MailHeader { key: "To", value: "[email protected]" }, MailHeader { key: "Subject", value: "test" }], ctype: ParsedContentType { mimetype: "text/plain", charset: "utf-8", params: {"charset": "utf-8"} }, body_bytes: [13, 10], subparts: [] }
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] >> parse header MailHeader { key: "Content-Type", value: "text/plain; charset=utf-8" }
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header key: "Content-Type"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header value: "text/plain; charset=utf-8"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] << parse header
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] >> parse header MailHeader { key: "From", value: "\"João Capucho\" <[email protected]>" }
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header key: "From"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header value: "\"João Capucho\" <[email protected]>"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] << parse header
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] >> parse header MailHeader { key: "To", value: "[email protected]" }
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header key: "To"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header value: "[email protected]"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] << parse header
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] >> parse header MailHeader { key: "Subject", value: "test" }
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header key: "Subject"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] header value: "test"
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] << parse header
[2022-05-29T14:22:47Z TRACE himalaya::msg::msg_entity] message: Msg { id: 0, subject: "test", from: Some(MailAddrList([Single(SingleInfo { display_name: Some("João Capucho"), addr: "[email protected]" })])), reply_to: None, to: Some(MailAddrList([Single(SingleInfo { display_name: None, addr: "[email protected]" })])), cc: None, bcc: None, in_reply_to: None, message_id: None, headers: {"content-type": "text/plain; charset=utf-8"}, date: None, parts: Parts([TextPlain(TextPlainPart { content: "\r\n" })]), encrypt: false, raw: [] }
[2022-05-29T14:22:47Z INFO  himalaya::msg::msg_entity] << build message from parsed mail

[ctem]

In case it helps, I reproduced this on 0.5.10 from NixOS 22.05.

For example, with content Aa Áá あぁ in both subject and body, the non-ASCII chars are mojibake’d in the subject (displaying as Aa Ãá ãã in locale en_US.UTF-8) but display correctly in the body.

The raw subject line of the received message reads as follows:

Subject: Aa =?utf-8?b?w4PCgcODwqEgw6PCgcKCw6PCgcKB?=

For comparative purposes, here is the raw subject line (for the correctly displaying text) from a message sent with a different client:

Subject: Aa =?utf-8?b?w4HDoSDjgYLjgYE=?=

Thank you for your work on this lovely project!

[soywod]

I was able to reproduce and to understand why it happens. There is 2 problems:

• The first one is when you write an email using the commands write, reply or forward: the headers are not parsed correctly from utf-8 but the body is good, I need to investigate why.
• The seconde one is when you send raw emails using the commands send or template send, and this is "normal" because those commands expect valid MIME emails. Which means clients (library consumers) have the responsibility to produce valid MIME emails, and this is not good. It will be fixed thanks to https://github.com/soywod/himalaya/issues/341, and it may fix the first point also.

[naba-nyan]

Hi, I found the same issue happens in himalaya-vim plugin. In the case of vim plugin, non ascii data on the headers are mojibake’d, and also so are body data.

OS: macOS 12.6 (locale en_US.UTF-8) himalaya: 0.6.1 neovim: 0.8.0

Here are email data (with concealed personal information).

[My vim preview] 
Content-Type: text/plain; charset=utf-8
From: 
To: 
Subject: test テスト

aaa
あああ

[Received mail]
<Subject>
test テスト 
<Body>
aaa
あああ
'

[Raw data] 
Subject: test =?utf-8?b?w6PCg8KGw6PCgsK5w6PCg8KI?=
From:
To: 
MIME-Version: 1.0
Date: Mon, 17 Oct 2022 14:04:20 +0000
Content-Type: multipart/mixed;
 boundary="vzmD7aOV1rmi7fAIKYSgxIiEwNDyhrf1JmoDeFq2"

--vzmD7aOV1rmi7fAIKYSgxIiEwNDyhrf1JmoDeFq2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64

YWFhDQrDo8KB4oCaw6PCgeKAmsOjwoHigJoNCic=
--vzmD7aOV1rmi7fAIKYSgxIiEwNDyhrf1JmoDeFq2--

[soywod]

The himalaya-vim plugin uses the CLI behind the scene, so once the CLI is fixed it should fix all UIs depending on it. I am actually working on #341 which should fix the issue, I hope to finish at the end of the week. I let you know!