ingest-file
ingest-file copied to clipboard
alephdata
e-mail messages with application/rtf body are imported as attachments, not message body
While importing an e-mail-archive in the (IMHO cursed) .PST-format, I came across a mailbox having all application/rtf for body type.
Content-Type: application/rtf
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename*=utf-8''rtf-body.rtf;
filename="rtf-body.rtf"
Yep, that's right: Content-Disposition: attachment, but still this is the actual e-mail body.
Now in Aleph, these messages will show up as empty, with rtf-body.rtf document as attachment.
I tried to work around it by unpacking the mail archive manually with readpst; then fixing the messages with a small python script (essentially replacing the rtf part with an html part. I used python's email.parser and simply checked if the first content_type would be application/rtf - if so, pipe that through unrtf and repack the message. Filthy, but working for the mail box itself).
This workaround would not help in Aleph, because the mime detection wizardry afterwards recognized text/html for mime type, instead of message/rfc822 - and actual attachments of the message would not be recognized anymore.
The latter may count as a separate bug: a message that starts with the following should IMHO not be detected as text/html?
Status: RO
User-Agent: none
From: "Firstname Lastname" <MAILER-DAEMON>
Subject: FW: Ticket 08-05
To: Name (Company Name)
Date: Tue, 09 May 2022 14:41:50 +0000
Message-Id: <AM5PR04MB53161D02E214FEEC35C2156FB6466@AM0PR04MB9122.eurprd02.prod.outlook.com>
X-libpst-forensic-sender: /O=EXCHANGELABS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=09C4BB2213F35544FEBBBBF1FD14B522
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="--boundary-LibPST-iamunique-887075155_-_-"
----boundary-LibPST-iamunique-887075155_-_-
Content-Type: text/html; charset="utf-8"
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
This (previously WIP, now abandoned) mentions the same problems https://github.com/alephdata/ingest-file/pull/20
Where is the mime detection done? I think it could work to try fix the output of readpst - by adding transport headers or otherwise; fix the RTF-parts of the e-mails, too. As I already walk over all e-mails to fix the RTF-parts, adding required headers for mime detection (message/rfc822 instead of text/html) could work, too.
Messages can pretty easily be "tricked" into being message/rfc822, by simply adding Received: from localhost (127.0.0.1) at the top of the message. IMHO as a workaround for the current state of things, this could be done right after readpst. I will investigate.
In order to fix messages that have an RTF-only message body, I'm manually starting a Python script:
#!/usr/bin/python3
import base64
import os
import sys
import re
import mimetypes
import email
from email.policy import default
from email.parser import BytesParser
import subprocess
plcy=default.clone(refold_source='none')
for fname in sys.argv[1:]:
try:
mail=open(fname,'rb')
except:
print(fname, "not found.")
continue
msg = BytesParser(policy=plcy).parse(mail)
mail.close()
totaal=list(msg.walk())
if (len(totaal)<2):
continue
if (totaal[1].get_content_type() == 'application/rtf'):
print("Converting", fname)
html=subprocess.run(['/usr/bin/unrtf'], input=totaal[1].get_content(), capture_output=True).stdout
totaal[1].set_content(html, maintype='text',subtype='html')
try:
mail=open(fname,'w')
except:
print("Error writing")
continue
print(totaal[0], file=mail)
mail.close()
It's a hack. But it works and it really helps the search process. This could be run right after readpst but I really don't think this is production quality. Anyway, maybe it helps someone make a proper fix.