text/x-mail MIME issues

[gh0stwizard]

Issue summary

I have found 2 issues when working with text/x-mail email messages:

1. Unable to view the body of a message with mime type text/x-mail.
2. Unable to release a message with mime type text/x-mail.

I am not including steps to reproduce these bugs, instead see patches below. Hope this will help. I am not exactly sure that this is correct way to fix the issue.

1. Allow view the body of a message

https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L541

--- /var/www/html/mailscanner/detail.php.orig   2019-07-15 14:49:21.311222291 +0300
+++ /var/www/html/mailscanner/detail.php     2019-07-15 14:49:54.533127253 +0300
@@ -533,7 +533,7 @@
                     $item['dangerous'] === 'N' ||
                     $_SESSION['user_type'] === 'A' ||
                     (defined('DOMAINADMIN_CAN_SEE_DANGEROUS_CONTENTS') && true === DOMAINADMIN_CAN_SEE_DANGEROUS_CONTENTS && $_SESSION['user_type'] === 'D' && $item['dangerous'] === 'Y')
-                ) && preg_match('!message/rfc822!', $item['type'])
+                ) && preg_match('!message/rfc822|text/x-mail!', $item['type'])
             ) {
                 echo '  <td><a href="viewmail.php?token=' . $_SESSION['token'] .'&amp;id=' . $item['msgid'] . '">' .
                     substr($item['path'], strlen($quarantinedir) + 1) .

2. Allow releasing a message

https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3584

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-15 13:21:48.935886617 +0300
@@ -3493,7 +3493,7 @@
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
         foreach ($num as $key => $val) {
-            if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+            if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
                 exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
                 if ($retval === 0) {

Installation

Version and method

• MailWatch Version: 1.2.7-dev (EFA 3.0.2.6)
• Install type: EFA https://efa-project.org
• Updated from an older MailWatch or fresh install: vendor (EFA)

Thank you!

[gh0stwizard]

I have added a fix for releasing messages from Message Operations list. Full patch is below.

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-16 10:27:39.143386292 +0300
@@ -3461,7 +3461,7 @@
             // Loop through each selected file and attach them to the mail
             foreach ($num as $key => $val) {
                 // If the message is of rfc822 type then set it as Quoted printable
-                if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+                if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                     $mime->addAttachment($list[$val]['path'], 'message/rfc822', 'Original Message', true, '');
                 } else {
                     // Default is base64 encoded
@@ -3492,8 +3492,9 @@
         // Use sendmail to release message
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
-        foreach ($num as $key => $val) {
-            if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+        $count = count($list);
+        for ($val = 0; $val < $count; $val++) {
+            if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
                 exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
                 if ($retval === 0) {

The issue with the foreach loop is due the fact how exactly the second argument is passed to quarantine_release in the /var/www/html/mailscanner/do_message_ops.php file. Pay attention on how $itemnum is counted. It means that $itemnum is always equal to [ 0 ] array: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/do_message_ops.php#L96-L113

But in the case when a message has an attachment like .zip and the message is considered as Bad Content by some reason, then the quarantined directory contains multiple files, for instance (got from the real blocked message):

• my.cnf | text/plain; charset=us-ascii | 20190716/3C0A11231E8.AD0E2/my.cnf
• diags.tar | application/x-tar; charset=binary | 20190716/3C0A11231E8.AD0E2/diags.tar
• message | message/rfc822; charset=us-ascii | 20190716/3C0A11231E8.AD0E2/message

The file order is major issue here. So, the loop foreach ($num as $key => $val) { will check first file in the order only (my.cnf to be exactly). Because $num in the loop is the value $itemnum from do_message_ops.php and this value always equal to [0] (precisely, $num = 0; $itemnum = array($num);). Hope I explained this clear.

In the time of writing this message I've realized that call of quarantine_learn() also is incorrect in the do_message_ops.php file. There is need a fix either in do_message_ops.php and/or both quarantine_release(), quarantine_learn(). Specifically, quarantine_learn() must pass to the spamassassin command only complete the email message file, IMHO. But quarantine_learn() does not contain such a check.

UPDATE

I have rewrote the patch taking into account that the value $num of quarantine_release($list, $num, $to, $rpc_only = false) is an array of selected items. But I see there is a mime type check, so probably the version below is better:

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-16 11:24:46.860228320 +0300
@@ -3461,7 +3461,7 @@
             // Loop through each selected file and attach them to the mail
             foreach ($num as $key => $val) {
                 // If the message is of rfc822 type then set it as Quoted printable
-                if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+                if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                     $mime->addAttachment($list[$val]['path'], 'message/rfc822', 'Original Message', true, '');
                 } else {
                     // Default is base64 encoded
@@ -3492,8 +3492,11 @@
         // Use sendmail to release message
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
+        if (count($num) < count($list)) {
+            $num = array_keys($list);
+        }
         foreach ($num as $key => $val) {
-            if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+            if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
                 exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
                 if ($retval === 0) {

[Skywalker-11]

Is there a standard or similar for the structure of text/x-mail? I could only find a reference to "x-..." prefix in general referencing unstandardized protocolls.

[gh0stwizard]

@Skywalker-11 I did not dig so deep, but see http://lists.roundcube.net/pipermail/dev/2013-January/022097.html

Ok, while writing this message I have found out what is a cause. On CentOS 6.10 there is file-libs package version 5.04. I got source from https://www.darwinsys.com/file/ and dig in inside:

$ grep -nrR 'x-mail' file-5.04
file-5.04/src/names.h:70:       { "mail",                                       "text/x-mail" },

As expected file returns:

$ file -bi text_x-mail.message.txt
text/x-mail; charset=us-ascii

I have posted an example email message here: https://pastebin.com/2LEhbzhz

Steps to reproduce:

1. Include any non-empty file with any blocked extension in MailScanner.
2. That's all.

[gh0stwizard]

@Skywalker-11 Here is a real cause, compare this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3498 and this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3516-L3517

On the detail page of the message you see message/rfc822 because of this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L380 https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L503 https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L524-L533

Conclusion. Depending on if a message is blocked or not quarantine_list_items() returns either hard-coded value message/rfc822, either what file -bi FILE has returned.

UPDATE

Easiest workaround instead of fixing all regular expressions with message/rfc822 is below. Changes:

1. Use text/x-mail as an alias for message/rfc822 in quarantine_list_items() function.
2. Fix quarantine_release() to look all extracted files within a message. If the [mime] type of a file is message/rfc822 then release the message. Tested on only when QUARANTINE_USE_SENDMAIL is true (see conf.php for details).
3. Fix quarantine_learn() to look all extracted files within a message. If the [mime] type of a file is message/rfc822 then pass the path of the file to spamassassin, e.g. complete message and not extracted from the message attachments.

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-16 16:55:31.642773396 +0300
@@ -3394,7 +3394,11 @@
                     $quarantined[$count]['to'] = $row->to_address;
                     $quarantined[$count]['file'] = $f;
                     $file = escapeshellarg($quarantine . '/' . $f);
-                    $quarantined[$count]['type'] = ltrim(rtrim(`/usr/bin/file -bi $file`));
+                    $type = ltrim(rtrim(`/usr/bin/file -bi $file`));
+                    if (preg_match('!^text/x-mail!', $type)) {
+                        $type = 'message/rfc822';
+                    }
+                    $quarantined[$count]['type'] = $type;
                     $quarantined[$count]['path'] = $quarantine . '/' . $f;
                     $quarantined[$count]['md5'] = md5($quarantine . '/' . $f);
                     $quarantined[$count]['dangerous'] = $infected;
@@ -3492,6 +3496,9 @@
         // Use sendmail to release message
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
+        if (count($num) < count($list)) {
+            $num = array_keys($list);
+        }
         foreach ($num as $key => $val) {
             if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
@@ -3566,7 +3573,14 @@
     if (!$rpc_only && is_local($list[0]['host'])) {
         //prevent sa-learn process blocking complete apache server
         session_write_close();
+        if (count($num) < count($list)) {
+            $num = array_keys($list);
+        }
         foreach ($num as $key => $val) {
+            if (!preg_match('!message/rfc822!', $list[$val]['type'])) {
+                continue;
+            }
+
             $use_spamassassin = false;
             $isfn = '0';
             $isfp = '0';

[Schroeffu]

Have had exactly the same problem with mails encoded text/plain; charset=us-ascii in mailwatch v1.2.10. Changing the two lines initially explained to:

(detail.php) ) && preg_match('!message/rfc822|text/plain!', $item['type']) (function.php) if (preg_match('/message\/rfc822|text\/plain/', $list[$val]['type'])) {

and change conf.php to define('QUARANTINE_USE_SENDMAIL', true); (usually i am using "false") then the release works.

But i am using usually QUARANTINE_USE_SENDMAIL false. This way the attachement called "message" is in a corrupt format, cant open it with any app (ok with atom f.e. but the content is not read-able by outlook. in a text editor atom i see the mail content is base64 encoded -.-

Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64)

[Schroeffu]

Today i've had the same issue with another mail release again. still no fix published ? :-(