mail-api icon indicating copy to clipboard operation
mail-api copied to clipboard

Published 20 hours ago • verified publisher icon jakartaee

SharedByteArrayInputStream instead mutipart with mail.pop3.forgettopheaders=true

Open demarco88 opened this issue 2 years ago • 4 comments

Hi. We face the problem, that we ant to use the flag mail.pop3.forgettopheaders=true in order to force new header request in RETR command, because of header changes of some providers between the TOP and the RETR Command.

At TOP Command the content-type is: Content-Type: application/pkcs7-mime; smime-type=authenticated-enveloped-data; name=smime.p7m At RETR Command the content-type is: Content-Type: multipart/mixed; boundary="----=_Part_45_761695249.1657878900597" Because between TOP Command and RETR Command the provider packed the mail inside a new mail envelope. This behaviour is part of a german healthcare communication standard specification called KIM.

Now when parsing the content of the email after the RETR Command content-type of the message is application/pkcs7-mime but it should be multipart/mixed. Further the getContent() delivers a SharedByteArrayInputStream instead of a Multipart-Object. Seams to be a bug in combination with forgettopheaders.

If we switch off the forgettopheaders parsing the content is working, but we have the "wrong" (first received ) header informations. If we switch of the mail.pop3.disablecapa or activate the mail.pop3.disabletop then we force the api to use directly the RETR Coammnd. In this case the email is also parsed correctly. But it isn't a good solution for us because we need to filter out some emails in case of specific header informations.

Tested with version 1.6.4 and also 1.6.7.

regards Marco

demarco88 avatar Jul 15 '22 10:07 demarco88

Hi @demarco88 are you able to provide a reproducer or some example code so I can get better understanding?.

jbescos avatar Jul 21 '22 10:07 jbescos

Hi @jbescos. Thanks for your request for additional information. It seems to be be a common problem of the cached headers if the RETR command delivers complete other headers than the TOP command. Especially when the content-type of the first part of the message has changed. And forgettopheaders=true does not help to fix the problem. Following simple parts of an example code.

public void receiveemails(Account account) throws Exception
{
  this.pop3properties.put("mail.pop3.ssl.socketFactory", this.sslsocketfactory);
  this.pop3properties.put("mail.pop3.disablecapa", true);
  this.pop3properties.put("mail.pop3.disabletop", false);
  this.pop3properties.put("mail.pop3.forgettopheaders", false);
  this.pop3properties.put("mail.debug", "true");

  Session session = getSession(this.pop3properties, account.getUser(), account.getPassword());
  URLName url = new URLName("pop3", pop3properties.getProperty("mail.pop3.host"),
      Integer.parseInt(pop3properties.getProperty("mail.pop3.port")), "", account.getUser(), account.getPassword());
  POP3Store store = (POP3Store)session.getStore(url);
  store.connect();
  POP3Folder folder = (POP3Folder)store.getFolder("INBOX");
  folder.open(Folder.READ_WRITE);

  Message[] msgs = folder.getMessages();
  for(int a = 0;a < msgs.length;a++)
    traverse(msgs[a]);
  folder.close(false);
}

private static void traverse(Part p) throws Exception
{
  String contenttype = p.getContentType();
  Object o = p.getContent();
  System.out.println("MESSAGE: content-type:" + contenttype + " object class:" + o.getClass());
  if(o instanceof Multipart)
  {
    Multipart mp = (Multipart)o;
    int count = mp.getCount();
    // recursive Aufruf
    for(int i = 0;i < count;i++)
      traverse(mp.getBodyPart(i));
  }

  if(o instanceof MimeMessage)
  {
    MimeMessage att = (MimeMessage)o;
    traverse(att);
  }
  return;
}

Furthermore the debug protocol of this execution. Some data are anonymized. The complete base-64 content is cutted (MIAGCyqGS.....................CUTTED...==). It does'nt make sence to post it here completely.

DEBUG: Jakarta Mail version 1.6.4
DEBUG: successfully loaded resource: /META-INF/javamail.default.providers
DEBUG: Tables of loaded providers
DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPTransport=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], com.sun.mail.imap.IMAPSSLStore=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], com.sun.mail.pop3.POP3Store=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], com.sun.mail.smtp.SMTPSSLTransport=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], com.sun.mail.imap.IMAPStore=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], com.sun.mail.pop3.POP3SSLStore=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
DEBUG: Providers Listed By Protocol: {imap=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], smtp=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], pop3=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], imaps=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], smtps=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], pop3s=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map
DEBUG: getProvider() returning javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle]
DEBUG POP3: mail.pop3.rsetbeforequit: false
DEBUG POP3: mail.pop3.disabletop: false
DEBUG POP3: mail.pop3.forgettopheaders: false
DEBUG POP3: mail.pop3.cachewriteto: false
DEBUG POP3: mail.pop3.filecache.enable: false
DEBUG POP3: mail.pop3.keepmessagecontent: false
DEBUG POP3: mail.pop3.starttls.enable: false
DEBUG POP3: mail.pop3.starttls.required: false
DEBUG POP3: mail.pop3.finalizecleanclose: false
DEBUG POP3: mail.pop3.apop.enable: false
DEBUG POP3: mail.pop3.disablecapa: true
DEBUG POP3: connecting to host "testserver.fancy.firm.net", port 995, isSSL true
+OK KOM-LE Clientmodul POP3
DEBUG POP3: authentication command trace suppressed
DEBUG POP3: authentication command succeeded
STAT
+OK 1 7424
NOOP
+OK
TOP 1 0
+OK
Return-Path: <[email protected]>
Delivered-To: [email protected]
Received: from gtegklve09183otherfancy.server.somewhere.de ([10.30.8.26])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	by gtegklvd09182otherfancy.server.somewhere.de with LMTPS
	id JG60JYd83mIprQEAz3/SOw
	(envelope-from <[email protected]>)
	for <[email protected]>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from mail.arv.kim.telematik-test ([10.30.8.26])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	by gtegklve09183otherfancy.server.somewhere.de with LMTPS
	id 3LyGI4d83mLb2gEAU3MrGQ
	(envelope-from <[email protected]>)
	for <[email protected]>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from unknown (unknown [10.23.16.42])
	by mail.arv.kim.telematik-test (Postfix) with SMTP id 075E8C1
	for <[email protected]>; Mon, 25 Jul 2022 13:20:12 +0200 (CEST)
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: [email protected]
To: [email protected]
Message-Id: <[email protected]>
Subject: KOM-LE-Nachricht
MIME-Version: 1.0
Content-Type: application/pkcs7-mime; name=smime.p7m; 
	smime-type=authenticated-enveloped-data
Content-Transfer-Encoding: base64
X-KOM-LE-Version: 1.0
Content-Disposition: attachment; filename=smime.p7m
X-KIM-Dienstkennung: Selbsttest;Lieferung;V1.0

.
RETR 1
+OK
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: [email protected]
To: [email protected]
Subject: Die Nachricht konnte nicht entschlüsselt werden.
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_286_2060398125.1658750487374"

------=_Part_286_2060398125.1658750487374
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Der f=C3=BCr die Entschl=C3=BCsselung der Nachricht ben=C3=B6tigte Schl=C3=
=BCssel wurde nicht gefunden. =C3=9Cberpr=C3=BCfen Sie ob die entsprechende=
 Karte gesteckt ist und leiten Sie diese Nachricht an Ihre eigene E-Mail-Ad=
resse ([email protected]) weiter. Beim n=C3=A4chsten Abho=
len wird der Entschl=C3=BCsselungsvorgang wiederholt.
------=_Part_286_2060398125.1658750487374
Content-Type: message/rfc822

Return-Path: <[email protected]>
Delivered-To: [email protected]
Received: from gtegklve09183otherfancy.server.somewhere.de ([10.30.8.26])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	by gtegklvd09182otherfancy.server.somewhere.de with LMTPS
	id JG60JYd83mIprQEAz3/SOw
	(envelope-from <[email protected]>)
	for <[email protected]>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from mail.arv.kim.telematik-test ([10.30.8.26])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	by gtegklve09183otherfancy.server.somewhere.de with LMTPS
	id 3LyGI4d83mLb2gEAU3MrGQ
	(envelope-from <[email protected]>)
	for <[email protected]>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from unknown (unknown [10.23.16.42])
	by mail.arv.kim.telematik-test (Postfix) with SMTP id 075E8C1
	for <[email protected]>; Mon, 25 Jul 2022 13:20:12 +0200 (CEST)
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: [email protected]
To: [email protected]
Message-Id: <[email protected]>
Subject: KOM-LE-Nachricht
MIME-Version: 1.0
Content-Type: application/pkcs7-mime; name=smime.p7m; 
	smime-type=authenticated-enveloped-data
Content-Transfer-Encoding: base64
X-KOM-LE-Version: 1.0
Content-Disposition: attachment; filename=smime.p7m
X-KIM-Dienstkennung: Selbsttest;Lieferung;V1.0

MIAGCyqGS.....................CUTTED...==

------=_Part_286_2060398125.1658750487374--

.
MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
	smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream
QUIT
+OK Logging out.
2

In the RETR Command we first have a multipart/mixed. Including a part text/plain and message/rfc822. But the additional debug logging in the travers method said:

MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
	smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream

```	
if this example is executed with the **mail.pop3.forgettopheaders=true** we will get the same output except the Type of the content object:

MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; smime-type=authenticated-enveloped-data object class:class javax.mail.util.SharedByteArrayInputStream

	
**Now the working example:** Only think that seems to work is setting the **mail.pop3.disabletop to true**. But that's no good option for us.
Then it will produce the following output:

DEBUG: Jakarta Mail version 1.6.4 DEBUG: successfully loaded resource: /META-INF/javamail.default.providers DEBUG: Tables of loaded providers DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPTransport=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], com.sun.mail.imap.IMAPSSLStore=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], com.sun.mail.pop3.POP3Store=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], com.sun.mail.smtp.SMTPSSLTransport=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], com.sun.mail.imap.IMAPStore=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], com.sun.mail.pop3.POP3SSLStore=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]} DEBUG: Providers Listed By Protocol: {imap=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], smtp=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], pop3=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], imaps=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], smtps=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], pop3s=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]} DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map DEBUG: getProvider() returning javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle] DEBUG POP3: mail.pop3.rsetbeforequit: false DEBUG POP3: mail.pop3.disabletop: true DEBUG POP3: mail.pop3.forgettopheaders: false DEBUG POP3: mail.pop3.cachewriteto: false DEBUG POP3: mail.pop3.filecache.enable: false DEBUG POP3: mail.pop3.keepmessagecontent: false DEBUG POP3: mail.pop3.starttls.enable: false DEBUG POP3: mail.pop3.starttls.required: false DEBUG POP3: mail.pop3.finalizecleanclose: false DEBUG POP3: mail.pop3.apop.enable: false DEBUG POP3: mail.pop3.disablecapa: true DEBUG POP3: connecting to host "testserver.fancy.firm.net", port 995, isSSL true +OK KOM-LE Clientmodul POP3 DEBUG POP3: authentication command trace suppressed DEBUG POP3: authentication command succeeded STAT +OK 1 7424 NOOP +OK RETR 1 +OK Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST) From: [email protected] To: [email protected] Subject: Die Nachricht konnte nicht entschlüsselt werden. MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_293_810607501.1658751499303"

------=_Part_293_810607501.1658751499303 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Der f=C3=BCr die Entschl=C3=BCsselung der Nachricht ben=C3=B6tigte Schl=C3= =BCssel wurde nicht gefunden. =C3=9Cberpr=C3=BCfen Sie ob die entsprechende= Karte gesteckt ist und leiten Sie diese Nachricht an Ihre eigene E-Mail-Ad= resse ([email protected]) weiter. Beim n=C3=A4chsten Abho= len wird der Entschl=C3=BCsselungsvorgang wiederholt. ------=_Part_293_810607501.1658751499303 Content-Type: message/rfc822

Return-Path: [email protected] Delivered-To: [email protected] Received: from gtegklve09183otherfancy.server.somewhere.de ([10.30.8.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by gtegklvd09182otherfancy.server.somewhere.de with LMTPS id JG60JYd83mIprQEAz3/SOw (envelope-from [email protected]) for [email protected]; Mon, 25 Jul 2022 13:20:39 +0200 Received: from mail.arv.kim.telematik-test ([10.30.8.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by gtegklve09183otherfancy.server.somewhere.de with LMTPS id 3LyGI4d83mLb2gEAU3MrGQ (envelope-from [email protected]) for [email protected]; Mon, 25 Jul 2022 13:20:39 +0200 Received: from unknown (unknown [10.23.16.42]) by mail.arv.kim.telematik-test (Postfix) with SMTP id 075E8C1 for [email protected]; Mon, 25 Jul 2022 13:20:12 +0200 (CEST) Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST) From: [email protected] To: [email protected] Message-Id: [email protected] Subject: KOM-LE-Nachricht MIME-Version: 1.0 Content-Type: application/pkcs7-mime; name=smime.p7m; smime-type=authenticated-enveloped-data Content-Transfer-Encoding: base64 X-KOM-LE-Version: 1.0 Content-Disposition: attachment; filename=smime.p7m X-KIM-Dienstkennung: Selbsttest;Lieferung;V1.0

MIAGCyqGS.....................CUTTED...==

------=_Part_293_810607501.1658751499303--

. MESSAGE: content-type:multipart/mixed; boundary="----=_Part_293_810607501.1658751499303" object class:class javax.mail.internet.MimeMultipart MESSAGE: content-type:text/plain; charset=utf-8 object class:class java.lang.String MESSAGE: content-type:message/rfc822 object class:class javax.mail.internet.MimeMessage MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream QUIT +OK Logging out. 2


Now the message has the correct parts. mail.pop3.forgettopheaders does not help or it's a bug in there.

MESSAGE: content-type:multipart/mixed; boundary="----=_Part_293_810607501.1658751499303" object class:class javax.mail.internet.MimeMultipart MESSAGE: content-type:text/plain; charset=utf-8 object class:class java.lang.String MESSAGE: content-type:message/rfc822 object class:class javax.mail.internet.MimeMessage MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream

demarco88 avatar Jul 25 '22 12:07 demarco88

@demarco88 Just ran in the same issue. How did you solve it?

Codein1 avatar Sep 15 '23 13:09 Codein1

I am debugging this. mail.pop3.forgettopheaders=true is ignored when mail.pop3.disabletop is not true.

This flag has to be true, and that can only happen if mail.pop3.disabletop=true or there is no TOP message. https://github.com/eclipse-ee4j/angus-mail/blob/62aae89956dfa26c92e854c04e2ac9c14528a10d/providers/pop3/src/main/java/org/eclipse/angus/mail/pop3/POP3Message.java#L641

You said that mail.pop3.disabletop=true is not a good option for you, but I don't see why is not an option. Could you help me to understand that better, please?.

jbescos avatar Mar 01 '24 11:03 jbescos