CFMail splitting long HTML lines into multiple lines breaking the HTML [LDEV-4039]

[dswitzer]

I've refactored CFMAIL to use quoted-printable content transfer encoding for all HTML parts.

On the suggestion of @zspitzer, I added a system property/environment variable (i.e. lucee.mail.use.7bit.transfer.encoding.for.html.parts) which allows switching back to the old behavior. This defaults to false which changes the behavior, but the only downside I can see to switching to quoted-printable as the default encoding is the message size will be a little larger. However, it fixes issues like:

• Line wrapping — quoted-printable always wraps lines to 76 characters.
• Does not break long strings — Long strings are encoded in the line wrapping, which means you could have a string (like a link) longer than 1000 characters without CFMAIL breaking the string after 997 characters.
• Encodes non-ASCII characters — quoted-printable will encode characters you cannot normally send with 7bit encoding, such as 8bit UTF characters.

IMPORTANT — This is the same patch I submitted for the 5.3 branch. However, I would recommend removing the lucee.mail.use.7bit.transfer.encoding.for.html.parts system property in 6.0 and just always use quoted-printable. I really don't see any upside to ever sending HTML emails with 7bit encoding. The odds are it's going to cause developers very subtle issues that are hard to troubleshoot and potentially hard to replicate.

https://luceeserver.atlassian.net/browse/LDEV-4039

[zspitzer]

@dswitzer @cfmitrah has fixed the mail.cfc for 6.0, can you rebase and merge your changes into it?

https://github.com/lucee/Lucee/pull/1713

[dswitzer]

@zspitzer @cfmitrah So Greenmail does not seem to get initialized when running the unit tests, so the mail tests don't actually run locally. Is there a way to run it locally so I can test before committing my changes?

[dswitzer]

@zspitzer So I've committed the test cases. I'm not sure why the build failed, seems to maybe be a Github/ANT configuration issue. I know the code compiles and runs tests locally.

However, I cannot actually test the Mail.cfc unit tests. I think I converted them correctly. I finally figured out how to try and configure the services to run, but I was seeing javax.mail.Provider: com.sun.mail.imap.IMAPProvider not a subtype exceptions in the native Lucee tags. We have a pretty old version of Greenmail running locally, so I'm thinking there's some incompatibility there causing issues because it's failing inside the native Java libs.

[cfmitrah]

@zspitzer @cfmitrah So Greenmail does not seem to get initialized when running the unit tests, so the mail tests don't actually run locally. Is there a way to run it locally so I can test before committing my changes?

you can run/test greenmail locally in docker using this comment docker run -t -i -p 3025:3025 -p 3110:3110 -p 3143:3143 -p 3465:3465 -p 3993:3993 -p 3995:3995 -p 8080:8080 greenmail/standalone:1.6.9

[dswitzer]

@cfmitrah

So there's even an easier way to get Greenmail running (no need for Docker at all). Download the Greenmail standalone jar and just run:

java -Dgreenmail.setup.test.all -Dgreenmail.users=test1:pwd1,test2:[email protected] \
     -jar greenmail-standalone.jar

This starts GreenMail for SMTP (test port 3025), SMTPS (test port 3465), IMAP (test port 3143), IMAPS (test port 3993), POP3 (test port 3110) and POP3S (test port 3995) using localhost/127.0.0.1 with the two users specified in the CLI arguments.

In theory, the ANT script could be updated to just spawn Greenmail w/out needing docker by spawning the JAR to run in the background during the tests or by just adding some Java or CFML to fire up Greenmail during the required tests:

https://stackoverflow.com/a/9663399/16607349

This would at least allow the tests to run in any environment without additional setup or configuration.

However, the problem I've been facing is the javax.mail.Provider: com.sun.mail.imap.IMAPProvider not a subtype exception I posted above. It seems like this is an issue with the build. I found this ticket from a while ago:

https://luceeserver.atlassian.net/browse/LDEV-3431

I'm trying to track down why I'm having the issue locally, but the the Github build tests are not.

[dswitzer]

So I still can't figure out why when running the ANT build, the Mail.cfc keeps throwing the javax.mail.Provider: com.sun.mail.imap.IMAPProvider not a subtype exception. I've spent too much time tracking it down, so I just had to verify the tests manually.

The Mail tests are now passing, I've updated the tests to include helpers for pulling down the raw message (i.e. the EML) so that I can actually parse the content to make sure the multi-part messages are formatted correctly.

However the build checks are are failing in another part of the build process which I did not mess with:

Error:  Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (ant-magic) on project lucee: An Ant BuildException has occured: The following error occurred while executing this line:
Error:  /home/runner/work/Lucee/Lucee/ant/build-extensions.xml:280: The following error occurred while executing this line:
Error:  /home/runner/work/Lucee/Lucee/ant/build-core.xml:220: /home/runner/work/Lucee/Lucee/cache/D46B46A9-A0E3-44E1-D972A04AC3A8DC10;version=1.0.19.24-${extChartVersion}.lex does not exist.
Error:  around Ant part ...<ant antfile="../ant/build-core.xml" target="buildLoaderMaven" />... @ 17:70 in /home/runner/work/Lucee/Lucee/loader/target/antrun/build-main.xml
Error:  -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (ant-magic) on project lucee: An Ant BuildException has occured: The following error occurred while executing this line:
/home/runner/work/Lucee/Lucee/ant/build-extensions.xml:280: The following error occurred while executing this line:
/home/runner/work/Lucee/Lucee/ant/build-core.xml:220: /home/runner/work/Lucee/Lucee/cache/D46B46A9-A0E3-44E1-D972A04AC3A8DC10;version=1.0.19.24-${extChartVersion}.lex does not exist.
around Ant part ...<ant antfile="../ant/build-core.xml" target="buildLoaderMaven" />... @ 17:70 in /home/runner/work/Lucee/Lucee/loader/target/antrun/build-main.xml

Not sure why it's bombing with the chart extension. Maybe it's having trouble downloading the extension.

I've try submitting a few empty commits (e.g. git commit -m "retrigger checks" --allow-empty) to retrigger the build checks but it keeps failing at the same spot. My pull request code is passing the tests though.

[dswitzer]

@zspitzer @cfmitrah Any ideas why the build checks are failing?

It looks like @michaeloffner recently modified the build-core.xml file and removed some of the extensions, so I'm wonder if that's why it's blowing up.

[zspitzer]

@dswitzer can you remove all the prefs and classpath changes please?

the 6.0 branch is clean, if you can rebase the PR that would also be great

[dswitzer]

@zspitzer ,

You’re only wanting the lucee.mail.use.7bit.transfer.encoding.for.html.parts preference removed, right?

You want no backward support for 7bit support and only quoted-printable support, correct?

Just a reminder I only added this on your recommendation.

[dswitzer]

@zspitzer,

I'm glad I asked! I would have done the wrong thing!

VScode always seems to want to change those files silently. It might be worth getting Git to ignore changes to those files, to prevent those issues happening with other users.

[dswitzer]

@zspitzer,

I've rebased the changes (hopefully did that correctly) and made the requested changes.

[dswitzer]

@zspitzer,

I've made those changes too.

[zspitzer]

@dswitzer excellent, all green now!