email parsing to issue fails when pictures are included in the email body

[jendik]

Hi Everyone,

since the time I installed your amazing plugin (btw. it really helps a lot! And thank you...) I experience a weird issue. To be honest I haven't been able to confirm so far the exact connection to your plugin, but I wanted to ask you for any idea before I'll do additional tests and troubleshooting.

We use email parsing a lot. We send emails to the mailbox that is checked by redmine and tasks are created from the messages. We'd been using this for years without an issue till we installed "additionals plugin".

After we installed additionals, we realized that email messages, that consists any of the emoji pictures (smileys) will not be parsed by the redmine. Whole message is marked as red and stays in the mailbox. What's more we suspects redmine to even delete some of the messages without creating the issue. Then, we found out that not only emojis/smileys are the issues, but probably most of the pictures as well(like logos in the signature, screenshots, etc..).

Have you got any idea if this can be related to your plugin or any configuration parameter/settings that it brings please?

Thanks so much Kind regards J.

[jendik]

Hello,

so I've just tested that redmine (the same database) without the additionals plugin doesn't have this issue and the same email, from the same mailbox, with the same rake command get parsed properly and get transformed through to the redmine issue as expected.

I started debug logging at redmine, but it doesn't say any specific or helping error.-( It shows all the steps, isue creation, etc..and then just ROLLBACK...

Please see last lines of the log below.

2019-10-05 14:20:12 +0200 (21670) Watcher Create (0.2ms) INSERT INTO watchers (watchable_type, watchable_id, user_id) VALUES ('Issue', 6318, 35) 2019-10-05 14:20:12 +0200 (21670) Watcher Exists (0.3ms) SELECT 1 AS one FROM watchers WHERE watchers.user_id = 4 AND watchers.watchable_type = 'Issue' AND watchers.watchable_id = 6318 LIMIT 1 2019-10-05 14:20:13 +0200 (21670) (4.0ms) ROLLBACK 2019-10-05 14:20:13 +0200 (21670) Setting Load (0.4ms) SELECT settings.* FROM settings WHERE settings.name = 'mail_handler_excluded_filenames' ORDER BY settings.id DESC LIMIT 1

When you compare it to the log where everything went well the difference is that instead of "rollback", there is "commit".

If you need more details, sample emails, rake command, full log for this particular parsing event, or anything, I'm more that happy to send it to you, but do not want to post it publicly as it may contain some sensitive information. Please let me know what you need.

I do not know what exactly causes the email being just read, but not imported. I can say that any text email gets probably through. Most html email as well, but any with smiley get stuck and some of the emails with logos, pictures as well.

I also tried to find any additional settings in your plugin to make the incoming emails working again, but without any luck.

Could you please check it? Could you please test it? Or try to reproduce and help me to solve the issue please?

Otherwise I will have to remove the plugin from our prod redmine, but it could cause a lot of issues to us and we really like the other functionalities you bring. I will be glad for any help.

Thanks a lot J.

[alexandermeindl]

Hi J.,

thanks for reporting this. Can you provide more information?

• How to you import email to Redmine? With https://www.redmine.org/projects/redmine/wiki/RedmineReceivingEmails (redmine:email:receive_imap or redmine:email:receive_pop3)
• Which Redmine version?
• Which additionals version?
• Which Ruby version?
• What database? Mysql, Postgresql... and which version If Mysql, you may find this article interesting: https://www.redmine.org/boards/2/topics/53240
• Which OS? (Windows, Linux, ...)
• Which other plugins are you using?
• Are you getting the same result, if you disable/enable "Smiley/Emoji Support" and/or "Add involved users as watcher automatically" in additionals plugin settings?

[jendik]

Hi!   thanks so much for your reaction and I'm sorry for late reply.  

• import via IMAP: /opt/bitnami/ruby/bin/bundle exec rake redmine:email:receive_imap RAILS_ENV="production" …  
• redmine version: 4.0.0.stable  
• additionals version: 2.0.21  
• Ruby versioin :~$ gem --version 2.6.14 ~$ ruby -v ruby 2.3.8p459 (2018-10-18 revision 65136) [x86_64-linux]   -redmine instalation running using prebuilt VM from Bitnami  
• DB Mysql /opt/bitnami/mysql/bin/mysql.bin  Ver 14.14 Distrib 5.6.42, for linux-glibc2.12 (x86_64) using EditLine wrapper  
• Linux DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"  
• other plugins: only this one from Undev (but it's simple and I believe it doesn't have anything to do with that Redmine Close Issue Button Plugin This plugins enables you to close issues quickly using the Close Issue button. http://github.com/Undev/redmine_close_button  
• Yes, this was the first thing I tried to enable/disable "smiley/emoji support" - this didn't help, not even with restarting the redmine.  
• I've just tried to disable "add watchers" as you proposed and woooa! -) All the stuck emails went through, besides those with emoji/smileys…. PERRFECT half of the issue solved,-) at least as long as you do not want to use this feature-) Thanks a lot so by so. 
 The bad news is that messages with this nice colorful smileys still gets stuck.

Any idea what to do please? Any idea how it could be related to your plugin if anyhow?

Thank you very much j.

[jendik]

example smiley here:

😊

[jendik]

OK, so as we solved most of the failed emails by unchecking adding watchers, I've just tried to feed problematic email by redmine instance without your plugin and the issue seems to be the same here. So this second problem will be probably in redmine itself.

Any idea please, how to solve it anyway please? I saw the links you shared, suggesting changing mysql db settings, but it sounds a bit scary to me... I know redmine 4.1 will bring the fix, so worst case scenario is waiting, it should be fine I guess.

Any idea please how to fix the watchers issue as it is very nice functionality, but can't be used when feeding issues from email.

Thanks so much Kind regards j.