dbmail
dbmail 3.3.x - Breaks attachments :(
Hi. After almost 6 years of uninterrupted mail server operation (dbmail 3.2.6), I decided to install the latest release 3.3. Already on the first 1000 e-mails, there were sporadic e-mails with broken attachments. My assumption is that when processing MIME type, something does not work correctly and attachments are cut off. Unfortunately I could not find more exact problems in the logs. I have Debian 10, postfix + dbmail + rspamd, postgresql. I only updated dbmail. I really hope that the problem can be solved. Thanks.
There are no known issues with 3.3 and attachments. The upgrade from 3.2 to 3.3 was to upgrade gmime from 2.8 to 3. As there were no database schema changes, can you clarify what you mean by the first 1000 e-mails? When you upgraded dbmail from 3.2 to 3.3 there would have been an upgrade from gmime 2.8 to version 3. Can you provide an email that fails to get imported correctly as I'm unable to reproduce the issue. Thanks, Alan
Salve,In order to try to find a fix, can you provide an example or two that get its attachments cutoff? (Remove parts that may impact privacy)Also, please provide also database(version, charset, collation) information. In the past bugs were found based on information related to the setup it self.As far as I know, the main difference(Alan can add more here) between 3.2.x and 3.3.x is the gmime which offers better mime integration and better support for s-mime.Thanks,Have a nice dayCC @ gmail -------- Original message --------From: Valery @.> Date: 6/25/23 11:58 (GMT+02:00) To: dbmail/dbmail @.> Cc: Subscribed @.***> Subject: [dbmail/dbmail] dbmail 3.3.x - Breaks attachments :( (Issue #193) Hi. After almost 6 years of uninterrupted mail server operation (dbmail 3.2.6), I decided to install the latest release 3.3. Already on the first 1000 e-mails, there were sporadic e-mails with broken attachments. My assumption is that when processing MIME type, something does not work correctly and attachments are cut off. Unfortunately I could not find more exact problems in the logs. I have Debian 10, postfix + dbmail + rspamd, postgresql. I only updated dbmail. I really hope that the problem can be solved. Thanks.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>
My mail is running on 2 different LXC containers, one mail server on the other Postgres.
Both have the same Debian version:
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
Postgres:
postgresql-14
postgresql-14-pgmemcache
postgresql-client-14
postgresql-client-common
postgresql-common
config:
data_directory = '/var/lib/postgresql/14/main'
hba_file = '/etc/postgresql/14/main/pg_hba.conf'
ident_file = '/etc/postgresql/14/main/pg_ident.conf'
external_pid_file = '/var/run/postgresql/14-main.pid' file
listen_addresses = '0.0.0.0'
port = 5432
max_connections = 300
unix_socket_directories = '/var/run/postgresql'
ssl = off
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'
shared_buffers = 1GB
temp_buffers = 320MB
work_mem = 64MB
maintenance_work_mem = 204MB
dynamic_shared_memory_type = posix
effective_io_concurrency = 2
max_worker_processes = 2
max_parallel_workers_per_gather = 1
max_parallel_workers = 1
wal_buffers = 16MB
max_wal_size = 3GB
min_wal_size = 2GB
checkpoint_completion_target = 0.4
effective_cache_size = 3GB
log_line_prefix = '%m [%p] %q%u@%d '
log_timezone = 'Europe/Moscow'
cluster_name = '14/main'
stats_temp_directory = '/var/lib/pgsql_stats_tmp'
datestyle = 'iso, dmy'
timezone = 'Europe/Moscow'
lc_messages = 'ru_RU.UTF-8'
lc_monetary = 'ru_RU.UTF-8'
lc_numeric = 'ru_RU.UTF-8'
lc_time = 'ru_RU.UTF-8'
default_text_search_config = 'pg_catalog.russian'
include_dir = 'conf.d'
autovacuum = on
random_page_cost = 2.0
fsync = on
synchronous_commit = off
row_security = off
join_collapse_limit = 3
Mail server:
i A gir1.2-gmime-2.6
i gir1.2-gmime-3.0
i libgmime-3.0-0
i A libgmime-2.6-0
postfix 3.4.23
dbmail 3.2.6 (got it back because of a problem)
pyzor 1:1.0.0-3
razor 1:2.85-4
stunnel4 3:5.55
spamassasin 3.4.6 (on another of my rspamd servers)
What is noticed in different tests: If there is an email received, for example with a pdf, opens normally - I forward it to any client and it comes broken. But at the same time at the host attachment opens normally... :(
I can send you a saved letter with a broken attachment, it is not particularly valuable.
Hello @vixfree yes if you could send an original and the broken one that would be helpful for me to debug, my personal email is [email protected]
Thanks for sending error_attach-pdf.eml The base64 attachment appears intact, though the decoded pdf appears to be truncated as the last stream doesn't have a matching endstream. grep stream "БАНКОВСКИЕ РЕКВИЗИТЫ ООО БАЗОВЫЙ ПОДШИПНИК.pdf" stream endstream stream endstream stream endstream stream endstream stream endstream stream endstream stream endstream stream endstream stream
If you can see the attachment in the incoming mail, all the data is stored properly in the database. The forwarded mail is not coming trough dbmail, this is, possible your MTA issue (exim, for example) or your mail client. We have seen mail truncation when using Thunderbird on some machines, but MS Outlook works well with the same servers.
My organization uses both outlook and thunderbird. With dbmail version 3.2.6, everything works equally and stably. But I can't say the same about version 3.3.1. The strange thing is that truncation works if pdf - there is at least file in attachment and if doc - attachment 0kb. MTA is postfix. If you have any doubts that my settings are wrong I can send you main.conf I don't think they are wrong, the mail server passes all the tests and has been running successfully for 5 years in this mode.
The main change from 3.2.6 to 3.3 is gmime upgrade from 2.8 to 3. It's unlikely there are any config issues with your servers. Local test show dbmail is working as expected with a variety of attachments. Without a failing test it's hard to track down this issue.
I have a suspicion that my problem will inevitably recur. It remains to wait... I can't get 3.3.1 back on the production server right now - the organization needs to work. On the understudy server, there is not enough mail volume to repeat the error. In 2 days there was only 1 time, and the most interesting thing is that I did not see a single mention of it in the logs. How to trace it, except stating the fact of spoiled attachment I do not know.
¯_(ツ)_/¯
One moment which may give some hint about the error, my database gradually migrated from 3.2.3 to 3.2.6, and then also to 3.3.1. I'm not sure what the likely cause, but I suppose that in any process of sending an email with an attachment or receiving it through a client, parsing the letter goes incorrectly. As far as I understood, gmime function implements this, or when requesting the letter from the database, something is wrong or when "assembling the letter". But I'm sorry, I'm not good at this. Comparing two letters (the original source of which is a live attachment and it opens normally (size 41.8kb) found out that when forwarding was cut off only the attachment and for some reason by 9.8kb - (the resulting size of 32kb). All is well with the letter. The mail servers were both on dbmail 3.3.1. But it doesn't matter, if I send a copy of the email to gmail in the same way, the result is the same... Attention - I read the original email normally, the attachment opens. If I send it - the attachment is cut off... I don't know how else to help. That's too bad. I'll wait and see if you find the error.
The dbmail package I built for Debian can be downloaded here, maybe it will help you if I for example built it wrong. Version 3.2.6 works fine!
Resource: ftp://ftp.qbpro.ru
debian/buster/...
It may be related to some utf-8 characters that are not handled the way it is supposed to be. The changes from gmime 2.3 to 3 was a big step, If I’m not late to the game, cand the mail in question can be send to me also?
It may be related to some utf-8 characters that are not handled the way it is supposed to be. The changes from gmime 2.3 to 3 was a big step, If I’m not late to the game, cand the mail in question can be send to me also?
Please email [email protected]. I will send the same example that I sent earlier, there are no others, as the server has rolled back to the previous version and so far everything is stable.
It may be related to some utf-8 characters that are not handled the way it is supposed to be. The changes from gmime 2.3 to 3 was a big step, If I’m not late to the game, cand the mail in question can be send to me also?
Please email [email protected]. I will send the same example that I sent earlier, there are no others, as the server has rolled back to the previous version and so far everything is stable.
Sent an email. Please check also the spam folder.