Enable zlib compression for maildir

[ponychicken]

This reduces the disk space needed for mailbox storage. It reduced the size of a 23.6 GB large maildir to 9.6 GB. It doesn't touch existing mails, just compresses newly sent/received ones.

[JoshData]

I was thinking we might want to migrate to a different mail storage format entirely:

http://wiki2.dovecot.org/MailboxFormat/dbox

[ponychicken]

I was wondering why you choose milder in the first place. We could add a switch to use milder only for existing installs and use dbox for new setups. A migration can take place using doveadm sync, but I don't think we should run it during the upgrade process as it might take a long time. Maybe adding an async button to start this operation in the admin panel?

Having some kind of doveadm sync intergration would be also great for migrating old IMAP accounts to mailinabox. This could make the on boarding process much easier. (I had to run imapsync manually)

[devzzm]

Regardless of if write zlib support is added, I think the server should still support read zlib out-of-the box. It makes it easier to migrate mail from other servers into this one. And all it takes is adding this line somewhere in the dovecot conf:

mail_plugins = $mail_plugins zlib

I'd be glad to submit a pull request for it if it would be accepted

[devzzm]

I think it makes sense to make this issue only be about zlib compression in the maildir, and then have a whole separate issue regarding changing the mailbox format and migration/conversion for existing users...because that seems like a much bigger task

[ponychicken]

I agree with @phareous, enabling zlib is a very straightforward thing, that needs no migration path or conversion. I've rebased the patch on the latest released version

[bronson]

Just curious: can anyone think of a downside to enabling this?

There are some gotchas listed on http://wiki.dovecot.org/Plugins/Zlib. But, as long as it's not compressing existing mails, only compressing newly-arriving ones, then it looks safe and very desirable. To me anyway.

[ponychicken]

I've been using it without problems, and it reduced the size of my mailbox greatly (I manually recompressed also old mails, but that is not part of this patch)

[dofl]

I think this is an excellent upgrade. It would be nice to see it integrated.

[niieani]

This is awesome. I've been using it in production too. Works really well!

[bronson]

Any chance this could be merged? I see lots of upside and no real downside.

[JoshData]

Dovecot's mail storage format conversion tool basically doesn't work, so I am afraid to make any changes to the storage format because it may be very difficult to go backwards or to move to a different format in the future.

[bronson]

Agreed, enabling zlib compression would be painful to undo. It doesn't seem like it makes it any harder to move to a different mail format though -- that's going to be hell either way.

Personally, I'm glad you went with maildir. It means I can look at my mail spools and process them with other tools. I found dbox brittle and opaque. I used it once years ago... the conversion tools were terrible and after a lot of frustration I ended up just rm -rfing everything. Hope to never repeat that experience.

But my mail use is pretty light. Having a 7TB mail spool (as in #531) might change things!

[bronson]

FWIW, I think I'm going to enable zlib compression on my box this week... It should really pack down giant email threads and, when it comes time to move mailbox formats, I don't think having them zlib compressed will make things any harder.

I'm not going to convert existing maildirs of course. Figuring it's better to act now, instead of waiting until my disk is 75% full before throwing the switch.

[marcosms]

OK. I supposed that to enable zlib compression in mail-in-a-box, i only have to do this: (Please, confirm that this is all right).

1. Change the file /etc/dovecot/dovecot.conf to add this lines (I added all of them at top of the file)

mail_plugins = $mail_plugins zlib

plugin { zlib_save_level = 6 # 1..9; default is 6 zlib_save = gz # or bz2, xz or lz4 }

2. change the file /etc/dovecot/conf.d/20-pop3.conf . You have to find this line

mail_plugins = $mail_plugins antispam

And change it to this line

mail_plugins = $mail_plugins antispam imap_zlib

And then restart the server. ¿It's all right?

OK. Now if it it's right ¿How can I compress the existing mails? I supposed that i can use the scritp of this page:

https://gist.github.com/cs278/1490556

But... before try it i want to be sure that this don't crash the system. ¿Anyone know how to compress all the maildirs?

Thanks.

[hopfi2k]

http://wiki2.dovecot.org/Plugins/Zlib http://wiki2.dovecot.org/Plugins/Zlib

read the last paragraph ;-)

Am 08.02.2017 um 10:42 schrieb marcosms [email protected]:

OK. I supposed that to enable zlib compression in mail-in-a-box, i only have to do this: (Please, confirm that this is all right).

Change the file /etc/dovecot/dovecot.conf to add this lines Enable zlib plugin globally for reading/writing:

mail_plugins = $mail_plugins zlib

Enable these only if you want compression while saving:

plugin { zlib_save_level = 6 # 1..9; default is 6 zlib_save = gz # or bz2, xz or lz4 }

change the file /etc/dovecot/conf.d/20-pop3.conf . You have to find this line mail_plugins = $mail_plugins antispam

And change it to this line

mail_plugins = $mail_plugins antispam imap_zlib

And then restart the server. ¿It's all right?

OK. Now if it it's right ¿How can I compress the existing mails? I supposed that i can use the scritp that there is in this page:

https://gist.github.com/cs278/1490556 https://gist.github.com/cs278/1490556 But... before try it i prefer ask you. ¿Anyone know how to compress all the maildirs?

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mail-in-a-box/mailinabox/pull/510#issuecomment-278279210, or mute the thread https://github.com/notifications/unsubscribe-auth/ACMPCwfEKj3JzLqArgxOWK6ukaROQ_Zxks5raY3vgaJpZM4FjZOM.

[marcosms]

OK. But i am searching any method to automatize this job. I have about 80 mailboxes and it's hard to do this on every folder.

I have a raspberry with iRedMail running on it. I try the script that I put in my last message in the raspberry, but it appears that it doesn't do anything, so i will not try to use it in the production machine with mail-in-a-box while i'm not sure if it works.

¿the code of the point 1 in my last message is right? with this code I don't see any change in the work of mail-in-a-box. I suppose that it is compressing new mails, but i don't know how to see if it works.

[hopfi2k]

Maybe this? https://github.com/extremeshok/dovecot-maildir-compression https://github.com/extremeshok/dovecot-maildir-compression

Am 08.02.2017 um 13:01 schrieb marcosms [email protected]:

OK. But i am searching any method to automatize this job. I have about 80 mailboxes and it's hard to do this on every folder.

I have a raspberry with iRedMail running on it. I try the script that I put in my last message in the raspberry, but it appears that it doesn't do anything, so i will not try to use it in the production machine with mail-in-a-box while i'm not sure if it works.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mail-in-a-box/mailinabox/pull/510#issuecomment-278309714, or mute the thread https://github.com/notifications/unsubscribe-auth/ACMPC0KS8p4oq5mTYQNkG--EsvqqDmn2ks5raa6agaJpZM4FjZOM.

[marcosms]

XD

This is the same scritp that I put in my first message. Anybody tryed the code? has someone tried that script?

[marcosms]

OK. I run the script and... nothing happens, like happend with my raspberry with iredmail. First, the script says that the folder /tmp/mail/srv/mail didn't exist, so i make the folder, but the script ends as fast as i push the enter key without any message. I did this (I rename the script to "comprimircorreo.sh":

root@box:/home/marcosms/scripts# chmod +x comprimircorreo.sh root@box:/home/marcosms/scripts# ls backup1 comprimircorreo.sh root@box:/home/marcosms/scripts# ./comprimircorreo.sh find: `/tmp/mail/srv/mail': No such file or directory root@box:/home/marcosms/scripts# cd /tmp root@box:/tmp# ls 4XlVAwN1T5 nsd.1266.task.0 nsd.1266.task.1 root@box:/tmp# mkdir mail root@box:/tmp# cd mail root@box:/tmp/mail# mkdir srv root@box:/tmp/mail# cd srv root@box:/tmp/mail/srv# mkdir mail root@box:/tmp/mail/srv# cd /home/marcosms/scripts root@box:/home/marcosms/scripts# ls backup1 comprimircorreo.sh root@box:/home/marcosms/scripts# ./comprimircorreo.sh root@box:/home/marcosms/scripts#

[dofl]

@JoshData Any progress on moving to the new mailbox format as suggested in 2015? After a couple of years my 40 GB VPS is getting fuller. The problem of not enough storage will only become a problem for more, and I still don't see any problems with enabling zlib.

[qyz]

FWIW, I'm storing the separate parts of /home/user-data on separate zfs file systems. zfs performs very well for mail server load, supports transparent compression and -- most importantly -- automatic snapshots with zfs-auto-snapshot.

root@mail:~# zfs list
NAME                              USED  AVAIL  REFER  MOUNTPOINT
zpool                            9.79G  4.68G    96K  /zpool
zpool/user-data                  9.67G  4.68G   116K  /home/user-data
zpool/user-data/backup            388M  4.68G   384M  /home/user-data/backup
zpool/user-data/dns              1.98M  4.68G   140K  /home/user-data/dns
zpool/user-data/mail             3.55G  4.68G  3.04G  /home/user-data/mail
zpool/user-data/owncloud         5.33G  4.68G  5.22G  /home/user-data/owncloud
zpool/user-data/owncloud-backup   272M  4.68G   272M  /home/user-data/owncloud-backup
zpool/user-data/ssl               336K  4.68G   256K  /home/user-data/ssl
zpool/user-data/www               144M  4.68G   144M  /home/user-data/www
root@mail:~# zfs get compressratio,usedbydataset,logicalused zpool/user-data/mail
NAME                  PROPERTY       VALUE   SOURCE
zpool/user-data/mail  compressratio  1.79x   -
zpool/user-data/mail  usedbydataset  3.04G   -
zpool/user-data/mail  logicalused    6.08G   -

[jvolkenant]

There are come caveat's with using zlib with dovecot. https://wiki2.dovecot.org/Plugins/Zlib

A better option may be to use btrfs with compression or zfs, both available with 18.04. Keep in mind, there will be a performance tradeoff with compression.

[binarykitchen]

Hi @ponychicken

This PR is > 8 years old. It is something you still want to achieve? If so, would you like to rebase with the latest master, resolve the conflicts and raise the remaining tasks, or else archive this PR?

We understand that you have invested some time in this and appreciate the efforts. But at the same time, we'd like to tidy up this repository. Thank you for your understanding.