docker-mailserver
NFS-Related Quirks Requiring Poor Defaults
This somewhat is piggybacking off of #163 and #162
Currently, the Docker Mailserver helm chart isn't compatible With NFS-backed PVs that are configured in a sane manner. Notably, is related to this issue on the main docker-mailserver github page. Currently, the image utilizes a lot of root-level file manipulation (particularly chown as root). Most containers should behave well under NFS-backed PVs given that the PV is pre-edited with group access and a GID matching a Pod's securityContext.fsGroup.
Docker Mailserver's helm chart doesn't appear behave in a functional way, utilizing a lot of file creation and then file ownership manipulation as the root user inside the container, meaning that if the NFS share is configured with the root_squash parameter (which is by default), the Pod will have a lot of issues of trying to reassign ownership these files and directories.
So far in testing, supplying a matching owning GID for the volume and fsGroup GID alone isn't enough, like it really should be. Wiping contents of and redeploying the NFS share with no_root_squash appears to solve issues for the container (at least in the mail-config volume, others have not been tested yet).
Based on factors present in the above mentioned issues, if DMS can run as a non-root user, this issue will likely fix itself. Otherwise, documentation should be created regarding the limitations and quirks of running DMS' helm chart with NFS-backed PVs.
Happy to review a PR. I've never actually used NFS, so I'm not going to be much help on this one.
if DMS can run as a non-root user
I don't think it can. Unlike most containers DMS is multi-process/multi-service. We have an entrypoint that will do initial setup for the container, then process management is handled by supervisord for each service DMS bundles.
Various services run as their own users and file ownership for security benefits, and since DMS is mostly integration glue mapping ENV and config to changes for each services config, we need to edit or create files for those services.
Likewise for convenience we have all runtime state symlinked to a central location for a single volume mount for all services to persist through. When our images packages are adjusted, or related changes occur that can affect the implicit UID/GID assignment for those services, then that persisted state needs to be corrected for a smooth image upgrade. As such we require the ability to use chown / chmod.
The issue you reference will not address these concerns (although it is focused on improving the situation with LDAP for /var/mail). However the problem should only affect your storage attached for persistence, so much of what I described is entirely fine within the containers own filesystem locally. I am not familiar with your usage of NFS (NOTE: I also have no experience with k8s/helm), but provided it's only impacted by the locations of our standard volume mounts, you can perhaps limit what volumes are NFS?
Perhaps an alternative to DMS is better suited for users with NFS if the ownership concern is problematic there, I assume it's not specific to k8s, but would be similar for a simple Docker container with NFS volume? Perhaps look into Mailcow (assuming they have a helm chart) as I think they take the approach of multiple containers, one for each service.
The main quirk that prevents seamless management from NFS volumes is specifically chown and other operations on files done by the user 'root', which is treated (by default) as an unprivileged user, causing errors. Given that it is currently a hard requirement for the image to redefine files in runtime via chown, it is unlikely that NFS will be made to be fully compatible without needing no_root_squash on utilized shares.
In my case, I will be testing the other default volume mounts to see what quirks if any might exist, but can get away with just the config PV being NFS and others being local storage. Additionally in my case, I do have my NFS configuration as a whole setup in a manner that needing no_root_squash is of mitigated risk, though of course this is not something every NFS setup will be capable of. Once I've done more testing to see if there are other quirks I might have missed (beside latency-related issues), I'll be looking into a PR for the essential notes in regard to using NFS shares with the current DMS chart.
As a side note, Mailcow's design seems to make it as a whole difficult to outright incompatible in a k8s environment (relying on docker-specific features, no helm charts nor recommended default k8s deployment/statefulset configuration).
While testing, just discovered another quirk relating to NFS volume permissions. Currently, rspamd-dkim (via setup config dkim keysize 2048 for sample dkim config) if the mail-config mount's permissions are below 7#5, as the directory and subdir permissions don't seem to be forcibly set during creation alongside chown for the _rspamd user. Additionally, processes run under the user _rspamd don't get the additional fsGroup if set in securityContext, meaning the group permissions and group ownership (either #5# or #7#) in the mail-config volume root don't really provide anything meaningful to the mail-config volume as either everything is owned by root and/or needs chown permissions, or doesn't inherit the fsGroup GID.
Manually modifying the permissions manually inside the container to allow rspamd to function properly theoretically could work as long as there isn't something that will modify them later.
Additional note when using the stock PV config: the mail-config volume (mounted at /tmp/docker-mailserver/ is left at 777 permissions. The files inside are not this permissive, but this permission would allow for any process in the container to write custom files or delete and replace any file inside the directory.
As I'm not experienced with k8s/helm related config I cannot comment much on decisions there. In Docker Compose a users volume mounts will replace anything our image had originally so it's permissions may vary. We could possibly attempt to correct that like we do with other volume managed content 🤔 (not sure if we can apply changes to the mountpoint dir itself, at least I know there are caveats with file bind mounts which we discourage)
The DMS config volume (/tmp/docker-mailserver) should generally be root owned files, I am aware of the Rspamd ownership, at least for DKIM. That is planned to be unified with OpenDKIM to generating DKIM keys agnostic to the service tool, with the key copied into the container when ownership can be managed instead, along with config generation.
For local storage-backed volume mounts, you can chmod and chown with out any restrictions (assuming adequate permissions). With NFS, chmod works the same as local storage and chown works if the NFS share has no_root_squash in its flags.
I am getting close to verifying a setup that should work with all four default persistent volumes in the helm chart bound to NFS share(s). I however cannot find neither here nor on the DMS docs if any of the volumes may be affected negatively in replication / high availability scenarios, if DMS' helm chart supports running more than one replica. Before I get to work on appending to the chart docs with my testing, I would like to confirm whether or not some of the volumes should be treated as pod-exclusive in replication scenarios.
I believe there was an issue or two on the main DMS repo regarding feature request for HA / cluster support? It did not receive a contribution.
Similarly I think you'd find some issues mentioning NFS. IIRC Dovecot docs have some guidance on proper NFS support / compatibility. We also in the past used flock for locking instead of lock files within some DMS scripts and that was better with flock (I think the current file lock approach might have been implemented a bit wrong too, but no complaints thus far). The change to lock files was contributed some years ago by a user that claimed flock wasn't compatible with NFS, but that seems like it might have actually been misconfiguration on their end.
I don't personally have experience with what you're wanting to do, so I can't really say how well it'd work in practice 😅
if DMS' helm chart supports running more than one replica.
I have never tested more than one replica. I do not know if previous maintainers have. So I would say it is not supported.
One last thing before I get to ensuring that suggestions I make in a docs-related PR aren't going to lead to a broken configuration. I have done testing at this point with being able to send/receive emails internally, externally according to my use case. My use case however doesn't utilize ldap nor pop3. Existing filesystem permissions in my current scenario look like everything should be fine. Are there any dynamically generated services that I may not have seen filesystem structure creation within the default PV regions?
I don't think so...but I'm not an expert on what dockermailserver does under the hood.
@j-heffron can this be closed since https://github.com/docker-mailserver/docker-mailserver-helm/pull/166 was merged? I think you were going to follow up with a 2nd PR with updated documentation? Do you still have time to do that?
More or less this can be closed. I thought I had sent a second PR already. Submitting the second PR shortly.