immich
immich copied to clipboard
immich-app
[Feature]: Support more storage backend
Feature detail
Right now, only a local folder is supported natively. Even though more can be supported through fuse mounted filesystems, but this method has drawbacks (performance hit mainly), so it would be nice to have support for more back-ends eventually.
S3 is an obvious choice that comes to mind and that would support a lot of providers, but there could be others (like WebDAV perhaps).
Platform
Server
Would this be local storage for thumbnails, and remote storage for files? That would maintain speed for scrolling.
Then you would need to pass a Uri to the app to retrieve the full file for viewing?
If it allows to have a backup also for the media, that would be awesome. Like 2 storage backends - s3 and G-drive. If one of them is not reachable, switch to other. This could be defined in a priority list.
In my mind it could have been for all objects, but @palitu you're right that thumbnails could still be local. And yes, @akashkj, that's how I felt about it. Having two or more storage back-ends possible, that can either work as RAID0 or RAID1 (not exactly RAID, but either mirror or concatenate the available storage).
In my mind, supporting a well defined platform such as S3-like APIs is a good idea and is something we can look into after the first major release is completed. As for the other idea of having multiple backends that we switch between, in my mind that is the job of the storage layer, not something that immich should natively support. We'll leave this here and probably get around to it at some point in the future!
Really great to hear your desire for an S3 type backend. FYI, i use the local filesystem for most of immich, then a mounted NFS share for the main photo storage. My server doesnt have the local storage for the bulk data (especially with all the video's), but thumbs is more than appropriate for local.
I do think that there are some delays with video playing as a result of this, but i assume in the future it will be improved with this S3 feature.
ta
I just started writing this up as a new feature request before discovering this one. I would love to be able to use S3 storage for all blob objects and make a raw disk volume optional. Big +1 to this feature request!
At home I run a Nomad cluster, like a Kubernetes cluster but with less yaml involved. We use CSI-managed iSCSI storage volumes that are mounted directly into running docker images (or other tasks). The preference here is a single writer job and maybe multiple read-only mounts for other volumes. This means that sharing a directory is hard while communicating over the network is easy. Right now the docker-compose files seem to suggest mounting one volume to multiple hosts that all have write access which is a little scary (and hard to do with iSCSI). The performance hit of switching to NFS is very visible on our network.
There's an official TS SDK for all the bells and whistles MinIO provides, but I'd strongly suggest looking at just using the S3 SDK which would allow folks to use any S3-compatible storage backend.
This is of course asking a lot, adding a new storage backend like S3 is completely different than writing to disk 😄 I'm perfectly happy to be patient for this sort of feature.
Would like to that I am interested in S3 storage access as well to offload.
I try mounting S3 with rclone and found that immich uses softlink which is not allowed in S3.
I guess this is one of the things to consider when implement this in the future.
Error: ENOSYS: function not implemented, link 'upload/04f8f839-9ee4-47fc-89b1-fb3883cf3008/original/WEB/33809e6c-1f1f-4bec-a8cc-7a0367493e8d.png' -> 'upload/04f8f839-9ee4-47fc-89b1-fb3883cf3008/2022/2022-12-11/hahaha.png'
