feat: Libraries including existing file scanning

[etnoy]

This PR adds a new top-level asset container called libraries. This is an effective and easy-to-use implementation of syncing existing files. This is the most-requested feature of immich, see #7, #34, #541, #1006, every reddit thread etc. etc.

Status: experimental

In addition to "upload" libraries (traditional immich), it also adds support for an "import" library that can take any number of "import paths" that will be scanned for existing assets. An import library is matched against its underlying filesystem, and if assets are removed or changed, the library can be rescanned to update its contents.

NOTE: currently there's no way to add an import library from the UI, that has to be done through the API via the POST /api/library endpoint

User can set import paths from the web, and perform the following operations:

Scan library: crawls the import path(s) and imports any new assets. Any existing assets with modified mtime are re-imported Force scan library (TODO): Same as above but re-reads any information from disk Empty trash: In the above scans, a file is marked as "offline" if it can't be read, or if it is not in the import paths. The Empty Trash job removes all offline files from the database.

Automatic filesystem watching should be very easy to implement after this PR

A special big thanks to @jrasm91 for your patient support

TODO:

• The new isOffline field of assets must be handled in some way
• Tests
• Currently no validation of user input
• Currently no authorization in library code
• There is support for per-library exclusion of paths, but there's no UI for it yet
• Frontend UI is very basic
• Documentation
• Thinking about edge cases.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
immich	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 20, 2023 10:59am

[EnochPrime]

Super excited for this. Thinking about the isOffline and trash parts it seems you could leverage that and generalize it to handle deletion of any asset. Have offline/trashed assets not appear in views and add a trash view that shows only those assets.

[etnoy]

Super excited for this. Thinking about the isOffline and trash parts it seems you could leverage that and generalize it to handle deletion of any asset. Have offline/trashed assets not appear in views and add a trash view that shows only those assets.

This will not be possible to generalize to other assets since only imported libraries have an external source of truth

[EnochPrime]

This will not be possible to generalize to other assets since only imported libraries have an external source of truth

What I meant is that the user could delete a file and it would just mark it as isOffline in the db first. Now I realize that would conflict with the library files being back online when rescanning.

[etnoy]

There have been discussions about a timed trash system, but it's not planned right now

[titro]

How is the import logic? Does it preserve the file structure with all subfolders? or does it only recognize all files in the configured folder?

[etnoy]

How is the import logic? Does it preserve the file structure with all subfolders? or does it only recognize all files in the configured folder?

At the current state it will ingest all files in a given folder recursively. There is support for ignoring paths internally, but there is currently no UI to configure that

[titro]

sounds fine. but does it preserve the folder/subfolder from the source or does it only import all files recursive? Would be great if you yould have the same folder / album structure like in your filesystem.

[etnoy]

sounds fine. but does it preserve the folder/subfolder from the source or does it only import all files recursive? Would be great if you yould have the same folder / album structure like in your filesystem.

Immich has no folder structure, all things are shown in a flat timeline.

[titro]

sounds fine. but does it preserve the folder/subfolder from the source or does it only import all files recursive? Would be great if you yould have the same folder / album structure like in your filesystem.

Immich has no folder structure, all things are shown in a flat timeline.

I know, that's why i asked if that is possible for that new functionality :-)

[etnoy]

I have future ideas that might end in certain libraries browseable with folder structure, but it's not a given that such a feature will be implemented. At least when you import files with this library, it stores all path metadata (i.e. folder structure) in the backend if that is to be used in the future

[EnochPrime]

My understanding is that the intent of the library feature is to use your existing data structure as-is. The files aren't uploaded to the upload folder they are used in place and aren't rearranged.

[etnoy]

My understanding is that the intent of the library feature is to use your existing data structure as-is. The files aren't uploaded to the upload folder they are used in place and aren't rearranged.

The files are uploaded in place, that is what this PR brings. But immich only supports browsing in a big timeline and that will not change with this PR

[titro]

My understanding is that the intent of the library feature is to use your existing data structure as-is. The files aren't uploaded to the upload folder they are used in place and aren't rearranged.

that was also my understanding what the gallery function should be that's why i asked etnoy. But it seems that this is not the case (in the first release). So at the end the same gallery behavior, only the import is different, right? :-(

[etnoy]

My understanding is that the intent of the library feature is to use your existing data structure as-is. The files aren't uploaded to the upload folder they are used in place and aren't rearranged.

that was also my understanding what the gallery function should be that's why i asked etnoy. But it seems that this is not the case (in the first release). So at the end the same gallery behavior, only the import is different, right? :-(

This is a huge step in the right direction for you. So try not to be negative

[Yacubane]

I'm sorry to bother you, but I'd like to make sure I understand things.

The files are uploaded in place, that is what this PR brings.

So all files from path that I chose will be uploaded to Immich and I would end up with duplicates on my drive? I quickly looked at code and I think that it's not – you store path for them, and they are all marked as read-only. If that's true, then I think that it's better to use words that "files are imported in place", by which I mean that files that user imports aren't copied or moved – they will be used within Immich from their original location.

[etnoy]

No, files will not be duplicated. They are read from whatever paths you specify and served from there. If you've used Plex before, it's the same idea. They are not copied over to a new location, however remember that Immich needs to generate metadata like thumbnails and video transcodes so for a large library you also need to have a reasonably large data folder for immich.

[Yacubane]

Yeah, so I believe that there was some miscommunication error, @titro 😄 I hope that you understand it well now.

Amazing work @etnoy! I'm sure that a lot of people will be grateful for implementing this! 🥳

[rxerium]

Do we have a rough timeline as to when this feature will be release? Thanks

[phoenixxx-1]

Firstly, thankyou @etnoy for the commit! By far one of the most anticipated features, awesome job :)

There is support for ignoring paths internally, but there is currently no UI to configure that

Is this possible to configure on the CLI when importing? I use a Synology NAS which creates hidden @eaDir folders containing thumbnails of the images in the parent folder. There is no way to disable these, as such when running the import, the CLI is processing the thumbnails as well as the full size originals, causing low-res duplicates.

If the answer is to just wait for the UI - that's cool too! I appreciate this is under very active development.

Thanks.

[andyxpert]

Awesome job man !! Looking forward to having this feature in an upcoming release.

One idea though - Most of us handle moving the files on our own with other automations but then we need to manually ask Immich to import them (4 example I have a script that adds new files in an external drive that I added to Immich as a read only library but I still have to manually run the Import CLI command). Having a job that does that would be great (importing the existing new files from the readonly drive)

[sebastianhorwege]

Keep on going! I would love to see this PR in Production ;-)

[etnoy]

Do we have a rough timeline as to when this feature will be release? Thanks

The "easy" bulk part is done, the difficult work with getting all the details right is undergoing. Now during summer vacation, things will slow down significantly, so don't expect it to land too soon.

[etnoy]

Firstly, thankyou @etnoy for the commit! By far one of the most anticipated features, awesome job :)

There is support for ignoring paths internally, but there is currently no UI to configure that

Is this possible to configure on the CLI when importing? I use a Synology NAS which creates hidden @eadir folders containing thumbnails of the images in the parent folder. There is no way to disable these, as such when running the import, the CLI is processing the thumbnails as well as the full size originals, causing low-res duplicates.

If the answer is to just wait for the UI - that's cool too! I appreciate this is under very active development.

Thanks.

This code is still in experimental stage. The db formats haven't been decided upon yet, UI parts are coming later.

[pharmarin]

Hello, thanks for your hard work for this very important feature ! I believe that glob should be in the dependencies array of the server. It is actually in the devDependencies. https://github.com/immich-app/immich/blob/8fc843dd4dd1db9da9211ef807fecf8c1ec1e8aa/server/package.json#L115

[etnoy]

Things are moving along nicely. I've rewritten the PR description and added new screenshots to show you the all-new user interface that allows you to create and scan libraries.

We now also support exclusion patterns, another oft-requested feature that allows you to exclude paths from being scanned.

In fact, at this point I want to invite daring users to try out this feature branch. Don't do this to your production instance, but give it a spin. Add your existing assets and see how it works out

Let's hope that I can submit this to the rest of the team and we can get it merged soon!

[cartman005]

I'm so excited for this feature. I am confused about one thing from the documentation. Towards the top, you state:

When assets are added, modified, or deleted, immich will re-read any changes.

But then in the list of features to consider for later, you list:

Filesystem watching (automatically rescan the library as soon as the filesystem changes)

So will this feature detect new files that are added to my import path or no? Do I just have to trigger the scan manually?

[etnoy]

I'm so excited for this feature. I am confused about one thing from the documentation. Towards the top, you state:

When assets are added, modified, or deleted, immich will re-read any changes.

But then in the list of features to consider for later, you list:

Filesystem watching (automatically rescan the library as soon as the filesystem changes)

So will this feature detect new files that are added to my import path or no? Do I just have to trigger the scan manually?

For the time being you need to initiate the scan manually. In the future this could be done automatically.

[cartman005]

For the time being you need to initiate the scan manually. In the future this could be done automatically. That works for me! Thank you!

[EnochPrime]

External libraries imply that the contained assets are read-only. This is not indicated in the UI. For instance, the delete button is still visible but clicking it returns a server error

My preference here would be that the admin can control whether the user has write access to the library and if so allow deletion from immich.