Lose entire library after drive disconnection

[cactushydrocodone]

Hi,

Im currently not using jellyfin because of the android apps, but thats another point. My question is, is this age old bug fixxed:

• Have medialibrary on a FUSE mountpoint
• Sometime you loose the mountpoint (Connection issue, Process reaper, unexpected circumstances, etc)
• Emby (bzw. jellyfin) deletes the entire library because it cant find the files anymore
• After a remount and restart of emby (bzw. jellyfin), the entire library is gone
• To restore, you need to do a full rescan, which can take days (My library is 60TB).

Workaround i had for this the problem:

• Keep a daily backup of the emby (bzw. jellyfin) data folder and restore it on mountpoint loss.

The original emby devs basically said: "ye we fix that, sometime." That was 3 years ago.

Maybe i have luck with you guys.

[cvium]

It's not fixed yet, but we want to. Personally, I prefer this model: https://support.plex.tv/articles/200289326-emptying-library-trash/

[EraYaN]

Kodi keeps the metadata in it's database for a while, and so re-adding the file at the same path will repopulate all data and watched status etc.

This should be added to the feature request page at https://features.jellyfin.org/ as something like "Support intermittent access to media files without removing files from the library." and we should probably add a flag to the UI to "offline" media.

[joshuaboniface]

I've never actually experienced this myself, either on Emby or Jellyfin, despite my CephFS volume going away all the time. Is there some particular action that triggers this? Even if the files aren't there, Jellyfin shouldn't be randomly deleting the entries in the DB, unless I'm missing something?

[EraYaN]

I believe a scan can clean up “removed” files.

[cactushydrocodone]

I've never actually experienced this myself, either on Emby or Jellyfin, despite my CephFS volume going away all the time. Is there some particular action that triggers this? Even if the files aren't there, Jellyfin shouldn't be randomly deleting the entries in the DB, unless I'm missing something?

Its as easy as this:

1. Emby / Jellyfun runs in a container, the FUSE mount is added with a docker mount
2. When the FUSE Mount on the host "goes away", it also goes away in the container
3. Emby / Jellyfin cant find its files anymore, at the next rescan, it begins to delete everything it doesnt have anymore (I do use a regular scan every 4 hours)
4. Even if you dont have a regular scan activated, at the next restart of the container (emby/jellyfin), it deletes ALL METADATA, EVERYTHING. All knowledge it ever had about those files.

[cactushydrocodone]

Another day on which ive lost a library. The full rescan will take days, weeks.

Why did the emby devs did this like this? It makes no fucking sense, im sorry.

[joshuaboniface]

I (well, @anthonylavado) did find this from their forums, which basically says that this is intentional behaviour to let Emby (and hence JF) be "self maintaining". And I see their point - if you delete a file, then rescan, the intent is that that file will be detected as missing and be removed from the library. So that does make sense in a way. That said, I definitely think it needs to handle the "whole library is missing" situation a little better. From their talk there it seems like it's not a simple problem, and...

Why did the emby devs did this like this?

Yea, that comes up a lot in our discussions...

This discussion was occurring around 3.2, so any changes they made should be in Jellyfin's codebase, unless it was later remove. They're never detailed enough in their comments there ("check latest beta" and similar shit, rather than "see this PR for the fix"...) to really know where or when it was fixed without serious digging, but it should be date-searchable if someone really does want to dig deep into it.

As a quick workaround, my first though would be to set some sort of heath check in the Docker container that checks if the mount is available (mount | grep /mountpoint, or test -f /mountpoint/testfile_that_is_never_deleted, or something to that effect) and kill the container if the healthcheck fails. Or disable auto-scanning, and scan on some sort of event in the filesystem (i.e. use inotifywait) via the API. Neither is a particularly nice workaround in Docker I don't think - I depend Jellyfin on my CephFS mount unit but that's at a VM level - but should definitely prevent the full failure you're getting.

Another option might be to create a "library" filled with symlinks to the FUSE filesystem, and then scan that "library", so that technically the files never "go away", which is one heck of a hack but should also work if my understanding of JF's file scanning is correct.

[JustAMan]

Another solution from JF point of view might be adding a checkbox which effectively disables removing files from a database even if they're gone, like if you never intend to delete a file, plus making a library rescan which is run manually to drop the files.

So they never get lost by themselves but the administrator can remove them and force a re-check when he/she is sure the library is present.

[cactushydrocodone]

I (well, @anthonylavado) did find this from their forums, which basically says that this is intentional behaviour to let Emby (and hence JF) be "self maintaining". And I see their point - if you delete a file, then rescan, the intent is that that file will be detected as missing and be removed from the library. So that does make sense in a way. That said, I definitely think it needs to handle the "whole library is missing" situation a little better. From their talk there it seems like it's not a simple problem, and...

Why did the emby devs did this like this?

Yea, that comes up a lot in our discussions...

This discussion was occurring around 3.2, so any changes they made should be in Jellyfin's codebase, unless it was later remove. They're never detailed enough in their comments there ("check latest beta" and similar shit, rather than "see this PR for the fix"...) to really know where or when it was fixed without serious digging, but it should be date-searchable if someone really does want to dig deep into it.

As a quick workaround, my first though would be to set some sort of heath check in the Docker container that checks if the mount is available (mount | grep /mountpoint, or test -f /mountpoint/testfile_that_is_never_deleted, or something to that effect) and kill the container if the healthcheck fails. Or disable auto-scanning, and scan on some sort of event in the filesystem (i.e. use inotifywait) via the API. Neither is a particularly nice workaround in Docker I don't think - I depend Jellyfin on my CephFS mount unit but that's at a VM level - but should definitely prevent the full failure you're getting.

Another option might be to create a "library" filled with symlinks to the FUSE filesystem, and then scan that "library", so that technically the files never "go away", which is one heck of a hack but should also work if my understanding of JF's file scanning is correct.

This is the best answer ive got on that topic till yet. Ive made some topics on the emby forum, but got my issue never resolved / never got any new ideas like the ones you brought to the table right now.

I tried jellyfin a couple days ago, its really neat and im looking to replace my proprietary emby with jellyfin as soon as possible.

[joshuaboniface]

Thanks, hopefully at least one of those workarounds helps :-)

[EraYaN]

Well at some point we could adopt the "Kodi" way, it can detect a file missing and a full mount point / station missing, and handles these differently when cleaning the database. (It asks when the whole mount is missing otherwise it just cleans em up)

[hereisderek]

please have a look into this, this is much needed

some potential measurement includes:

• allows the user to manually opt in to manual clean up only
• allows the user to select the auto clean schedule (weekly monthly)
• only clean up the metadata that's expired for a specified time

[luispabon]

Kodi fixes this problem by only adding new found media when rescanning, and having a library cleanup action that removes missing files as a separate action. The library cleanup has to be triggered manually.

[PrplHaz4]

Kodi fixes this problem by only adding new found media when rescanning, and having a library cleanup action that removes missing files as a separate action. The library cleanup has to be triggered manually.

Sounds like a reasonable approach - but probably requires the ability to handle missing media gracefully on all clients...

[MattMckenzy]

I've had this issue at least once a month since I've started using Jellyfin, it's so very frustrating to spend hours recreating all my libraries. I'm about to fork Jellyfin and try to fix this myself I think.

Thought of a good solution: create a hidden file at root of library upon creation. Check for file when scanning and if not found, don't clean up library but log an error and skip scanning it.

[arabcoders]

While i think this problem should be fixed in jellyfin, as workaround, you could have a cronjob that check for mount status, if it's not available "transport disconnected" or w/e, you could kill off jellyfin instance until the mount become available again, it's a batch work job but at least you dont have to rescan the entire library specially when it's in the cloud and mounted via rclone or whatever. I think there is a related config option for systemd that kill off services if mount is disconnected https://blog.thewatertower.org/2019/12/29/mount-dependencies-with-systemd/

[joshuaboniface]

I'm about to fork Jellyfin and try to fix this myself I think.

Well, we'd absolutely welcome a PR to fix this, since no one so far has had the time or inclination to fix it.

[MattMckenzy]

I think I might have gotten something working for this. I've created a draft pull-request at #8814 in case someone would like to have a look or try it out!

I still want to build it as a container and deploy it to my home set-up for testing before I sign off on the pull request, hopefully it won't take too long.

[Skalmansbror]

I agree. This is an absolute disgrace. I have a collection of ~2000 digitally recorded old VHS movies. The task of getting posters and backdrops for them all was a REAL hassle. Weeks and weeks of work. I finally got it all set up, and I was thrilled to be able to browse my gallery digitally.

My NAS then lagged a bit after rebooting, jellyfin recognized no files were present, and wiped my entire library. To go through it all again feels absolutely disheartening, especially since it can happen again at any moment!

Fix this issue asap! It sucks!

[Bond-009]

This is being worked on: https://github.com/jellyfin/jellyfin/pull/9291

@AndrewDavis If it's so simple you can always contribute a fix.

@Skalmansbror you can put the posters and backdrops in the same folder as your media, that way they will never get deleted by Jellyfin.

[chris-NR]

I hit this issue as I am working on a solution to save power, the planet and my bank balance by moving my media archive to storage that is usually offline and wakes on demand. I have not been able to reproduce this problem (I do not lose the entire library).

This is what I have observed in my testing with version 10.8.9:

1. Add a directory from a read-only NFS share to a new library (contains ~ 600 movies and artwork all in a single directory i.e. no sub-directories)
2. Scan library: all OK, artwork present and can play movies
3. Unmount nfs share
4. Observations:
   1. Library entries persist and can be browsed
   2. Artwork is largely replaced by a blurred version of the original image, but not entirely, some remains. I suspect this is due to caching.
   3. Errors in log due to missing artwork
   4. Movies cannot be played, error is “Playback Error - This client isn't compatible with the media and the server isn't sending a compatible media format.”
5. From the dashboard -> “Scan All Libraries” (took 19 seconds)
6. Observations:
   1. Log contains errors related to missing media
   2. Library entries persist and can still be browsed
   3. All affected artwork is now replaced by a “clapperboard” icon
   4. Movies cannot be played (same error as above)
7. Re-mount nfs share
8. Observations:
   1. Library entries persist and can be browsed
   2. No artwork present (still clapperboard icons)
   3. Movies can be played
9. From the dashboard -> “Scan All Libraries” (took 98 minutes and 50 seconds)
10. Observations:
    1. Most clapperboard icons replaced by a single incorrect image (from the cache?) and gradually replaced by the correct images (a different bug?)
    2. CPU and Memory maxed out for the entire scan duration (due to re-processing image files?)
    3. All images restored at the end of the scan
    4. Movies can be played

For comparison plex (v1.31.1.6782) behaves differently when the NFS share is unmounted:

1. The library can still be browsed ok
2. Images are not replaced with blurred versions
3. Library entries are marked as “Unavailable” in the browser
4. Movies cannot be played, error is “Playback Error - Please check that the file exists and the necessary drive is mounted.”
5. Scanning the library whilst unmounted completes instantly and does not remove any images or replace them with an icon
6. A ‘Full Metdata Refresh’ can be performed on the offline library without loss of data
7. After remounting the share the library can be browsed and the entries are no longer marked as “Unavailable”
8. Movies can be played ok
9. Re-scanning the Library completes instantly with no image reprocessing

[Dryadwoods]

version 10.8.9

Well... I am also using version 10.8.9 And sadly I can say that this is not fixed.

This week I was busy with life and I only noticed 2 or 3 days later that I had problems with my NAS. Meanwhile, jellyfin during his automatic tasks did something ...... since was NOT able to reach the files anymore... therefor ALL my libraries were lost..... really everything.

I am already refreshing them and this is going to take me forever to finish.

[Fruckiwacki]

While I am also interested in a solution, it just baffles me that people speak about their precious and hard to curate libraries, yet seemingly have no backup solution in place.

If whatever catastrophic damage happens to your library, simply restore it from a backup.

Edit: library in this case refers to Jellyfin's database.

[Dryadwoods]

While I am also interested in a solution, it just baffles me that people speak about their precious and hard to curate libraries, yet seemingly have no backup solution in place.

If whatever catastrophic damage happens to your library, simply restore it from a backup.

When I say that the "library" was lost, I meant the Jellyfin database data... not the files it self (movies, subtitles). Of course, that also some of the "mappings" or CUSTOM metadatas (might be in a file, or in the DB) might also be affected. And when you refer backup... well... what type are you referring too that applies to this case?

[luispabon]

I back up everything, more so irreplaceable data like that.

[Fruckiwacki]

While I am also interested in a solution, it just baffles me that people speak about their precious and hard to curate libraries, yet seemingly have no backup solution in place. If whatever catastrophic damage happens to your library, simply restore it from a backup.

When I say that the "library" was lost, I meant the Jellyfin database data... not the files it self (movies, subtitles). Of course, that also some of the "mappings" or CUSTOM metadatas (might be in a file, or in the DB) might also be affected. And when you refer backup... well... what type are you referring too that applies to this case?

I refer to the data you mentioned - the Jellyfin config directory, which includes all metadata, configuration etc.

[Dryadwoods]

I completely agree with the use of backups, no discussion on that.

Although, I might highlight that fact that not everyone knows how to perform backups (of different sources or types) and that is really not the point of this open ticket, the issue is that Jellyfin is losing it, and that should be the main discussion.

I would like to suggest that one easy way to fix this problem would be to have a BIT field associated with one library that says "Lock library"... in the sense that no matter what happens, simple do not change anything about it. Keep the database entries, and everything else the way it is, do not delete, neither do not add or update it.

[cvium]

Feel free to take a look through the code base. You might feel different.

[chris-NR]

Have we determined the root cause of the problem? In my testing I did not lose the entire database 'only' the artwork. Is there some other (automatic?) process, apart from "Scan All Libraries", that could be causing this?

[chris-NR]

I respectfully disagree that

The only proper way to fix this problem would be to do what I said https://github.com/jellyfin/jellyfin/issues/1714#issuecomment-1442855762

as this means users will not be able to browse their library for "offline" media i.e. removeable drives would need to be attached or nfs shares mounted to determine what media is available.

In order to save power, money & the planet I wish to keep my media archive offline and only start the archive server as necessary, ideally automatically. I have had some success using systemd automount and an nfs share but this issue is preventing me from making further progress.

My 'ideal' solution would be for it to work in a similar way to Plex but not touch the media files until they need to be played or scanned (to prevent the share from automounting until absolutely necessary)