audiobookshelf

audiobookshelf copied to clipboard

Describe the issue

While trying to figure out how to use libraries I created a new library that shared a disc folder with my older, original library - specifically, both included my Audible books, while my main included books I'd bought from audiobooks.com and so on.

When I started the scan on the new library, I began seeing duplicates showing up in my original library. Not every book was duplicated, but the key seems to be the presence of BOTH an odm file, AND an abs file with different contents (for example because I ran a match).

It's possible that this kind of folder sharing cannot be done using metadata-in-folders, and that's a fine workaround. But probably the system should catch people doing that and at least warn.

Steps to reproduce the issue

1. Turn on metadata-in-book-folders (abs files). This is necessary to produce the duplicates.
2. Add two folders with a reasonable number of books in each. In my case I have odm files with every book, but I'm not certain that's essential -- but it does make it easier to control the metadata, which in this case must be different from what a Match will produce.
3. Run a match on a few of the titles in the one folder you intend to share, producing different text you can easily distinguish.
4. Create a new library containing only the folder you planned to share. Scan it.
5. Confirm that the books with new metadata now appear twice in your original library.
6. Revert to your last backup and smile because this superb software performs backups.

Audiobookshelf version

2.1,2

How are you running audiobookshelf?

Docker

I'm glad the backups are working!

Were the 2 folder paths the exact same in both libraries? Or was the folder path a subfolder of a folder in the other library?

When you say "odm file" do you mean OPF file?

1. I know, I LOVE that backups passed this no-stress test! I'm so darn happy with this app, thank you!

2. Yes, they were the exact same Linux pathname, no links or anything. Not even a subfolder -- EXACTLY the same path.

3. Doh! Yes, sorry. OPF. I don't think it's NECESSARY, but I did in fact always use them. I do think abs-in-folder is necessary, and I strongly suspect they're what messed me up (because OF COURSE I shouldn't expect metadata stored in the folders to be stable).

This won't happen again to me now that I know, but gut feeling is that we can probably at least warn, and maybe possibly prevent it completely.

-Wm

On Fri, Aug 12, 2022 at 3:08 PM advplyr @.***> wrote:

I'm glad the backups are working!

Were the 2 folder paths the exact same in both libraries? Or was the folder path a subfolder of a folder in the other library?

When you say "odm file" do you mean OPF file?

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/906#issuecomment-1213554145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6K354ZYIY4ZU4YQQKLVY3DOHANCNFSM56L45XUQ . You are receiving this because you authored the thread.Message ID: @.***>

I tested this and wasn't able to reproduce it. I don't think we should warn the user but rather fix the bug if there is one because this should work fine. Are you able to reproduce this creating new libraries and new folders with just a few books?

I'll set this up later today.

On Fri, Aug 12, 2022 at 3:59 PM advplyr @.***> wrote:

I tested this and wasn't able to reproduce it. I don't think we should warn the user but rather fix the bug if there is one because this should work fine. Are you able to reproduce this creating new libraries and new folders with just a few books?

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/906#issuecomment-1213576978, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6OVBXLVAAUFSEPCGU3VY3JNDANCNFSM56L45XUQ . You are receiving this because you authored the thread.Message ID: @.***>

I agree I didn't get the same results (I got worse ones). No duplicates, but ...

Following my instructions: I copied a couple of books into a new folder. I edited the OPFs so they'd stand out, made a library "test1" that used it, and scanned the library. No problem. I made the new library "test2" pointing to the same folder. Here I expected to see dupes in "test1", but there weren't any. I did edit some titles and author names, but I didn't get any dupes.

So I deleted both libraries (from the settings screen), and was about to post how I'd tried but failed.

But for some reason I went to my main library and found an "Issue" -- one of the two books thinks it's missing every 4th file. It's not -- nothing changed, all of the files are there and all are right there in the list of files. Nothing at all changed, nothing is wrong EXCEPT that ABS somehow is convinced something used to be there (but it never was).

This is very strange.

I'm not sure those things are related. The goal needs to be to reproduce the bug consistently with as few steps as possible.

I may have asked you this already or a few times, are you using a network file system? What OS are you running docker on and what OS are your files stored? Does the watcher work for you?

Oh, I should have put those into the initial report!

1. Ubuntu Linux 22.04 LTS.
2. All work done on a single local 8TB spinning-rust drive, bootup partition on SSD (I forget how big, but we shouldn't be touching it).
3. The watcher works great.

Side comment: is there any way to export my book info (especially after a match) aside from keeping ABS files in the book folders? I'm 99% sure this is causing this problem (I kind of think it's a race condition).

It's probably notable that the shared folder I originally exported had some 5000 books (I have an addiction to Audible), so the fact that only SOME of my books were duplicated might have only proved that it's going to be hard to recreate this.

-Wm

On Sun, Aug 14, 2022 at 7:27 AM advplyr @.***> wrote:

I'm not sure those things are related. The goal needs to be to reproduce the bug consistently with as few steps as possible.

I may have asked you this already or a few times, are you using a network file system? What OS are you running docker on and what OS are your files stored? Does the watcher work for you?

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/906#issuecomment-1214389475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6J2PKRQFA3SVRLPSUTVZD66NANCNFSM56L45XUQ . You are receiving this because you authored the thread.Message ID: @.***>

I just got a tiny bit of evidence that the watcher is not consistently working for me -- specifically I copied in a new Audible book, and it didn't get automatically added until I manually scanned the next day.

Then, when I manually scanned, the new book got added, an old book got a new duplicate made with half of the metadata of the new book and half of its own metadata and all the files of the old book, and the old book's old entry got marked as missing even though all of its files are present.

Side comment: is there any way to export my book info (especially after a match) aside from keeping ABS files in the book folders?

If you don't keep the abs files inside your library item folders then they are stored in /metadata/items

On the original issue:

One possible reason this could happen is if the watcher triggered a scan on a new library item then while it is being scanned in the watcher is triggered again before the first finishes. That could cause a duplicate but would be very uncommon. The watcher waits 4 seconds after detecting a change before it triggers a scan, so the first scan would need to be taking an unusually long time for an overlap to occur.

That being said it IS possible and may be what you encountered so I updated the scanner to queue up file updates if a watcher scan is currently running.

That makes a lot of sense ... a few books out of about 5000 seems "rare" :) . I'll leave my settings the same for now, and watch for when this code change gets pushed so I can hammer things around...

I have seen a RARE few books changed since I posted my last update, so although we'll never be certain (sorry, just can't make this 100%) I think I can make it sure enough.

-Wm

On Wed, Aug 31, 2022 at 3:44 PM advplyr @.***> wrote:

Side comment: is there any way to export my book info (especially after a match) aside from keeping ABS files in the book folders?

If you don't keep the abs files inside your library item folders then they are stored in /metadata/items

On the original issue:

One possible reason this could happen is if the watcher triggered a scan on a new library item then while it is being scanned in the watcher is triggered again before the first finishes. That could cause a duplicate but would be very uncommon. The watcher waits 4 seconds after detecting a change before it triggers a scan, so the first scan would need to be taking an unusually long time for an overlap to occur.

That being said it IS possible and may be what you encountered so I updated the scanner to queue up file updates if a watcher scan is currently running.

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/906#issuecomment-1233502207, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6MCU3BONUNLWMS5HSTV37N3XANCNFSM56L45XUQ . You are receiving this because you authored the thread.Message ID: @.***>

Updated in v2.1.5

Adding lock to prevent 2 watcher scans running at the same time.

OK, I backed up and recreated the situation with 2.1.4 (pre-fix) and got some of this error; I reverted, upgraded, backed up, and confirmed we DO NOT see the error.

I'm optimistic about this fix.

On Mon, Sep 5, 2022 at 2:29 PM advplyr @.***> wrote:

Closed #906 https://github.com/advplyr/audiobookshelf/issues/906 as completed.

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/906#event-7325667351, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6LFAG6LTX4QJBY3CATV4ZQ3NANCNFSM56L45XUQ . You are receiving this because you authored the thread.Message ID: @.***>