[Bug]: Restoring from AudioBookShelf backup results in broken Author images and "Internal Server Error"

[ZLoth]

Describe the issue

Because of a botched upgrade of the TrueCharts docker container, I ended up killing my original AudioBookShelf instance and creating a new instance. I then restored the backup from the latest nightly AudioBookShelf backup from Settings → Backup. While my book images and data was restored properly, most of the Author images were broken. In checking the directory /metadata/authors, I did not see the picture uploaded. I ended up deleting the broken image and then doing a quick match to restore the image.

Example screenshot:

Here is a recording of the issue: https://github.com/advplyr/audiobookshelf/assets/6700159/68e47f52-48f9-4398-817a-ac95a7a1b273

The UUID attached to the author is adeeccd0-cb1a-4647-9516-f8a63fdb24be on my instance, but when I check the /metadata/authors directory, the image does not exist:

Steps to reproduce the issue

1. Make a backup from Settings → Backup
2. Configure a new Docker instance and establish book mapping.
3. Restore the backup into a new instance.

WHAT SHOULD HAPPEN: Author metadata and images are restored as well as book images and metadata. WHAT REALLY HAPPENS: Author images are not restored.

WORKAROUND: Individually remove the image, then do a quick match again to restore data.

Audiobookshelf version

2.9.0

How are you running audiobookshelf?

Docker

[Mega-Cookie]

Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup.

I'm also running a Docker container

MY WORKARROUND:

1. mkdir authors/ in metadata/
2. extract metadata-authors/ from the backup file (with WinRAR in my case)
3. copy contents of metadata-authors/ over to metadata/authors/

[ZLoth]

Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup.

Issue is still occurring in 2.11 . But, I was able to restore the authors per your workaround.

[nichwall]

Are you able to get any logs of the backup restore? I'm not able to reproduce the issue of author images not being restored using Docker version 2.11.0

My debug steps:

• Made backup
• Delete all images in /metadata/authors
• Ensured author images were missing after purging cache in main server settings (checked /metadata/cache to make sure images were missing and that they did not show up in the UI)
• Used the "Restore" button next to a backup
• Verified no errors in logs, author images show up in /metadata/authors, and author images appear in web interface

I also tried:

• Made backup
• Changed an author image
• Used the "Restore" button next to a backup
• Verified no errors in logs, author images show up in /metadata/authors, and author images appear in web interface. The author image I changed after the backup was made did correctly restore to the original version

[ZLoth]

@nichwall : My apologies for the delay as I was not able to attempt a reproduction of the issue until this weekend. My configuration is that I'm running TrueNAS Scale running Dragonfish-24.04.2, and I'm using a Docker container. My testing is that I set up a second test instance of ABS, with the only difference is that my production Config Storage and Metadata Storage, configured to use Host Path, pointed to /mnt/pool/app-config/abs-test instead of my production directory of /mnt/pool/app-config/audiobookshelf , and the port used was different from production.

STEPS TO REPLICATE:

1. Created a second ABS instance with the name of audiobookshelf-test.
2. Created a login account.
3. Went into Settings → Backup.
4. Restored from my backup. During the import, the socket got disconnected and reconnected.
5. Refreshed my screen and re-logged in.
6. Went into Authors.

WHAT SHOULD HAPPEN: Pictures should have been restored. WHAT REALLY HAPPENS: Multiple errors thrown in the logs, and no pictures in the authors directory.

Here is a recording of the issue. My apologies for the big yellow box, as I don't want my users to be displayed. https://github.com/user-attachments/assets/4a7dc434-2f7a-4a5c-a457-bf7b9c7dcc4e

Here are the log files: Logs.zip

[nichwall]

Thanks for providing more information. I just tried following your steps (using the same hardware, but an entirely new directory path/container). I am not seeing the socket disconnect and reconnect or any errors in my logs. Granted, the largest backup file I have on hand for this is only about 200 MB.

I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try? Based on the log file showing that /metadata/authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg cannot be found while restoring the backup, I'm wondering if it could be:

• permissions (does not make much sense since other things are restored)
• file is missing (does not make sense if you can extract the backup and restore images manually)
• backup file is too large or something is not copied correctly (also does not make sense because StreamZip is used)

Can you answer/do the following:

1. How many author images are in the backup?
2. How big/what percentage of the backup size is author images?
3. How big/what percentage of the backup size is from metadata-items?
4. Verify that metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg exists in the backup.
5. Look at /metadata/authors after the extraction and see if any files exist in the file system?

[ZLoth]

I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try?

Negative. This is my daily backup of my production instance, and has over 4,400 books (thanks Plus catalog)

1. How many author images are in the backup?

2. How big/what percentage of the backup size is author images?
3. How big/what percentage of the backup size is from metadata-items?

4. Verify that metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg exists in the backup.

5. Look at /metadata/authors after the extraction and see if any files exist in the file system?

As you can see from the following five-minute recording where I went from adding the app in TrueNAS scale as a second instance, setting up the paths, restoring the backup, etc , both /metadata/items and /metadata/authors do not exist until I attempt restore from backup. After restoration from backup, items exist, but authors does not exist. It's only when I update the photo does the authors directory exist.

https://github.com/user-attachments/assets/410d5c42-9287-41af-baa1-09b13c03b5d5

Log files: Logs.zip

[nichwall]

Why is absdatabase.sqlite in your metadata folder (at 2:27)? Having config and metadata at the same mount point may be causing some of the problems.

[ZLoth]

Here is a recording where I created the subdirectories config and metadata as subdirectories under /mnt/pool/app-config/abs-test:

https://github.com/user-attachments/assets/3728bc04-e08a-471d-9da8-1fc1d699f962

Here is a recording where I created directories /mnt/pool/app-config/abs-config and /mnt/pool/app-config/abs-metadata:

https://github.com/user-attachments/assets/de0d6df3-ca9f-4cb7-9ac2-2febd59342e3

Still same issue.

[ZLoth]

And, just for fun, I'm using the IX defaults for the metadata and config:

https://github.com/user-attachments/assets/7fc44168-7ace-4862-823e-8dc4b00a82a5

[ZLoth]

What is the order of the restoration of the backup? Is it items, then the sqlite DB, then authors?

[nichwall]

SQLite, items, then authors https://github.com/advplyr/audiobookshelf/blob/master/server%2Fmanagers%2FBackupManager.js#L171-L233

[nichwall]

If you're willing to DM a download link for the backup file, I can try and recreate the issue specifically from your backup. (Though, maybe change the admin user password to "password" or something so it's easier for me to actually log in and verify things)

[ZLoth]

@nichwall : As the file is 1.5 GB in size, a WeTransfer link has been sent to you directly. The link will expire on July 24th.

[nichwall]

Thanks, I was able to reproduce the server crash during restoring the backup with your backup file and was able to log in using the new password you created. I'll see if I can find more information.

[nichwall]

Okay, pretty sure I found the problem.

In your /metadata/authors folder, there are only files (no directories). Backups from my main server (running on Synology) have a hidden folder @eaDir in /metadata/authors. It looks like folders are only created during extraction if there is a folder in the path, so when metadata-authors/ is extracted from the backup, you don't have a folder under metadata-authors/, so the authors directory is not created in the file system. This does not matter for the database because it is extracting a single file to an existing folder, and the metadata-items/ part of the backup is a bunch of folders so /metadata/items gets created.

I added a dummy folder (tried different names to make sure alphabetical sorting didn't matter) to your /metadata/authors folder, made a backup, and then was able to successfully restore the backup on a new server. I also verified that deleting the hidden folder from my backup caused restoring from backup to fail.

[ZLoth]

As a test, I manually created the authors and items folders, chown-ed them to be the apps owner, then performed the restore. This time, no crash, and the authors were restored. So, per your pull request, ensuring that authors and Items folders are created prior to restore is a good fix.

https://github.com/user-attachments/assets/54cb0b9b-980f-4dc8-9703-6d5311e5a990

And all of the authors were restored.

https://github.com/user-attachments/assets/0b2b7995-103e-4f19-b1c9-a8738fff7d3f

Thank you for tracking this issue down. I look forward to the fix, although hopefully, I won't have to do a restore for quite a while.

[ZLoth]

Thank you!

[github-actions[bot]]

Fixed in v2.12.0.