audiobookshelf [Bug]: ABS crashes with giant ebook libraries

What happened?

Server collapses temproary with big ebook libraries

What did you expect to happen?

ABS wotking as smooth as before adding high amounts of books.

Steps to reproduce the issue

Just import a high amount of ebooks to ABS.
Scan your ebook libraries to find the new ebooks.
Once it ends, open ABS, on Web or Andoid.

Audiobookshelf version

v2.12.2

How are you running audiobookshelf?

Docker

What OS is your Audiobookshelf server hosted from?

Linux

If the issue is being seen in the UI, what browsers are you seeing the problem on?

None

Logs

Runing it on Debug mode, that's the only i get:


AudioBookShelf-PRO  | [2024-08-09 09:45:03.137] INFO: [LibraryScanner] Library scan 6bb91882-9ff2-4af9-93da-e7820a915f2c completed in 0:02.0 | 0 Added | 0 Updated | 0 Missing
AudioBookShelf-PRO  | [2024-08-09 09:45:03.185] DEBUG: [ApiCacheManager] library.afterUpdate: Clearing cache (ApiCacheManager.js:20)
AudioBookShelf-PRO  | [2024-08-09 09:46:51.651] DEBUG: [SocketAuthority] User Offline User1 (SocketAuthority.js:131)
AudioBookShelf-PRO  | [2024-08-09 09:46:51.652] INFO: [SocketAuthority] Socket zTQXEt2L100DD9diAAAr disconnected from client "User1" after 190709ms (Reason: transport close)
AudioBookShelf-PRO  | [2024-08-09 09:48:06.208] DEBUG: [ApiCacheManager] count: 0 size: 0 (ApiCacheManager.js:47)
AudioBookShelf-PRO  | [2024-08-09 09:48:06.222] DEBUG: [ApiCacheManager] Cache miss: {"user":"User1","url":"/libraries"} (ApiCacheManager.js:58)
AudioBookShelf-PRO  | [2024-08-09 09:48:06.285] INFO: [SocketAuthority] Socket Connected V_n3vYJNxeU_3lpvAAAt
AudioBookShelf-PRO  | [2024-08-09 09:48:06.286] DEBUG: [ApiCacheManager] count: 1 size: 2603 (ApiCacheManager.js:47)
AudioBookShelf-PRO  | [2024-08-09 09:48:11.340] DEBUG: [SocketAuthority] User Online User1 (SocketAuthority.js:204)
AudioBookShelf-PRO  | [2024-08-09 09:48:11.915] DEBUG: Loaded filterdata in 5.62s (libraryFilters.js:537)
AudioBookShelf-PRO  | [2024-08-09 09:48:11.949] DEBUG: [ApiCacheManager] Cache miss: {"user":"User1","url":"/libraries/69c6c533-51d3-4d30-8b93-926ca636fd63?include=filterdata"} (ApiCacheManager.js:58)
AudioBookShelf-PRO  | [2024-08-09 09:48:12.773] DEBUG: [ApiCacheManager] count: 2 size: 1777698 (ApiCacheManager.js:47)
AudioBookShelf-PRO  | [2024-08-09 09:48:13.373] DEBUG: Loaded 1 of 1 items for "Continue Listening/Reading" in 0.59s (LibraryItem.js:626)
AudioBookShelf-PRO  | [2024-08-09 09:48:13.878] DEBUG: Loaded 0 of 0 items for "Continue Series" in 0.51s (LibraryItem.js:643)
AudioBookShelf-PRO  | [2024-08-09 09:48:14.336] DEBUG: Loaded 10 of 56997 items for "Recently Added" in 0.46s (LibraryItem.js:673)
AudioBookShelf-PRO  | [2024-08-09 09:48:14.411] DEBUG: Loaded 5 of 5153 series for "Recent Series" in 0.07s (LibraryItem.js:689)
AudioBookShelf-PRO  | [2024-08-09 09:48:15.767] DEBUG: Loaded 10 of 36047 items for "Discover" in 1.35s (LibraryItem.js:704)
AudioBookShelf-PRO  | [2024-08-09 09:48:16.199] DEBUG: Loaded 0 of 0 items for "Listen/Read Again" in 0.43s (LibraryItem.js:735)
AudioBookShelf-PRO  | [2024-08-09 09:52:19.592] DEBUG: Loaded 10 of 19259 authors for "Newest Authors" in 243.39s (LibraryItem.js:751)
AudioBookShelf-PRO  | [2024-08-09 09:52:19.593] DEBUG: Loaded 5 personalized shelves in 246.81s (LibraryItem.js:754)
AudioBookShelf-PRO  | [2024-08-09 09:52:19.595] DEBUG: [ApiCacheManager] Cache miss: {"user":"User1","url":"/libraries/69c6c533-51d3-4d30-8b93-926ca636fd63/personalized?minified=1&include=rssfeed,numEpisodesIncomplete"} (ApiCacheManager.js:58)
AudioBookShelf-PRO  | [2024-08-09 09:52:19.596] DEBUG: [ApiCacheManager] Caching with 1800000 ms TTL (ApiCacheManager.js:61)
AudioBookShelf-PRO  | [2024-08-09 09:52:19.600] DEBUG: [SocketAuthority] User Offline User1 (SocketAuthority.js:131)
AudioBookShelf-PRO  | [2024-08-09 09:52:19.604] INFO: [SocketAuthority] Socket V_n3vYJNxeU_3lpvAAAt disconnected from client "User1" after 253319ms (Reason: transport close)



### Additional Notes

I just made a big library with all family ebooks, like 57000. Seems that ABS is trying to process or making a list of the 57000 ebooks which maybe is making the sqlite database hang. Then i get those "Socket disconnected" errors and i get the orange icon about ABS disconected from the client.

There should be another way of processing them without collapsing the server. Jellyfin and Kavita uses SQLite databases also, not having that issue there.

Thanks for the effort :D

Tell me whatever you need!

Aug 09 '24 07:08 hardwareadictos

Update: Still happening on 2.12.3

Aug 10 '24 09:08 hardwareadictos

Related to https://github.com/advplyr/audiobookshelf/issues/2750, https://github.com/advplyr/audiobookshelf/issues/2522, and https://github.com/advplyr/audiobookshelf/issues/2526.

Aug 10 '24 12:08 nichwall

Is there an actual crash happening or is it getting hung up? Can you check /metadata/logs/crash_logs.txt to see if there are any crashes in there? I can see that the query for Newest Author is taking 243.39s which is a problem that needs to be fixed but I want to make sure you are not hitting other crashes related to ebooks that we've seen.

Aug 10 '24 13:08 advplyr

That's what i get, some sqlite crashes on constraints:

{"timestamp":"2024-07-05 22:15:38.692","source":"Server.js:165","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-06 05:26:33.891","source":"Server.js:165","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-06 06:36:18.936","source":"Server.js:165","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-06 09:52:04.258","source":"Server.js:165","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-10 10:49:37.321","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-10 14:15:09.785","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-10 14:35:06.162","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-10 17:38:01.692","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-10 21:16:31.716","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 09:53:53.523","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 10:24:06.802","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 10:37:44.011","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 11:07:26.215","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 11:17:04.468","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 11:26:32.424","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}
{"timestamp":"2024-07-12 11:53:59.869","source":"Server.js:171","message":"[Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {\n  <rejected> Error\n      at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)\n      at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50\n      at new Promise (<anonymous>)\n      at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)\n      at /node_modules/sequelize/lib/sequelize.js:315:28\n      at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n      at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)\n      at async /node_modules/sequelize/lib/model.js:2541:13\n      at async Promise.all (index 1)\n      at async /node_modules/sequelize/lib/model.js:2522:9 {\n    name: 'SequelizeUniqueConstraintError',\n    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],\n    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {\n      errno: 19,\n      code: 'SQLITE_CONSTRAINT',\n      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n    },\n    fields: [ 'bookId', 'seriesId' ],\n    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'\n  }\n}","levelName":"FATAL","level":5}

It's weird because now i forced it 2 or 3 times and it didn't get refected on crash_log, but again the server collapsed when i get into books library and the container restarted by itself.

Aug 10 '24 19:08 hardwareadictos

Is there anything i can do?

That's making ABS tottally unusable, as my Books library is the first one to show upon app starts, Container collapses inmediatelly. Could be nice to add an option on library section to disable a concrete library instead of only deleting it :)

Aug 23 '24 09:08 hardwareadictos

Is there anything i can do?

Can you try using the edge docker tag? That uses the latest commit from GitHub and I think the giant ebook library issue is the same as what I tried to fix in https://github.com/advplyr/audiobookshelf/pull/3312.

To clarify, your server is crashing at boot or when you try to scan the library?

Aug 23 '24 13:08 nichwall

Is there anything i can do?

Can you try using the edge docker tag? That uses the latest commit from GitHub and I think the giant ebook library issue is the same as what I tried to fix in #3312.

To clarify, your server is crashing at boot or when you try to scan the library?

Okey! I will fire an edge ABS container on my docker test machine. Will take at least 2 days to generate all metadata for that amount of books. Will report back when ready!

Container is crashing with any task related to Books library, container boots fine and works until you do something with that library.

Aug 23 '24 15:08 hardwareadictos

Well, books, podcasts and audiobooks metadata synced really fast on my test deployment. Now with 57000 books, books library isn't making the container to crash, so seems that this fix worked really well.

So go ahead, waiting patiently for the next release :D

EDIT: Half an hour later, container just crashed on Books library, as i see now, just when a library scan started.

That's what i saw on the log:

AudioBookShelf-TEST  | [2024-08-24 11:10:03.306] INFO: [SocketAuthority] Socket tzm53mcNEvX4ZKVRAAAB disconnected from client "User1" after 897431ms (Reason: ping timeout)
AudioBookShelf-TEST  | [2024-08-24 11:10:03.321] INFO: [SocketAuthority] Socket hmd64nrFPwb6a-3iAAAD disconnected from client "User1" after 864222ms (Reason: ping timeout)
AudioBookShelf-TEST  | [2024-08-24 11:10:03.440] INFO: [SocketAuthority] Socket Connected tAMxR9-AUOvfd_2DAAAR
AudioBookShelf-TEST  | [2024-08-24 11:10:09.145] INFO: [SocketAuthority] Socket Connected HJtAZOvMws7928jXAAAT

So not sure if the container crashed or simply the socket went down. There's nothing on crash_logs.txt

I'm going to follow if this thing comes up again during the weekend.

EDIT2: I had a crash rear a scan regarding with a Java OOM:

AudioBookShelf-TEST  | [2024-08-24 13:50:08.963] DEBUG: [LibraryScan] "Libros": 57000 item data found in folder "/libros" (LibraryScan.js:131)
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | <--- Last few GCs --->
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | [7:0x7f4e072736a0]  9477038 ms: Mark-Compact 949.5 (986.2) -> 943.3 (987.9) MB, 1326.83 / 0.00 ms  (average mu = 0.478, current mu = 0.068) allocation failure; scavenge might not succeed
AudioBookShelf-TEST  | [7:0x7f4e072736a0]  9478426 ms: Mark-Compact 951.3 (987.9) -> 945.2 (989.9) MB, 1314.44 / 0.00 ms  (average mu = 0.320, current mu = 0.053) allocation failure; scavenge might not succeed
AudioBookShelf-TEST  |
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | <--- JS stacktrace --->
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
AudioBookShelf-TEST  | ----- Native stack trace -----
AudioBookShelf-TEST  |
AudioBookShelf-TEST exited with code 0

Aug 24 '24 09:08 hardwareadictos

Thanks for helping test! Can you change your log level to Debug on the test server and then try running the scan again? That should help narrow down what is happening.

Aug 24 '24 14:08 nichwall

So the same happened again:

AudioBookShelf-TEST  | [2024-08-24 18:24:06.408] DEBUG: [LibraryScan] "Libros": 57000 item data found in folder "/libros" (LibraryScan.js:131)
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | <--- Last few GCs --->
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | [7:0x7f42438276a0]  2022770 ms: Mark-Compact (reduce) 960.6 (981.9) -> 959.7 (982.2) MB, 1156.52 / 0.00 ms  (average mu = 0.183, current mu = 0.003) allocation failure; scavenge might not succeed
AudioBookShelf-TEST  | [7:0x7f42438276a0]  2023953 ms: Mark-Compact (reduce) 960.9 (982.2) -> 959.9 (982.4) MB, 1179.83 / 0.00 ms  (average mu = 0.100, current mu = 0.003) allocation failure; scavenge might not succeed
AudioBookShelf-TEST  |
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | <--- JS stacktrace --->
AudioBookShelf-TEST  |
AudioBookShelf-TEST  | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
AudioBookShelf-TEST  | ----- Native stack trace -----
AudioBookShelf-TEST  |
AudioBookShelf-TEST exited with code 0

I am on debug mode but that's the only thing appearing related to the crash.

Aug 24 '24 16:08 hardwareadictos

You can try increasing the max memory size. To do this you pass an environment variable in your docker command.

The environment variables is: NODE_OPTIONS=--max-old-space-size=8192

Where 8192 is size in MB so you can lower that if you need to.

Aug 24 '24 18:08 advplyr

Sorry for my clumsiness, now i saw on library settings that i set up the autoscan to work every 15 minutes accidentally this morning when i created the library. Is ABS able to abort scans if theres already one on progress? Maybe those scan overlaps were causing those OOMs.

Anyways i added the env var to my compose file and im testing it, for the moment 0 crashes. Let me keep testing the container.

Aug 24 '24 20:08 hardwareadictos

Well, rear one and a half day of testing (no more OOMs once automatic scan each 15 mins was disabled), i didn't see any crash, i got one "Socket disconection", and the interface was not responding, but it happened only one time, researching i didn't find anything related to it on container logs and on crash_log.txt.

My conclusion is: If the Edge branch didn't fully fix the issue, it probably fixed it by 99%.

Another thing to take into account, general ABS is really slow performant on the UI side since i added those 57000 ebooks, it responds, but once you get into ebooks library, performance is slowish. What can be tuned for this? I understand that those are lot's of files to process inside ffmpeg for retrieving miniatures and for a SQLite database, but what can be done?

Thank you :)

Aug 26 '24 09:08 hardwareadictos

Just an update: I can confirm that without the Java Memory bar, container doesn't crash also. 😃

Aug 27 '24 19:08 hardwareadictos

Thanks for testing with a large library. We still have a ways to go in fully migrating to the new data models on sqlite (we previously used JSON files to store data). So, queries and indexes are far from optimized.

I could look at your specific database to see where the slow points are and try to improve them. We are working on setting up some scripts to seed large databases that we can use for testing. Right now I only have a large audiobook db and a large podcast db.

Aug 27 '24 19:08 advplyr

Glad to know! I can send you on private my DB if you want. Let's hope that helps other with big libraries also.

What about let users to use 3rd party DBs like MySQL/Postgres?I know is a pain to make those type of integrities, but for big libraries maybe it's the key. Other services like for example Kavita/Jellyfin have also issues when dealing with big DBs and SQLite.

Only an idea :D

Aug 28 '24 09:08 hardwareadictos

Yeah you can send it via Discord or email [email protected]. The reason I chose Sequelize ORM is so that users could eventually supply their own db. There is still a long way to go before that would be feasible. In order to make the migration in some reasonable timeline some of the queries were written raw for sqlite.

Aug 28 '24 13:08 advplyr

I also don't think the bottleneck we are hitting has to do with sqlite. Sqlite can handle much more data than that.

Aug 28 '24 13:08 advplyr

Yeah you can send it via Discord or email [email protected]. The reason I chose Sequelize ORM is so that users could eventually supply their own db. There is still a long way to go before that would be feasible. In order to make the migration in some reasonable timeline some of the queries were written raw for sqlite.

Sent you a friend request on DIscord!

Aug 28 '24 15:08 hardwareadictos

UPDATE: Rear a week of testing (excluding the OOMs which were solved), Edge tag with those fixes is way more stable than the last release for me. Right now i cannot switch my v2.12.3 Production image to edge because i don't know which countereffects it would produce, but at least edge solved the crashes rear a week of testing.

Aug 31 '24 10:08 hardwareadictos

UPDATE: Deployed on production v2.13.0 and for the moment problem seems solved, even ABS performance increased a lot! Will report back here if i see that the problem reproduces :)

Thanks a lot for the hard work. Most projects doesn't have all the great support we have on ABS by the devs. That's really appreciated.

Sep 01 '24 11:09 hardwareadictos

Update: Sadly, this afternoon, now with 2.13.0 deployed on "Production" container i got various "crashes" using the aplication and trying to open Books library.

Same sympthoms as always. My monitoring throwing service down errors (Uptime Kuma, receiving alerts because the URL doesnt respond with a 200 code), app unusable throwing Socket disconnection errors / ping errors and nothing on logs.

Container doesn't crash, it simply stops responding and recovers itself on 2/3 minutes.

Maybe Test container doesn't have those issues because there's only one user created and online.

Let me know what can i provide when you can.

Sep 01 '24 18:09 hardwareadictos

Can you update to 2.13.1? There was a problem with 2.13.0 which was fixed a few hours ago.

Sep 01 '24 18:09 nichwall

Tried with 2.13.1 and 2.13.2, problem persists.

Sep 02 '24 06:09 hardwareadictos

Updated to 2.13.3, same thing

Sep 03 '24 08:09 hardwareadictos

hey! Is anything more i can debug for myself?

The problem is (rear deoploying edge branch on test and latest stable on production):

ABS hangs (container doesnt stop or reboot but just freezes) when opening Books library, which contains about 57000 ebooks (before adding this library it worked fine, even i have an audiobook library with 10000 audiobooks), UptimeKuma reports the service down as it doesn't report http 200 code and on about 2/3 minutes it goes live again. Sometimes it doesn't hang, but it does on a 9/10 ratio more or less.

Noting appears on logs, even on container logs or crash_log, i'm on debug mode.

What can i do? I love this app but for now is unusable on my case, had to go back to AntennaPod again.

And again: Thanks a lot for the hard work :)

Sep 07 '24 12:09 hardwareadictos

Do you have periodic library scans or the watcher enabled? I would try disabling those.

Sep 07 '24 13:09 nichwall

That was the first thing I switched off a month ago when it was reported:

Screenshot_20240907-170918 Screenshot_20240907-170915~2

Sep 07 '24 15:09 hardwareadictos

Updated to 2.13.4: Issue still present.

Sep 10 '24 06:09 hardwareadictos

To give some more info:

AudioBookShelf-PRO  | [2024-09-10 11:30:32.841] INFO: Received ping
AudioBookShelf-PRO  | [2024-09-10 11:31:02.841] INFO: Received ping
AudioBookShelf-PRO  | [2024-09-10 11:31:10.606] INFO: [SocketAuthority] Socket Connected cUhmTAqHa-WgGnepAABS
AudioBookShelf-PRO  | [2024-09-10 11:31:10.659] DEBUG: [SocketAuthority] User Online User1 (SocketAuthority.js:209)
AudioBookShelf-PRO  | [2024-09-10 11:31:32.839] INFO: Received ping
AudioBookShelf-PRO  | [2024-09-10 11:31:50.501] DEBUG: [SocketAuthority] User Offline User1 (SocketAuthority.js:140)
AudioBookShelf-PRO  | [2024-09-10 11:31:50.503] INFO: [SocketAuthority] Socket cUhmTAqHa-WgGnepAABS disconnected from client "User1" after 39897ms (Reason: transport close)
AudioBookShelf-PRO  | [2024-09-10 11:31:51.357] INFO: [SocketAuthority] Socket Connected pXUYj2_m5UgTGAteAABU
AudioBookShelf-PRO  | [2024-09-10 11:31:51.358] DEBUG: [ApiCacheManager] count: 0 size: 0 (ApiCacheManager.js:47)
AudioBookShelf-PRO  | [2024-09-10 11:31:51.366] DEBUG: [ApiCacheManager] Cache miss: {"user":"User1","url":"/libraries"} (ApiCacheManager.js:58)
AudioBookShelf-PRO  | [2024-09-10 11:31:51.399] DEBUG: [SocketAuthority] User Online User1 (SocketAuthority.js:209)
AudioBookShelf-PRO  | [2024-09-10 11:31:51.477] DEBUG: [ApiCacheManager] count: 1 size: 2543 (ApiCacheManager.js:47)
AudioBookShelf-PRO  | [2024-09-10 11:31:56.340] DEBUG: Loaded filterdata in 4.86s (libraryFilters.js:539)
AudioBookShelf-PRO  | [2024-09-10 11:31:56.358] DEBUG: [ApiCacheManager] Cache miss: {"user":"User1","url":"/libraries/69c6c533-51d3-4d30-8b93-926ca636fd63?include=filterdata"} (ApiCacheManager.js:58)
AudioBookShelf-PRO  | [2024-09-10 11:31:57.404] DEBUG: [ApiCacheManager] count: 2 size: 1778049 (ApiCacheManager.js:47)
AudioBookShelf-PRO  | [2024-09-10 11:31:57.849] DEBUG: Loaded 1 of 1 items for "Continue Listening/Reading" in 0.44s (LibraryItem.js:657)
AudioBookShelf-PRO  | [2024-09-10 11:31:58.607] DEBUG: Loaded 0 of 0 items for "Continue Series" in 0.76s (LibraryItem.js:674)
AudioBookShelf-PRO  | [2024-09-10 11:31:58.989] DEBUG: Loaded 10 of 56214 items for "Recently Added" in 0.38s (LibraryItem.js:704)
AudioBookShelf-PRO  | [2024-09-10 11:31:59.061] DEBUG: Loaded 5 of 5046 series for "Recent Series" in 0.07s (LibraryItem.js:720)
AudioBookShelf-PRO  | [2024-09-10 11:32:00.674] DEBUG: Loaded 10 of 36048 items for "Discover" in 1.61s (LibraryItem.js:735)
AudioBookShelf-PRO  | [2024-09-10 11:32:01.045] DEBUG: Loaded 0 of 0 items for "Listen/Read Again" in 0.37s (LibraryItem.js:766)

That's the last thing i see on logs rear the hangs and disconnections. Then ABS is tottally inresponsibe for 2 mins and recovers itself:

imagen

Sep 10 '24 09:09 hardwareadictos

audiobookshelf audiobookshelf copied to clipboard

[Bug]: ABS crashes with giant ebook libraries

What happened?

What did you expect to happen?

Steps to reproduce the issue

Audiobookshelf version

How are you running audiobookshelf?

What OS is your Audiobookshelf server hosted from?

If the issue is being seen in the UI, what browsers are you seeing the problem on?

Logs

audiobookshelf
audiobookshelf copied to clipboard