ar-io-node
ar-io-node copied to clipboard
ar-io
BlockImporter Error: worker died, SQLITE_CORRUPT
Hi. after a reboot my gateway seems to be in a broken state. i see this messages for block 1246846 in an endless loop.
can i somehow debug or should i delete all docker volumes?
core_1 | error: Error in StandaloneSqlite worker: database disk image is malformed {"code":"SQLITE_CORRUPT","stack":"SqliteError: database disk image is malformed\n at StandaloneSqliteDatabaseWorker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:419:60)\n at StandaloneSqliteDatabaseWorker.sqliteTransaction [as insertBlockAndTxsFn] (/app/node_modules/better-sqlite3/lib/methods/transaction.js:65:24)\n at StandaloneSqliteDatabaseWorker.saveBlockAndTxs (file:///app/dist/database/standalone-sqlite.js:623:14)\n at MessagePort.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1935:28)\n at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)","timestamp":"2024-09-07T08:15:43.935Z"}
core_1 | error: Error importing block at height 1246846 Worker error {"class":"BlockImporter","stack":"Error: Worker error\n at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1619:32)\n at Worker.emit (node:events:519:28)\n at MessagePort.<anonymous> (node:internal/worker:263:53)\n at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)","timestamp":"2024-09-07T08:15:43.935Z"}
core_1 | error: Uncaught exception: Worker error {"stack":"Error: Worker error\n at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1619:32)\n at Worker.emit (node:events:519:28)\n at MessagePort.<anonymous> (node:internal/worker:263:53)\n at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)","timestamp":"2024-09-07T08:15:43.936Z"}
core_1 | info: Importing block... {"class":"BlockImporter","height":1246846,"timestamp":"2024-09-07T08:15:43.986Z"}
core_1 | error: Too many errors in StandaloneSqlite worker, exiting. {"timestamp":"2024-09-07T08:15:44.071Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-09-07T08:15:44.074Z"}
core_1 | error: Error importing block at height 1246846 worker died {"class":"BlockImporter","stack":"Error: worker died\n at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1634:41)\n at Worker.emit (node:events:519:28)\n at [kOnExit] (node:internal/worker:315:10)\n at Worker.<computed>.onexit (node:internal/worker:229:20)","timestamp":"2024-09-07T08:15:44.074Z"}
core_1 | error: Uncaught exception: worker died {"stack":"Error: worker died\n at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1634:41)\n at Worker.emit (node:events:519:28)\n at [kOnExit] (node:internal/worker:315:10)\n at Worker.<computed>.onexit (node:internal/worker:229:20)","timestamp":"2024-09-07T08:15:44.074Z"}
one way to solve:
rm data/sqlite/*
info: Importing block... {"class":"BlockImporter","height":214,"timestamp":"2024-09-07T22:41:05.990Z"}
lets see how long it takes..
After some months without problems, my node got again randomly corrupted ..
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.463Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.472Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.476Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.502Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.505Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.517Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.560Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.564Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.586Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.616Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.646Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.757Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.353Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.355Z"}
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.356Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.383Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.385Z"}
core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.393Z"}
core_1 | error: Error retrying missing transactions: {"class":"TransactionRepairWorker","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.394Z"}
core_1 | error: Uncaught exception: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "#<Object>". {"code":"ERR_UNHANDLED_REJECTION","name":"UnhandledPromiseRejection","stack":"UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".\n at throwUnhandledRejectionsMode (node:internal/process/promises:389:7)\n at processPromiseRejections (node:internal/process/promises:470:17)\n at process.processTicksAndRejections (node:internal/process/task_queues:96:32)","timestamp":"2024-11-27T07:06:55.394Z"}
this seems to help:
docker-compose down -v
docker system prune
rm -rf data/sqlite
docker-compose up -d
Hi @ThomasBlock - before you remove the sqlite data, can you run a df -h on your server to see how much of the available file system is being used by your ar-ionode. It seems likely you are running of storage space causing SQLITE to fail writing new data.
After some months without problems, my node got again randomly corrupted ..
core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.463Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.472Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.476Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.502Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.505Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.517Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.560Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.564Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.586Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.616Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.646Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.757Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.353Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.355Z"} core_1 | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.356Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.383Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.385Z"} core_1 | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.393Z"} core_1 | error: Error retrying missing transactions: {"class":"TransactionRepairWorker","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.394Z"} core_1 | error: Uncaught exception: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "#<Object>". {"code":"ERR_UNHANDLED_REJECTION","name":"UnhandledPromiseRejection","stack":"UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".\n at throwUnhandledRejectionsMode (node:internal/process/promises:389:7)\n at processPromiseRejections (node:internal/process/promises:470:17)\n at process.processTicksAndRejections (node:internal/process/task_queues:96:32)","timestamp":"2024-11-27T07:06:55.394Z"}this seems to help:
docker-compose down -v docker system prune rm -rf data/sqlite docker-compose up -d
Try to see if all the db migrations ran correctly - that was my issue seeing these errors
@jim-toth - can you share the command you ran to run the migration via docker?
@jim-toth - can you share the command you ran to run the migration via docker?
ye
docker-compose exec core /nodejs/bin/node dist/migrate.js up
you may need to prepend sudo or use docker compose depending on your docker setup
tho, all i used this for was to isolate the migration error i was getting. after fixing the issue by manually dropping the related table in the migration that didn't finish i just started the stack and let core migrate. the advantage of letting core migrate is that when it's done the node will just be up - some of these migrations take a long time ;)
Closing. We still have some work to do to improve our migration experience, but the original was addressed.
