`PDOException(code: HY000): General error: 10 disk I/O error` - when indexing models

[itslit-tech]

trafficstars

Hi there, I am getting the following error when indexing 2 of my models. The command is able to index one of the models but the second one does not show synchronized status when using the php artisan scout:status command. Any help will be greatly appreciated. Running the command multiple times does fix the issue but is not ideal.

Stack trace

SQLSTATE[HY000]: General error: 10 disk I/O error {"exception":"[object] (PDOException(code: HY000): SQLSTATE[HY000]: General error: 10 disk I/O error at /var/app/current/vendor/teamtnt/tntsearch/src/Indexer/TNTIndexer.php:231)

[stacktrace]
#0 /var/app/current/vendor/teamtnt/tntsearch/src/Indexer/TNTIndexer.php(231): PDO->commit()
#1 /var/app/current/vendor/teamtnt/laravel-scout-tntsearch-driver/src/Engines/TNTSearchEngine.php(63): TeamTNT\\TNTSearch\\Indexer\\TNTIndexer->indexEndTransaction()

#2 /var/app/current/vendor/laravel/scout/src/Jobs/MakeSearchable.php(42): TeamTNT\\Scout\\Engines\\TNTSearchEngine->update(Object(Illuminate\\Database\\Eloquent\\Collection))

#3 [internal function]: Laravel\\Scout\\Jobs\\MakeSearchable->handle()

#4 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)

#5 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/Util.php(36): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()

#6 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))

#7 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))

#8 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)

#9 /var/app/current/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\\Container\\Container->call(Array)

#10 /var/app/current/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(Laravel\\Scout\\Jobs\\MakeSearchable))

#11 /var/app/current/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Laravel\\Scout\\Jobs\\MakeSearchable))

#12 /var/app/current/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))

#13 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(83): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(Laravel\\Scout\\Jobs\\MakeSearchable), false)

#14 /var/app/current/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}(Object(Laravel\\Scout\\Jobs\\MakeSearchable))

#15 /var/app/current/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Laravel\\Scout\\Jobs\\MakeSearchable))

#16 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(85): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))

#17 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(59): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\\Queue\\Jobs\\SqsJob), Object(Laravel\\Scout\\Jobs\\MakeSearchable))

#18 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(88): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\SqsJob), Array)

#19 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(354): Illuminate\\Queue\\Jobs\\Job->fire()

#20 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(300): Illuminate\\Queue\\Worker->process('sqs', Object(Illuminate\\Queue\\Jobs\\SqsJob), Object(Illuminate\\Queue\\WorkerOptions))

#21 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(253): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\SqsJob), 'sqs', Object(Illuminate\\Queue\\WorkerOptions))

#22 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(112): 
Illuminate\\Queue\\Worker->runNextJob('sqs', 'staging-tashi-m...', Object(Illuminate\\Queue\\WorkerOptions))

#23 /var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(96): Illuminate\\Queue\\Console\\WorkCommand->runWorker('sqs', 'staging-tashi-m...')

#24 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()

#25 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)

#26 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/Util.php(36): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()

#27 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))

#28 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))

#29 /var/app/current/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#30 /var/app/current/vendor/laravel/framework/src/Illuminate/Console/Command.php(201): Illuminate\\Container\\Container->call(Array)
#31 /var/app/current/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#32 /var/app/current/vendor/laravel/framework/src/Illuminate/Console/Command.php(188): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#33 /var/app/current/vendor/symfony/console/Application.php(1012): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 /var/app/current/vendor/symfony/console/Application.php(272): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#35 /var/app/current/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#36 /var/app/current/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#37 /var/app/current/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(131): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#38 /var/app/current/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#39 {main}
"}

[tarreislam]

Make sure your app can write to the storage path, also make sure you have enough drive space

[itslit-tech]

@tarreislam Thanks for your response. I have checked. The app can write to the storage path as one of the models gets indexed and yes, there is sufficient disk space.

[SlivaNilow]

same issue when file more 20mb. index crash

[thewebartisan7]

I have test with large file (100k + records circa 60MB), and it works fine.

But I am able to reproduce this error if for example I try to read from index when is doing a build index.

I use table plus and if I have open the index file and try to reload it during build of index, then this error happen.

I am concerened about this issue and I see also others has similar, because of sqlite lock down database during write operation.

Even using queue, when job is executed, users that try to search could potentionally generate this error.

Would be good a mysql database option..

[SlivaNilow]

may be this can help. https://www.sqlite.org/wal.html WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently

i am not good enginier and dont know how it setup but from description it is good

UPD working with hardcode in vendor/teamtnt/tntsearch/src/Indexer/TNTIndexer.php on line 183 add $this->index->exec('PRAGMA journal_mode=WAL');

like this

public function createIndex($indexName)
    {
        $this->indexName = $indexName;
        if (file_exists($this->config['storage'].$indexName)) {
            unlink($this->config['storage'].$indexName);
        }

        $this->index = new PDO('sqlite:'.$this->config['storage'].$indexName);
        $this->index->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
        $this->index->exec('PRAGMA journal_mode=WAL'); // add this line
        $this->index->exec("CREATE TABLE IF NOT EXISTS wordlist (

if anyone know better solution plz help

[hunterwebsiteservices]

I can see the code by @gelid line has made it's way into core. I am having the reverse problem. Using Homestead 11.3.3 I have to remove this line to have an index successfully run.

$this->index->exec("PRAGMA journal_mode=wal;");

Any chance we could have this set via some kind of optional flag? Such as an additional variable to the $config array?

[SachinAgarwal1337]

I am facing same issue. And this is breaking my production code. And my application is heavily dependent on this. The solution provided by @feltoar is working for me as well. If I comment out that line

$this->index->exec("PRAGMA journal_mode=wal;");

It is working totally fine for me.

[SachinAgarwal1337]

This looks like issue with the base txtsearch library. I have opened a PR https://github.com/teamtnt/tntsearch/pull/234 for this. Let's hope it gets merged soon

[hunterwebsiteservices]

Thanks @SachinAgarwal1337

[chadidi]

this is still an issue: package version: "teamtnt/laravel-scout-tntsearch-driver": "^11.1",

[stokic]

did you set "wal" config value?

[chadidi]

@stokic nope, I didn't set it as it wasn't in the docs, I was able to make this work by using a subfolder storage_path('tntsearch') not sure why!! I have permissions and over 200GB of space!!

[Siddufed]

After changing storage path , you will get "Unable to open database error". For that we need to create that folder manually. After creating that , again same error is coming ( 10 disk I/O error). How did you make this work?

[dinorain]

Have somebody solved this?

[mugenyi]

I solved this by adding 'wal' = false, in the scout.php

'tntsearch' => [
    'storage'  => storage_path("search"), //place where the index files will be stored
    'fuzziness' => env('TNTSEARCH_FUZZINESS', false),
    'fuzzy' => [
        'prefix_length' => 2,
        'max_expansions' => 50,
        'distance' => 2
    ],
    'asYouType' => false,
    'searchBoolean' => env('TNTSEARCH_BOOLEAN', false),
    'wal' => false,
], 

[racVert]

I found that mugenyi's suggestion worked only after I cleared out any of the following generated files:

*.index *.index-shm *.index-wal