[FEATURE REQUEST] increase speed of archive reading

[afterdelight]

increase speed when reading through archive file list. it takes too long and too slow.

[mikf]

I personally do not know how to further increase read performance for an archive implemented as an SQLite database, sorry.

Increasing write speeds can be done by setting several different options with PRAGMA statements, but that can potentially lead to slower reads:

PRAGMA journal_mode = WAL
PRAGMA synchronous = NORMAL 

Are you sure that it is actually database reads that are slow, and not anything else like unnecessary network requests or --sleep timeouts?

[AlttiRi]

PRAGMA journal_mode=WAL;

can increase the speed of writing. Significantly. However you will see the difference only if you run multiple instances of gallery-dl at one time.

I think that your suggestion just is wrong about the reading. The only things that can be slow are related to a network.

[AlttiRi]

how to further increase read performance

Using of a separated table for each service will reduce the DB size, also technically it will slightly increase the speed of reading, since it's O(log(N)), not O(1) .

[Hrxn]

how to further increase read performance

Using of a separated table for each service will reduce the DB size, also technically it will slightly increase the speed of reading, since it's O(log(N)), not O(1) .

Just do what I do, use different archive files for different services/sites. At least the one's with heavy usage.. Already supported, just set-up the galley-dl config accordingly..

[afterdelight]

okay, thank you very muach for the infromations