Slow background thumbnail creation/caching

[MWP]

Describe the bug

Not really a bug, but undesired normal behaviour.

It appears that background thumbnail creation is only single-threaded, or rate limited?

I have a library with about 20,000 new RAW images (no thumbs cached as yet). If i scroll down in the lightroom view, the thumbnails are created quite quickly - about 10 thumbs per second. If i let background thumbnail creation do its thing, its very slow, about 1 thumbnail per second (watching the output of -d cache). (im using embedded RAW JPGs for thumbnails, "use raw file" is set to "never")

Is there a way to speed up background creation?

Steps to reproduce

Clean out the thumb cache. Enable background thumbnail creation. Watch the result with -d cache.

Expected behavior

create thumbs like speedy gonzales!

Logfile | Screenshot | Screencast

This is some output from -d cache while it's doing background creation:

505.9039 [mipmap_cache] generate mip 6 for image 81660 from embedded jpeg
 506.0453 [mipmap_cache] generate mip 5 for image 81660 from embedded jpeg
   506.1816 [mipmap_cache] generate mip 4 for image 81660 from embedded jpeg
   506.3215 [mipmap_cache] generate mip 3 for image 81660 from embedded jpeg
   506.3329 [mipmap_cache] grab mip 2 for image 81660 from disk cache
   506.4571 [mipmap_cache] generate mip 1 for image 81660 from embedded jpeg
   506.7826 [mipmap_cache] generate mip 7 for image 81661 from embedded jpeg
   506.9172 [mipmap_cache] generate mip 6 for image 81661 from embedded jpeg
   507.0463 [mipmap_cache] generate mip 5 for image 81661 from embedded jpeg
   507.1721 [mipmap_cache] generate mip 4 for image 81661 from embedded jpeg
   507.2988 [mipmap_cache] generate mip 3 for image 81661 from embedded jpeg
   507.3100 [mipmap_cache] grab mip 2 for image 81661 from disk cache
   507.4276 [mipmap_cache] generate mip 1 for image 81661 from embedded jpeg
   507.7472 [mipmap_cache] generate mip 7 for image 81662 from embedded jpeg
   507.8860 [mipmap_cache] generate mip 6 for image 81662 from embedded jpeg
   508.0223 [mipmap_cache] generate mip 5 for image 81662 from embedded jpeg
   508.1540 [mipmap_cache] generate mip 4 for image 81662 from embedded jpeg
   508.2844 [mipmap_cache] generate mip 3 for image 81662 from embedded jpeg
   508.3890 [mipmap_cache] grab mip 2 for image 81662 from disk cache
   508.5108 [mipmap_cache] generate mip 1 for image 81662 from embedded jpeg
   508.8415 [mipmap_cache] generate mip 7 for image 81663 from embedded jpeg
   508.9783 [mipmap_cache] generate mip 6 for image 81663 from embedded jpeg
   509.1103 [mipmap_cache] generate mip 5 for image 81663 from embedded jpeg
   509.2488 [mipmap_cache] generate mip 4 for image 81663 from embedded jpeg
   509.3805 [mipmap_cache] generate mip 3 for image 81663 from embedded jpeg
   509.3917 [mipmap_cache] grab mip 2 for image 81663 from disk cache
   509.5146 [mipmap_cache] generate mip 1 for image 81663 from embedded jpeg
   509.8366 [mipmap_cache] generate mip 7 for image 81664 from embedded jpeg
   509.9741 [mipmap_cache] generate mip 6 for image 81664 from embedded jpeg
   510.1051 [mipmap_cache] generate mip 5 for image 81664 from embedded jpeg
   510.2325 [mipmap_cache] generate mip 4 for image 81664 from embedded jpeg
   510.3594 [mipmap_cache] generate mip 3 for image 81664 from embedded jpeg
   510.3698 [mipmap_cache] grab mip 2 for image 81664 from disk cache
   510.4890 [mipmap_cache] generate mip 1 for image 81664 from embedded jpeg
   510.8057 [mipmap_cache] generate mip 7 for image 81665 from embedded jpeg
   510.9402 [mipmap_cache] generate mip 6 for image 81665 from embedded jpeg
   511.0684 [mipmap_cache] generate mip 5 for image 81665 from embedded jpeg
   511.1943 [mipmap_cache] generate mip 4 for image 81665 from embedded jpeg
   511.3194 [mipmap_cache] generate mip 3 for image 81665 from embedded jpeg

Commit

No response

Where did you obtain darktable from?

self compiled

darktable version

4.6.1

What OS are you using?

Linux

What is the version of your OS?

Linux Mint 21.2

Describe your system?

Xeon E5 10 core, 64GB ram, Quadro RTX 4000 (nv drivers)

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[TurboGit]

@jenshannoschwalm :

If i let background thumbnail creation do its thing, its very slow, about 1 thumbnail per second (watching the output of -d cache).

That's the expected behavior, we don't want the background thread - activated when the user is not using dt GUI (so probably something else) - to slow down the computer for other tasks.

[MWP]

That's the expected behavior, we don't want the background thread - activated when the user is not using dt GUI (so probably something else) - to slow down the computer for other tasks.

Okay, but this is still chewing on those same thumbnails many hours later. Can multiple threads be given a lower priority? Or can there be a setting added to specify a number of threads?

I would just use "darktable-generate-cache", but it's just as slow as the background creation.

[jenshannoschwalm]

We use just one background thread for updating as that is using multiple threads via omp for performance, also in opencl we would have to care about.

[MWP]

I still don't understand why it can't be done.

DT can generate them quicker (as proved when scrolling through a gallery). If DT is idle (as is the requirement of background thumbnail creation), why can't it use a bunch of resources to do it?

[jenshannoschwalm]

It does! Workload depends on size of generated image ...

[wpferguson]

DT can generate them quicker (as proved when scrolling through a gallery

TLDR; because you're only generating 1 size and using the embedded jpg as the source

Explanation:

The mipmap cache has 9 sizes, 0 - 8. 0 is tiny and 8 is 1:1.

My current camera is a Canon R7. The raw is 32MP. The embedded jpg is only useful up to mipmap size 3. After that the raw has to be opened and processed to generate mipmap sizes 4 - 8.

I have a 4K screen. The cache size for lighttable is 3 and a full preview is 6.

If I do an import and just generate the sizes I need it looks like this:

cache generated for 924 images at scale 3 in 10 seconds - embedded jpg preview cache generated for 924 images at scale 6 in 300 seconds - opened and processed

darktable cache generation uses some tricks to speed up generating all the sizes by generating the largest preview, then down sampling to generate the rest.