Efficient Background Processing with darktable-cli

[gnwekge78707]

Efficient Background Processing with darktable-cli

Summary

Is it possible to run darktable-cli as a background job to avoid loading the program and pixelpipe with each call?

Description

My application requires frequent and fast image processing using darktable-cli, leveraging the GPU. Currently, each call to darktable-cli involves:

• Loading the program: ~0.5s
• Creating the pixelpipe: ~1s
• Processing the image: ~0.1s

( My command is darktable-cli setubal.orf setubal.orf.xmp test.jpg --core -d perf -d opencl )

This results in a significant overhead per call, primarily due to the repeated loading and pixelpipe creation times.

Request

Can we reduce the total processing time to approximately 0.1s by pre-loading the program and pixelpipe? Is there an existing feature or a recommended approach to achieve this efficiency?

Additional Context

The ability to maintain darktable-cli as a background job or service would greatly enhance performance for applications requiring rapid image processing.

[ralfbrown]

Not at the moment, though you can have darktable-cli process an entire directory of images in one call. But if you look more closely, the ~1s "creating the pixelpipe" is mostly spent reading the image into memory and decoding it. That won't go away no matter what. A quick test just now with an image in ramdisk has 0.299s elapsed time for loading and 0.065s for actual pixelpipe setup, which involves going through all of the modules and computing ROIs and possibly distortion -- it's basically the planning phase of the image processing.

One thing that may speed up the loading is to point at an empty config, especially if you have lots of styles and geolocations defined in your default config. On my system, doing so shaves 0.17s off the startup time.

[gnwekge78707]

Thanks for the advice. After running some tests, I have a couple of new questions:

1. Processing a Folder of Images with Different Parameters: When using darktable-cli to process a folder of images, is it possible to specify different parameters for each image? If so, how can this be achieved?
2. Loading IOP Modules and Presets: Before creating the pixelpipe, a significant portion of time (~0.62 seconds) is spent on loading IOP modules and presets (0.623257 [init] startup took 0.623257 seconds ). Are there any ways to prevent loading unused presets and IOP modules to speed up the startup?

Thank you for your help and guidance. I'll attach the debug log here. debug_all.txt

[ralfbrown]

If you don't specify an XMP file on the commandline, it should use any existing sidecar file. I'd have to go through the code to be sure, but I think only the first one will be used if you have multiple edits for an image.

That startup is done once per run, not once per image. Pointing at a config directory containing only a default data.db without your presets would reduce that, but the savings will be negligible overall if you're processing even ten images per run.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[ralfbrown]

No further response from OP for two months, so the issue is presumably resolved.