oxipng
oxipng copied to clipboard
shssoichiro
Option to control parallel file processing
Processing multiple files in parallel can bring some nice speed improvements but can also cause problems:
- Higher memory usage (#683)
- Confusion with verbose output (#572)
- Non-deterministic order of files processed (#590)
We need a switch to control whether parallel processing should be enabled or not.
The question is whether this should on by default and require (e.g.) --no-parallel-files to disable it, or off by default and require --parallel-files to enable it.
Keen to hear thoughts on this.
I vote for parallel by default, but w/ gracefully scaled auto-fallback as requested when there aren't enough resources for fully-parallel processing.
Gracefully handling the situation is unfortunately a pretty difficult problem. Knowing how much memory you need up front is not really feasible if you're processing a big queue of files in parallel, but more than that, available memory can change in a millisecond as your system and other processes continue doing things.
@AlexTMjugador Do you have any thoughts here? I'm leaning towards off-by-default to be safe. It's just one flag to enable for those who need to.
What about parallel files but single filter strategies?
I've ended up using -f 9 -t 1, because the brute filter is generally always best but trying the other strategies in parallel massively increases memory usage.
-t 1 is because of #590, making it hard to read a large list of files. Something like #678 which displays a sorted list of the processed files after encoding could fix it, but I'm getting sidetracked.
@jonnyawsom3 I'm not quite sure what you're suggesting, would you be able to clarify? Single filter strategies are of course always possible by just selecting one filter as you show. With the above proposal, you would do -f 9 --parallel-files to do parallel files with a single filter. But by not doing parallel files, you could avoid the confusion with the output that you mention.
Side note: I'd recommend always including filter 0 (-f 0,9) as it can quite often be better.
[edit] If you're wanting to do multiple filters in single-threaded mode but still do files in parallel, you should use -t 1 in conjunction with xargs.
I would've assumed "brute force the best filter" would include 0 internally, but good to know. And yeah, that's what I meant. Doing the filters in parallel is very memory intensive on large files, so singethreading the filters avoids sudden memory spikes in folders of mixed resolutions (In my case, 4K and 8K images). Unfortunately I'm on Windows, so xargs isn't an option. Maybe -t for filter threads and -T for file threads or something similar? Sorry if I'm asking a lot.
@AlexTMjugador Do you have any thoughts here? I'm leaning towards off-by-default to be safe. It's just one flag to enable for those who need to.
One of Oxipng's key features over other PNG optimizers is its default use of parallel processing, so making that behavior opt-in would be a significant regression there in my opinion.
Instead, I suggest refining the default parallel count calculation to take into account a memory usage estimate: by checking available system memory and conservatively choosing a count that's unlikely to exceed it, we can mitigate what's probably the main downside of parallel processing, too high memory usage, without sacrificing too much performance out of the box. To estimate memory usage, we could analyze Oxipng's behavior on well-chosen inputs and develop a heuristic function based on those results (a very simple linear regression could work well).
If you want to keep it simple, just do the raw image size multiplied by filters and threads per image. Then try adding the next file until it passes available memory.
The usage is only marginally higher than the raw image buffer per compression trial, so it could be an easy way to estimate.
but more than that, available memory can change in a millisecond as your system and other processes continue doing things.
If you want to keep it simple, just do the raw image size multiplied by filters and threads per image. Then try adding the next file until it passes available memory.
Combining the above & in light of #561, I further propose that this is continually adjusted dynamically, w/ reasonably smart "next trial" heuristics (e.g., if next file/filter would exceed available resources, perhaps another in the list wouldn't & retry when available).
These are all good thoughts guys but I'm afraid I still have big concerns about feasibility, effectiveness, and complexity of any such thing. This is not something I intend to pursue myself, sorry.
Keep in mind that memory usage isn't the only concern about parallel file processing (see notes at the top).
That said, it does sound like there's reluctance to change the default behaviour here so maybe I'll go ahead and implement --no-parallel-files. It's a bit of a mouthful though, anyone have any other naming suggestions? Maybe --sequential-files, but it isn't really much better.
As a long-form option, it should retain some degree of clarity about the meaning. I realise we have a number of options already which don't (such as --nx, --nz etc) but this is bad practice and I'd prefer not to make the problem any worse 😅
@jonnyawsom3 The brute heuristic does include filter 0 in its selection strategy and may use it for some lines but it's unlikely to try using it for the entire image. Despite being called brute, it's not the same as trying all the others combined.
For parallel files on windows, you could perhaps try using forfiles. Though you might like to try out --no-parallel-files in this test build to see what impact it makes. More generally, you can lower memory usage by lowering the thread count without necessarily dropping it right down to 1.
@andrews05
It's a bit of a mouthful though...
Well the opposite to parallel is serial...
Is it essential to provide as a parameter? Personally I script using oxipng, so just pass it one file at a time. What situations is this not an option?
@ace-dent If you're just doing one file at a time then you don't need to use this parameter. I'm not sure what you mean about it not being an option?
I was thinking of --single-file or --single-queue, but could still be confusing.
@andrews05 A clarification, please: Is --sequential ≡ -t 1? Or does something like --sequential -t 8 work as intuited?
Depends what “intuited” means 😄 I had hoped the help was clear enough (let me know if it isn’t) but sequential only applies to files. Oxipng will continue to use the specified threads to run multiple filters on the same file in parallel.
I would've assumed "brute force the best filter" would include 0 internally, but good to know.
Brute Force I would guess does include 0:no filter but per-line. So most likely for some images the Brute Force mode gives the best filter for each line but different filters for most lines, and this changing of filters need to be stored to the image data, which ends up added more data than just having no filter(but worst compression when not counting the filter data).
Also I'm write this in case someone else stumbles onto this, And really wondered.
