Halide
Halide copied to clipboard
halide
Reuse compiled pipeline on multiple threads.
I'm building a scientific academic research application, that uses Halide. I have access to a beefy 40-core machine. Is it possible to reuse a compiled Halide function/pipeline on multiple threads in parallel? In terms of: setting pipeline parameters, setting input buffers, setting output buffers, making sure internal temporary buffers are not shared.
I would like to save on compilation time, as I'm now literally JIT compiling the same pipeline 40 times before the application starts it's actual work.
tl;dr We need to add some support to do this, though it isn't too difficult.
While multithreaded calls are fully supported for AOT compilation, there are two obstacles to making them work for JIT. (Note I'm going to use Pipeline instead of Func below as the actual implementation of functionality is at the Pipeline level. There should be pass through methods from Func to Pipeline for everything mentioned and thus the advice should apply to both.)
First, the Pipeline has to be compiled before it is called. The easiest way to do this is to arrange to call compile_jit() on the Pipeline on a single thread before doing realizations on other threads.
Second, JIT calls typically take their arguments from global Param and ImageParam objects and these are not threadsafe. The solution is to call the argv function directly with values taken from some passed in data structure, such as a map that has named entries corresponding to the names of the Param and ImageParam objects. If you need it in a hurry and are willing to make an array of pointers to the raw arguments, all this would require is making an accessor on Pipeline to the argv_function on the compiled module. But we should be able to make a threadsafe method that ensures the compilation has happened and takes all the args from passed in values...
I am most definitely not familiar with the internals of Halide. The argv_function you are referring to, is that the function that is returned by compile_jit()? If so, does it take the same parameters as if I would generate a AOT header file?
The return value of compile_jit() could be documented better, I think.
That would be a very helpful functionality, are there any plans on doing this? Just curious.
Yes, it is on my TODO list. Probably not tomorrow, but probably within a month.
Please take a look at https://github.com/halide/Halide/pull/2657 and comment if it does not meet your needs per this issue. I expect this will get merged within a week.
Fixed by https://github.com/halide/Halide/pull/2657 . Pass a ParamMap as the last argument of realize to access the new functionality.
I'm reopening this (5 years later!) because there is a proposed PR (https://github.com/halide/Halide/pull/6777) that may address this in a more flexible and useful way (and may eventually allow us to deprecate ParamMap entirely).
Could any of the original commenters (@mcourteaux and @edubois specifically) comment on whether they are (still?) using ParamMap (or Halide in general), and/or take a look at #6777 to see how it looks for their purposes?
