wrappers for opencv

[timotheecour]

IMO providing good integration with opencv will be key to dcv's success. The idea is to get quickly to a point where dcv has enough features to be useful for production use without waiting for opencv algorithms to be reimplemented. Some of these algorithms can in a later stage, if needed, be reimplemented as drop-in replacement for the opencv ones, maybe in some cases without changing public APIs.

I have done something like this using SWIG to auto-generate wrappers for a large portion of opencv (intermediate layer) along with higher-level idiomatic D layer and a tensor class that could be (shallow copy) converted to/from cv::Mat_<T>.

For dcv one could either use swig or take advantage of the newer extern(C++,ns) syntax, and provide shallow conversion from Slice to/from cv::Mat_<T>.

Happy to discuss this further.

[9il]

Low level deimos-like wrapper should be located in libmir/opencv. High level ndslice wrappers in libmir/mir-opencv.

[ljubobratovicrelja]

@timotheecour Thanks for bringing this up. I too agree we should make nice wrappers for opencv. And like opencv's python wrapper uses numpy.array natively, so should our wrapper use ndslice, as @9il points out.

High level ndslice wrappers in libmir/mir-opencv.

Hmm, how about having opencv configuration inside dcv, that acts like that higher level interface? Nice thing about this is that we don't need to wrap stuff that's already inside the dcv, and also, we could over time replace opencv implementation with ours, without changing the interface. In other words - let's use opencv as optional back-end?

I would propose the following - have low-level (opencv's c++ interface bind, maybe with SWIG) in libmir/opencv, and selectively wrap that interface into dcv to match it's interface.

[9il]

We can have 3 levels the same way as with lubeck

cblas -> mir-blas \
                      ->    lubeck
lapack -> mir-lapack /

For DCV it should looks like:

opencv -> mir-opencv -> dcv

[9il]

mir-lapack and mir-blas are selective too.

[timotheecour]

maybe it's better to discuss this with a concrete example, say detectMultiScale from http://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html?highlight=detectmultiscale#cascadeclassifier-detectmultiscale

C++:
class CascadeClassifier{
  void detectMultiScale(const Mat& image, vector<Rect>& objects, double scaleFactor=1.1, int minNeighbors=3, int flags=0, Size minSize=Size(), Size maxSize=Size());
};

maybe in libmir/opencv or libmir/swig-opencv

module mir.swig.opencv;
// autogenerated via swig/other, can be a large subset of opencv, can be available soon although not with D-idiomatic api

// NOTE: her, Mat, Rect, vector are swig-generated, and as such not idiomatic D
class CascadeClassifier{
  void detectMultiScale(const ref Mat image, ref vector!(Rect) objects, double scaleFactor, int minNeighbors, int flags, Size minSize, Size maxSize);
}

maybe libmir/opencv or libmir/mir-opencv

module mir.opencv;
import mir.swig.opencv;
// manually generated, will have to be maintained and this'll grow over time starting from 0; but stays close to swig generated APIs for simplicity, just provides friendlier more idiomatic API's
class CascadeClassifier{
  // Note: this has to be restricted to the types that were compiled in in swig and opencv, as we obviously can only use C++ templates with types explicitly instantiated in the bindings
  void detectMultiScale(T)(const ref T image, Rect[] objects, double scaleFactor, int minNeighbors, int flags, Size minSize, Size maxSize) if(isSlice!T);
}

There's already API choices that need to be made in this simple example:

• use a dynamic-like types such as cv::Mat (maybe a D type called Mat) which is agnostic to element type+number of channels, or use templates with template constraint (eg: isSlice!T && (T.ndims==2 || T.ndims==3)) analog to cv::Mat_<T>; The former may be easier (although giving less type safe API's) given that most opencv API's are in terms of cv::Mat, not cv::Mat_<T>
• what to do with things like C++ vector: use D's [] or have a class wrapping C++'s vector; the former is tricky as it either requires copy or requires managing ownership of memory between D's gc and C++'s heap allocated vector; Note that swig handles releasing resources allocated in C++ ; we can probably use same logic
• use a native D class for things like Rect (struct Rect{int x, int y, ...}) (but requires conversions via a deep copy) or just use swig types or just use swig types? Probably best to use native D types for light objects such as Rect etc, and only use shallow copies for things like cv::Mat (which can be shallow-copied to nativeSlice from mir)

maybe in libmir/dcv:

module mir.dcv.detection;
import mir.swig.opencv;
import mir.opencv;

// this can be a higher level completely reworked API, with different arguments / classes, and can use opencv wrapped methods as implementation detail (and potentially for a subset of types) or native implementation, as implementat detail, without changing external API.

[9il]

How it is implemented in Python bindings?

[timotheecour]

swig EDIT: actually I remembered wrong, looks like custom tooling instead:

http://answers.opencv.org/question/6618/how-python-api-is-generated/

however, it's possible to wrap opencv via swig; I did that at some point (for whatever functions from opencv I needed), it was painful to get it right but it can be done (and pays off because it's automatic). Here's an example of another project going that route: https://github.com/renatoGarcia/opencv-swig

[ljubobratovicrelja]

swig

Not really. This answer is old but I doubt they've changed much.

boost::python::numpy is worth a look. Maybe something like this should be done via PyD for ndslice?

[9il]

boost::python::numpy is worth a look. Maybe something like this should be done via PyD for ndslice?

https://github.com/libmir/mir-algorithm/issues/43

[timotheecour]

Besides manual (via extern(C++,ns)) and automated (via SWIG), there is another approach to autogenerate C++<>D bindings but I haven't tried it: calypso:

https://forum.dlang.org/post/[email protected] Calypso: Direct and full interfacing to C++

http://www.digitalmars.com/d/archives/digitalmars/D/Calypso_and_the_future_of_D_253820.html#N253820 digitalmars.D - Calypso and the future of D

https://github.com/Syniurge/Calypso

[9il]

Wow, Calypso is active!

[timotheecour]

• calypso: it would definitely make interfaces easier. However it's also a risky approach (eg it's based on a fork so risks further limiting the ecosystem on which it'd be available). Just asked here: https://github.com/Syniurge/Calypso/issues/51

• For swig bindings, would be nice to improve SWIG D generator which is far from perfect, specifically: adding support for extern(C++, ns) [right now only uses extern(C) which makes things more complex than they should]; allow swig to translate C++ templates to D templates (i had done something like this before extern(C++) but that was messy; with extern(C++) it should be easier; but again, requires modifying SWIG interface genrator

• semi-manual approach: IIRC that's hows dmd was ported to D (don't know the details), but made strong assumptions about dmd's specific codebase and may not work well with opencv

Also relevant discussion: https://internals.rust-lang.org/t/interfacing-d-to-legacy-c-code-a-summary-of-a-competing-languages-capabilities/1406 + video from walter bright: https://www.youtube.com/watch?v=IkwaV6k6BmM + slides http://walterbright.com/cppint.pdf

In any case, investing in an automatic solution is the right thing to do (pays down the line)

[henrygouk]

I have done something like this using SWIG to auto-generate wrappers for a large portion of opencv (intermediate layer) along with higher-level idiomatic D layer and a tensor class that could be (shallow copy) converted to/from cv::Mat_.

@timotheecour is this publicly available?

[timotheecour]

not at the moment unfortunately. Would be happy to contribute on this though if that approach is considered

[timotheecour]

a lot of bugs have recently been fixed in calypso making non-trivial opencv programs work in D, so IMO this should be preferred way forward

[infinityplusb]

Do you have some working examples of opencv with D and Calypso?