Cadquery running in threads

[MarcSallent]

Hello,

I am generating hundreds of different geometries in a single script and, aiming to speed execution, I have tried to run CadQuery in multiple separate threads. What I've seen is that it is not faster (actually a bit slower) than running everything in a single thread.

Is that expected?

Thanks

Marc

[jmwright]

I tried a few experiments several months ago with running tessellation with multiple threads and got similar results. I didn't dig into why that is, but probably will at some point. I suspect running things in parallel really only shows improvement over single-threaded operation with very complex models.

I wouldn't be surprised if there's a community member that understands and can explain why.

[adam-urbanczyk]

OCP does not release the GIL. Some of the operations (for sure bool ops) are parallelized internally.

[michaelgale]

I've often wondered about the opportunity for parallelism. Typically I observe that CQ python process saturates one core--tempting me to think about parallelism. For some operations, I can launch more than one in separate terminal processes; i.e. for building intermediate solids. However, when it comes to compositing a final complex model, its single-core all the way! When browsing the OCC API I do see references to "parallelism" which seem to be introduced for v. 7.x+

[SDI8]

I might be late to the game, but I am successfully using pythons multiprocessing in multiple projects to generate large numbers of parts in parallell. I use copyreg to register serialization- and deserialization-helpers for common cadquery objects. https://gist.github.com/SDI8/3137ee70649e4901913c7c8e6b534ec8

As stated above, some OCC operations do use multithreading internally, so the performance benefit will vary. Also, pythons multiprocessing has a performance overhead, as everything going in or out has to be serialized. In my cases, I still get somewhere between x4 and x8 performance boost on a 16 core system - which is worth the effort.

[jmwright]

@SDI8 Could you also post a link to some code that's using the serialization helpers so we can see how everything works together?

[SDI8]

Sure! Here is an example of generating 9 cubes of increasing size in parallel. I am using Pool.map, but any way of spawning a Process using multiprocessing should work. As noted before, my copyreg helpers only cover the cq objects I had a need for. There is likely more to add if need be.

from multiprocessing import Pool
import cadquery as cq
from cq_serialize import register as register_cq_helper

register_cq_helper()

def make_box(size: float):
    return cq.Workplane("XY").box(size, size, size).findSolid()

with Pool() as p:
    boxes = p.map(make_box, range(1, 10))

Also, obviously this example is too small to demonstrate a performance gain.

[jmwright]

Thanks @SDI8 !