'scalable_allocation_command' not found

[alenauus]

Adding: scalable_allocation_command(TBBMALLOC_CLEAN_ALL_BUFFERS, 0); for cleaning thread caches causes the following error: undefined reference to `scalable_allocation_command'

Is it related to FindTBB.cmake or other issues?

Thanks, Alena

[schuhschuh]

Hi Alena,

Sorry for the late replay. I have not encountered this error before. It could well be related to FindTBB and the imported CMake build targets it adds. If this is still an issue, could you give some more context? Such as your operating system, your CMakeCache.txt, and the complete error output? E.g., is this a compiler error or linker error?

Thanks, Andreas

[alenauus]

Hi Andreas,

Thank you for your reply. As far as I remember it was a linker error and I managed to fix it - but 'scalable_allocation_command' does not work in general (I tested it separately).

The main issue is that TBB threads do not release memory after termination until the end of the main program. And it causes issues with RAM for SVR reconstruction of large ROIs (we use it for deformable SVR of fetal body).

It would be great if you know how to fix it.

DSVR code will be published online soon as a part of SVRTK (MIRTK-based package): https://github.com/SVRTK/SVRTK .

Thanks, Alena

[schuhschuh]

Thanks for providing more information. I found some TBB issue reports on this that might be useful:

1. https://software.intel.com/en-us/forums/intel-threading-building-blocks/topic/498725
2. https://software.intel.com/en-us/forums/intel-threading-building-blocks/topic/777497

In MIRTK itself, I am not sure scalable_allocator is actually used. It's introduced into the mirtk namespace here when MIRTK_COMMON_WITH_TBB_MALLOC is enabled. However, the default setting in the mirtk/Parallel.h source file is MIRTK_COMMON_WITH_TBB_MALLOC=0, i.e., to not make use of it.

In MIRTK code, I only used the cache_aligned_allocator in one source of the Deformable package (cf. commit 3fc1668c66b91397005f8d69fdae84cd55e13ba8). In the Packages/Deformable/BasisProject.cmake file, this package looks for TBB{tbb,malloc}, i.e., including the tbbmalloc library, whereas Modules/Common/BasisProject.cmake only requries TBB{tbb}. The source file Packages/Deformable/src/ImageEdgeDistance.cc starts with the line #define MIRTK_COMMON_WITH_TBB_MALLOC 1.

To be sure that the tbbmalloc dependency is not inherited from this package, you could either disable it (MODULE_Deformable=OFF) when you build MIRTK, or remove malloc from TBB{tbb,malloc} in the BasisProject.cmake file of all packages and change any MIRTK_COMMON_WITH_TBB_MALLOC definition to 0.

I can see that in your SVRTK package you declare TBB{tbb,malloc} as dependency. You could try to remove the malloc library from the dependencies, i.e., change this line to TBB{tbb}.

Otherwise, if you want TBB's memory functionality, have you tried using memory_pool instead of scalable_allocator as suggested by Alex' answer in [1]?

[alenauus]

Yes, using tbb::memory_pool would be the optimal solution (I tested it separately on a simple code and it resolves the issue). But in reality I am simply not sure how it should be incorporated into MIRTK code / variables and functions (the main question is where to begin). I do understand that it is not a trivial task.

Or would it make sense to use additional c11 multithreading options?

Thanks, Alena

[schuhschuh]

Have you tried in SVRTK where you need this memory pool to use the tbb::memory_pool_allocator in the declaration of, e.g., the Array (std::vector) you use? Once you found a solution in your SVRTK package, we can think about how to merge it into the core modules. Where is the code of the TBB body that causes the memory issues?

[alenauus]

As far as I remember I tried using memory_pool_allocator for 'RealImage' but eventually gave up. Will try it again.

It seems that the issue originates in the logic of implementation of this TBB function: void custom_scheduler<SchedulerTraits>::local_wait_for_all( task& parent, task child )* in [ttb_folder]/srs/tbb/custom_scheduler.h - Ln 356 (but I am not sure).

I will ask a question at tbb github forum - whether there is a way to fix it.

[schuhschuh]

Let me know if you find an answer. Otherwise, after looking at SVRTK, I think there are a few possibilities to restructure the code in a way that doesn't require memory allocation/deallocation each time a new thread is spawned.

I think this issue is not caused and cannot really be addressed within the MIRTK source code, because the MIRTK libraries don't actually depend on / make use of the TBB malloc library (except for libMIRTKDeformable). It seems to be the case that this is a dependency introduced with SVRTK. If there is an easy way for me to reproduce the error with data you can provide, I could try and have a look.

Do you observe a substantial runtime penalty for using the STL memory allocations instead of TBB's? Otherwise, it is best not to have memory allocations to take place within each thread, but instead pre-allocate your memory (and pre-instantiate your objects) before the parallel_for or parallel_reduce operations. That way you avoid any runtime penalty caused by concurrent memory allocation/deallocations and also the need for TBB malloc in the first place.

For example, if you need ImageTransformation objects in multiple threads, you can either specify the number of threads and provide as many pre-instantiated ImageTransformation objects to use by these with a thread-safe container such as tbb:: concurrent_unordered_set (or using mutex try_lock semantics) to recycle registration objects between threads.

[alenauus]

Thanks for the suggestion it really makes sense to pre-instantiate objects. I've sent you the code and a dataset example to @imperial email.

p.s we have the same memory issue with IRTK-based reconstruction (if we use it for very large ROIs - instead of brain only)

[schuhschuh]

Thanks, Alena, for sharing your code. I'll try and have a look this week.

Follow up and remind me if I don't :-).