Multiple parallel configure with Colcon and CPM: not supported

[huand]

Hello, Im not too sure this is the cause of the problem, but when using colcon (a build tool for ros2 packages), I think colcon starts all leaf dependencies config and build in "parallel", if there are multiple duplicate dependencies among those parallel build, CPM fails. Example, where MyProject1 is used by several leaf projects and downloaded by cpm:

[ 11%] Creating directories for 'MyProject1'
[ 22%] Performing download step (git clone) for 'MyProject1'
Cloning into 'fc4e6fc51c1494235cb68e60c5b7fb54c701a612'...
fatal: shallow file has changed since we read it
fatal: destination path 'fc4e6fc51c1494235cb68e60c5b7fb54c701a612' already exists and is not an empty directory.
fatal: destination path 'fc4e6fc51c1494235cb68e60c5b7fb54c701a612' already exists and is not an empty directory.
-- Had to git clone more than once:
          3 times.

however if i start the process again, then it works. would it be possible for CPM to check if another CPM is already active for some deps and if another cpm process wants the same deps to just wait for the other to finish instead of failing? I understand this is probably a niche case for build system like Colcon.

[TheLartians]

Hey, thanks for the issue! I don't know about how the colcon build tool works, but I think if we want to support parallel builds we would need to introduce a sort of lock for the CPM source cache directory. There will probably be some complexity involved if we want to avoid creating hangups and edge-cases.

Does deactivating the source cache, e.g. by calling unset CPM_SOURCE_CACHE before building, resolve the issue?

[egorodet]

Hi @TheLartians, I'm experiencing a similar issue in CLion IDE, which allows to work with multiple CMake configurations and runs cmake configure for active configurations in parallel. In such case it became impossible to use CPM_SOURCE_CACHE because of update collisions, so I had to fallback to separate source locations, which seems to be slowly working. I wish CPM would have a directory locking protection to allow parallel use cases.

[TheLartians]

Is this fixed in v0.37.0 with #427?

[egorodet]

Is this fixed in v0.37.0 with #427?

@TheLartians I've just tested v0.37 with multiple CMake configurations initializing dependencies in parallel in CLion and it worked just fine without any errors! Thanks a lot for fixing this painful issue!

[TheLartians]

Great, thanks for testing! I close the issue for now as I assume this directly relates to the Colcon error as well.