Windows conda build broken by using versioned python modules

[peci1]

Environment

• OS Version: Win 10
• Source or binary build? Source, sdf13

Description

#1143 broke conda build on Windows. I now get this error when building:

  Actor.obj : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance
     Creating library D:/programovani/ign5-ws/build/sdformat13/lib/Release/sdformat13.lib and object D:/programovani/ign5-ws/build/sdformat13/lib/Release/sdformat13.exp
  Generating code
[Processing: sdformat13]sdformat13:build - 29.9s]
  Finished generating code
  sdformat13.vcxproj -> D:\programovani\ign5-ws\build\sdformat13\bin\Release\sdformat13.dll
LINK : fatal error LNK1149: output filename matches input filename 'D:\programovani\ign5-ws\build\sdformat13\lib\Release\sdformat13.lib' [D:\programovani\ign5-ws\build\sdformat13\python\pysdformat13.vcxproj]

Before #1143, build is okay.

[peci1]

@j-rivero FYI

[j-rivero]

@j-rivero FYI

Do you have the log of the build at hand?

[peci1]

Do you have the log of the build at hand?

Not at the moment, but I can get it later. What I've posted is the only part that seems like errors in the log.

[traversaro]

Probably this is similar to https://github.com/pantor/ruckig/pull/18, in particular setting ARCHIVE_OUTPUT_NAME appropriately we should avoid name collisions.

[peci1]

Build log of current `sdf13` branch:

-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.19044.
-- sdformat13 version 13.0.0~pre2
-- Operating system is Windows
-- Checking for module 'tinyxml2'
--   Found tinyxml2, version 9.0.0
-- Looking for TINYXML2 - found

-- Looking for GzURDFDOM - found

-- Could NOT find PY_psutil (missing: PY_PSUTIL) 
-- Found ruby executable: D:/Programy/Conda/envs/ign5-ws/bin/ruby.exe
-- Looking for gz-math7 -- found version 7.0.0~pre2
-- Searching for dependencies of gz-math7
-- Looking for gz-utils2 -- found version 2.0.0~pre2
-- Searching for dependencies of gz-utils2
-- Looking for gz-math7 - found

-- Looking for gz-utils2 -- found version 2.0.0~pre2
-- Searching for dependencies of gz-utils2
-- Searching for <gz-utils2> component [cli]
-- Looking for gz-utils2-cli -- found version 2.0.0~pre2
-- Searching for dependencies of gz-utils2-cli
-- Looking for gz-utils2 - found

-- Searching for Python - found version 3.10.6.
-- Searching for pybind11 - found version 2.10.0.
CMake Warning at D:/programovani/ign5-ws/install/share/cmake/gz-cmake3/cmake3/GzConfigureBuild.cmake:69 (message):
   CONFIGURATION WARNINGS:
   -- Python psutil package not found. Memory leak tests will be skipped
Call Stack (most recent call first):
  CMakeLists.txt:157 (gz_configure_build)


-- Configuring library: sdformat13
-- The program [cppcheck] was not found! Skipping codecheck setup
-- Build configuration successful
-- Build type: RelWithDebInfo
-- Install prefix: D:/programovani/ign5-ws/install
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Configuration successful. Type make to compile sdf
-- Configuring done
-- Generating done
-- Build files have been written to: D:/programovani/ign5-ws/build/sdformat13
Microsoft (R) Build Engine verze 16.8.2+25e4d540b pro .NET Framework

Copyright (C) Microsoft Corporation. Vçechna pr va vyhrazena.



  EmbeddedSdf.cc

  Actor.cc

  AirPressure.cc

  Altimeter.cc

  Atmosphere.cc

  Box.cc

  Camera.cc

  Capsule.cc

  Collision.cc

  Console.cc

  Converter.cc

  Cylinder.cc

  Element.cc

  Ellipsoid.cc

  Error.cc

  Exception.cc

  Filesystem.cc

  ForceTorque.cc

  Frame.cc

  FrameSemantics.cc

  Geometry.cc

  Gui.cc

  Heightmap.cc

  Imu.cc

  InterfaceElements.cc

  InterfaceFrame.cc

  InterfaceJoint.cc

  InterfaceLink.cc

  InterfaceModel.cc

  InterfaceModelPoseGraph.cc

  Joint.cc

  JointAxis.cc

  Lidar.cc

  Light.cc

  Link.cc

  Magnetometer.cc

  Material.cc

  Mesh.cc

  Model.cc

  NavSat.cc

  Noise.cc

  OutputConfig.cc

  Param.cc

  ParamPassing.cc

  ParserConfig.cc

  ParticleEmitter.cc

  Pbr.cc

  Physics.cc

  Plane.cc

  Plugin.cc

  Polyline.cc

  PrintConfig.cc

  Root.cc

  SDF.cc

  SDFExtension.cc

  Scene.cc

  SemanticPose.cc

  Sensor.cc

  Sky.cc

  Sphere.cc

  Surface.cc

  Types.cc

  Utils.cc

  Visual.cc

  World.cc

  XmlUtils.cc

  gz.cc

  parser.cc

  parser_urdf.cc

  Actor.obj : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance

     Creating library D:/programovani/ign5-ws/build/sdformat13/lib/Release/sdformat13.lib and object D:/programovani/ign5-ws/build/sdformat13/lib/Release/sdformat13.exp

  Generating code

  Finished generating code

  sdformat13.vcxproj -> D:\programovani\ign5-ws\build\sdformat13\bin\Release\sdformat13.dll

  _gz_sdformat_pybind11.cc

  pyAirPressure.cc

  pyAltimeter.cc

  pyAtmosphere.cc

  pyBox.cc

  pyCamera.cc

  pyCapsule.cc

  pyCollision.cc

  pyCylinder.cc

  pyEllipsoid.cc

  pyError.cc

  pyForceTorque.cc

  pyFrame.cc

  pyGeometry.cc

  pyGui.cc

  pyHeightmap.cc

  pyIMU.cc

  pyJoint.cc

  pyJointAxis.cc

  pyLidar.cc

  pyLight.cc

  pyLink.cc

  pyMagnetometer.cc

  pyMaterial.cc

  pyMesh.cc

  pyModel.cc

  pyNavSat.cc

  pyNoise.cc

  pyParserConfig.cc

  pyParticleEmitter.cc

  pyPbr.cc

  pyPhysics.cc

  pyPlane.cc

  pyPlugin.cc

  pyPolyline.cc

  pyRoot.cc

  pyScene.cc

  pySemanticPose.cc

  pySensor.cc

  pySky.cc

  pySphere.cc

  pySurface.cc

  pyVisual.cc

  pyWorld.cc

  pybind11_helpers.cc

LINK : fatal error LNK1149: output filename matches input filename 'D:\programovani\ign5-ws\build\sdformat13\lib\Release\sdformat13.lib' [D:\programovani\ign5-ws\build\sdformat13\python\pysdformat13.vcxproj]

[peci1]

I've tried setting the RUNTIME_OUTPUT_NAME of BINDINGS_MODULE_NAME for MSVC and with that change, build succeeds. With ARCHIVE_OUTPUT_NAME it does not.

However, I still have not been able to use the built module. The generated module is in install\lib\python and its name is pysdformat13.cp310-win_amd64.pyd.

$ call install/setup.bat
$ set PYTHONPATH=D:\programovani\ign5-ws\install\lib\python

$ python -c "import pysdformat13"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: DLL load failed while importing sdformat: Module not found

$ dumpbin -dependents install\lib\python\pysdformat13.cp310-win_amd64.pyd
Microsoft (R) COFF/PE Dumper Version 14.28.29335.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file install\lib\python\pysdformat13.cp310-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    python310.dll
    sdformat13.dll
    gz-math7.dll
    MSVCP140.dll
    VCRUNTIME140_1.dll
    VCRUNTIME140.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    KERNEL32.dll
    api-ms-win-crt-math-l1-1-0.dll

  Summary

        3000 .data
        7000 .pdata
       36000 .rdata
        1000 .reloc
        1000 .rsrc
       B4000 .text

$ Dependencies -chain -depth 1 install\lib\python\pysdformat13.cp310-win_amd64.pyd
├ pysdformat13.cp310-win_amd64.pyd (ROOT) : install\lib\python\pysdformat13.cp310-win_amd64.pyd
|  ├ python310.dll (Environment) : D:\Programy\Conda\envs\ign5-ws\python310.dll
|  ├ sdformat13.dll (Environment) : D:\programovani\ign5-ws\install\bin\sdformat13.dll
|  ├ gz-math7.dll (Environment) : D:\programovani\ign5-ws\install\bin\gz-math7.dll
|  ├ MSVCP140.dll (WindowsFolder) : C:\WINDOWS\system32\MSVCP140.dll
|  ├ VCRUNTIME140_1.dll (WindowsFolder) : C:\WINDOWS\system32\VCRUNTIME140_1.dll
|  ├ VCRUNTIME140.dll (WindowsFolder) : C:\WINDOWS\system32\VCRUNTIME140.dll
|  ├ api-ms-win-crt-heap-l1-1-0.dll (ApiSetSchema) : C:\WINDOWS\system32\ucrtbase.dll
|  ├ api-ms-win-crt-string-l1-1-0.dll (ApiSetSchema) : C:\WINDOWS\system32\ucrtbase.dll
|  ├ api-ms-win-crt-runtime-l1-1-0.dll (ApiSetSchema) : C:\WINDOWS\system32\ucrtbase.dll
|  ├ KERNEL32.dll (WellKnownDlls) : C:\WINDOWS\system32\kernel32.dll
|  ├ api-ms-win-crt-math-l1-1-0.dll (ApiSetSchema) : C:\WINDOWS\system32\ucrtbase.dll

[traversaro]

I've tried setting the RUNTIME_OUTPUT_PATH of BINDINGS_MODULE_NAME for MSVC and with that change, build succeeds. With ARCHIVE_OUTPUT_PATH it does not.

Just a small nitpick for other people reading: if I am not wrong there is no officially documented RUNTIME_OUTPUT_PATH nor ARCHIVE_OUTPUT_PATH property in CMake, but there exists RUNTIME_OUTPUT_PATH and RUNTIME_OUTPUT_DIRECTORY and ARCHIVE_OUTPUT_DIRECTORY .

[peci1]

I've tried setting the RUNTIME_OUTPUT_PATH of BINDINGS_MODULE_NAME for MSVC and with that change, build succeeds. With ARCHIVE_OUTPUT_PATH it does not.

Just a small nitpick for other people reading: if I am not wrong there is no officially documented RUNTIME_OUTPUT_PATH nor ARCHIVE_OUTPUT_PATH property in CMake, but there exists RUNTIME_OUTPUT_PATH and RUNTIME_OUTPUT_DIRECTORY and ARCHIVE_OUTPUT_DIRECTORY .

Ahh, typo! I meant RUNTIME_OUTPUT_NAME. I've edited the previous post to use the correct property names.

[traversaro]

Ah, then that is quite strange. Which value of ARCHIVE_OUTPUT_NAME did you used?

[peci1]

I basically just transplanted https://github.com/pantor/ruckig/pull/18, so:

if (NOT MSVC)
  set_target_properties(${BINDINGS_MODULE_NAME} PROPERTIES
    OUTPUT_NAME "sdformat${PROJECT_VERSION_MAJOR}")
else()
  set_target_properties(${BINDINGS_MODULE_NAME} PROPERTIES
    RUNTIME_OUTPUT_NAME "sdformat${PROJECT_VERSION_MAJOR}")
endif()

If I used ARCHIVE_OUTPUT_NAME instead of RUNTIME_OUTPUT_NAME, build had failed. Should I try setting both on MSVC and manually set ARCHIVE_OUTPUT_NAME to a different value?

[traversaro]

That is on me, that ruckig PR is wrong, so linking it was confusing. : ) The relevant information of the PR is not the PR itself, but the comment https://github.com/pantor/ruckig/pull/18#issuecomment-813101625 and the corresponding commit https://github.com/pantor/ruckig/commit/d494c08a9fff7564d97aee185c38d9094795882a .

The naming collision is caused by the both the import library of sdformat library and the import library of the pybind11 bindings library are called sdformat13.lib. If we set ARCHIVE_OUTPUT_NAME of the bindings to be sdformat13, we do not do anything to avoid this.

The right thing to do is:

set_target_properties(${BINDINGS_MODULE_NAME} PROPERTIES
  OUTPUT_NAME "sdformat${PROJECT_VERSION_MAJOR}")
set_target_properties(${BINDINGS_MODULE_NAME} PROPERTIES
  ARCHIVE_OUTPUT_NAME "python_sdformat${PROJECT_VERSION_MAJOR}")

Instead of "python_sdformat${PROJECT_VERSION_MAJOR}" you can use any string that you prefer, the important thing is that it is different from "sdformat${PROJECT_VERSION_MAJOR}".

[j-rivero]

Instead of "python_sdformat${PROJECT_VERSION_MAJOR}" you can use any string that you prefer, the important thing is that it is different from "sdformat${PROJECT_VERSION_MAJOR}".

Disclaimer: excuse me if this is a silly question. If we rename the python file, code like import sdformat13 is still going to be working.

[traversaro]

Instead of "python_sdformat${PROJECT_VERSION_MAJOR}" you can use any string that you prefer, the important thing is that it is different from "sdformat${PROJECT_VERSION_MAJOR}".

Disclaimer: excuse me if this is a silly question. If we rename the python file, code like import sdformat13 is still going to be working.

Yes, as we are just changing the name of the import library (.lib), not the dynamic(-link) library (.dll or pyd). At runtime, the only file used is the dynamic(-link) library, the import library will only be used by the C/C++ linker if the bindings library is linked to another C++ library.

[j-rivero]

Yes, as we are just changing the name of the import library (.lib), not the dynamic(-link) library (.dll or pyd). At runtime, the only file used is the dynamic(-link) library, the import library will only be used by the C/C++ linker if the bindings library is linked to another C++ library.

It was a silly question, indeed. Thanks Silvio.