rattler-build icon indicating copy to clipboard operation
rattler-build copied to clipboard
verified publisher icon prefix-dev

Files from dependency non-deterministically included in package built on Windows

Open traversaro opened this issue 10 months ago • 4 comments

In the last weeks I experienced a non-deterministic issue when building Windows package with rattler-build . In particular, what is happening is that sometimes files from the dependency are included in the built packages.

The problem is non-deterministic so it is tricky to reproduce, but I was able to put together a mwe in https://github.com/traversaro/rattler-build-clobber-bomb-mre, that you can run (on Windows) with:

git clone https://github.com/traversaro/rattler-build-clobber-bomb-mre
cd rattler-build-clobber-bomb-mre
pixi run reproduce

What happens (just in some cases) is that the build packages also include some files that are part of the dependencies. For example, see the run in https://github.com/traversaro/rattler-build-clobber-bomb-mre/actions/runs/12931376501/job/36065083136 the log includes:

2025-01-23T14:34:46.1642155Z  │ │   -- Generating done (0.4s)
2025-01-23T14:34:46.1653624Z  │ │   -- Build files have been written to: %SRC_DIR%/build
2025-01-23T14:34:46.2494056Z  │ │   1>
2025-01-23T14:34:46.3053405Z  │ │   -- Install configuration: "Release"
2025-01-23T14:34:46.3054427Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/joint_limiters.xml
2025-01-23T14:34:46.3059494Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits
2025-01-23T14:34:46.3062788Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits
2025-01-23T14:34:46.3067122Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limiter_interface.hpp
2025-01-23T14:34:46.3073451Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limits.hpp
2025-01-23T14:34:46.3079327Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limits_rosparam.hpp
2025-01-23T14:34:46.3086998Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limits_urdf.hpp
2025-01-23T14:34:46.3092331Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_saturation_limiter.hpp
2025-01-23T14:34:46.3099217Z  │ │   -- Installing: %PREFIX%/Library/lib/joint_limiter_interface.lib
2025-01-23T14:34:46.3106185Z  │ │   -- Installing: %PREFIX%/Library/bin/joint_limiter_interface.dll
2025-01-23T14:34:46.3113286Z  │ │   -- Installing: %PREFIX%/Library/lib/joint_saturation_limiter.lib
2025-01-23T14:34:46.3123299Z  │ │   -- Installing: %PREFIX%/Library/bin/joint_saturation_limiter.dll
2025-01-23T14:34:46.3140403Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/package_run_dependencies/joint_limits
2025-01-23T14:34:46.3154232Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/parent_prefix_path/joint_limits
2025-01-23T14:34:46.3168332Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/ament_prefix_path.bat
2025-01-23T14:34:46.3176745Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/ament_prefix_path.dsv
2025-01-23T14:34:46.3183116Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/path.bat
2025-01-23T14:34:46.3189856Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/path.dsv
2025-01-23T14:34:46.3196663Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/local_setup.bat
2025-01-23T14:34:46.3203574Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/local_setup.dsv
2025-01-23T14:34:46.3210254Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/package.dsv
2025-01-23T14:34:46.3217591Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/packages/joint_limits
2025-01-23T14:34:46.3225316Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/joint_limits__pluginlib__plugin/joint_limits
2025-01-23T14:34:46.3233918Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/export_joint_limitsExport.cmake
2025-01-23T14:34:46.3242039Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/export_joint_limitsExport-release.cmake
2025-01-23T14:34:46.3249146Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/ament_cmake_export_targets-extras.cmake
2025-01-23T14:34:46.3255729Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/ament_cmake_export_dependencies-extras.cmake
2025-01-23T14:34:46.3263694Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/joint_limitsConfig.cmake
2025-01-23T14:34:46.3272786Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/joint_limitsConfig-version.cmake
2025-01-23T14:34:46.3279068Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/package.xml
2025-01-23T14:34:46.3287253Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/version.h
2025-01-23T14:34:46.3734995Z  │ │ %SRC_DIR%\build>if errorlevel 1 exit 1 
2025-01-23T14:34:46.3755201Z  │ │
2025-01-23T14:34:46.3755742Z  │ ╰─────────────────── (took 77 seconds)
2025-01-23T14:34:46.7787695Z  │
2025-01-23T14:34:46.7788107Z  │ ╭─ Packaging new files
2025-01-23T14:34:46.7788437Z  │ │ Copying done!
2025-01-23T14:34:46.8068570Z  │ │ Post-processing done!
2025-01-23T14:34:46.8068971Z  │ │ Writing test files
2025-01-23T14:34:46.8075644Z  │ │ Writing metadata for package
2025-01-23T14:34:46.8201346Z  │ │ Copying license files
2025-01-23T14:34:46.8201869Z  │ │ Copying recipe files
2025-01-23T14:34:46.8305865Z  │ │ Creating entry points
2025-01-23T14:34:46.8306264Z  │ │ 
2025-01-23T14:34:46.8306529Z  │ │ Files in package:
2025-01-23T14:34:46.8307190Z  │ │   - Library/bin/joint_limiter_interface.dll
2025-01-23T14:34:46.8307983Z  │ │   - Library/bin/joint_saturation_limiter.dll
2025-01-23T14:34:46.8308735Z  │ │   - Library/cmake/console_bridge-config-version.cmake
2025-01-23T14:34:46.8309498Z  │ │   - Library/cmake/console_bridge-config.cmake
2025-01-23T14:34:46.8310448Z  │ │   - Library/cmake/console_bridge-targets-release.cmake
2025-01-23T14:34:46.8311268Z  │ │   - Library/cmake/console_bridge-targets.cmake
2025-01-23T14:34:46.8312012Z  │ │   - Library/cmake/urdfdom_headers-config-version.cmake
2025-01-23T14:34:46.8312760Z  │ │   - Library/cmake/urdfdom_headers-config.cmake
2025-01-23T14:34:46.8313474Z  │ │   - Library/cmake/urdfdom_headersExport.cmake
2025-01-23T14:34:46.8314363Z  │ │   - Library/include/joint_limits/joint_limits/joint_limiter_interface.hpp
2025-01-23T14:34:46.8315337Z  │ │   - Library/include/joint_limits/joint_limits/joint_limits.hpp
2025-01-23T14:34:46.8316296Z  │ │   - Library/include/joint_limits/joint_limits/joint_limits_rosparam.hpp
2025-01-23T14:34:46.8317206Z  │ │   - Library/include/joint_limits/joint_limits/joint_limits_urdf.hpp
2025-01-23T14:34:46.8317990Z  │ │   - Library/include/joint_limits/joint_limits/joint_saturation_limiter.hpp
2025-01-23T14:34:46.8318753Z  │ │   - Library/include/joint_limits/joint_limits/version.h
2025-01-23T14:34:46.8319211Z  │ │   - Library/lib/joint_limiter_interface.lib
2025-01-23T14:34:46.8319675Z  │ │   - Library/lib/joint_saturation_limiter.lib
2025-01-23T14:34:46.8320287Z  │ │   - Library/share/ament_index/resource_index/joint_limits__pluginlib__plugin/joint_limits
2025-01-23T14:34:46.8321001Z  │ │   - Library/share/ament_index/resource_index/package_run_dependencies/joint_limits
2025-01-23T14:34:46.8321589Z  │ │   - Library/share/ament_index/resource_index/packages/joint_limits
2025-01-23T14:34:46.8322175Z  │ │   - Library/share/ament_index/resource_index/parent_prefix_path/joint_limits
2025-01-23T14:34:46.8323077Z  │ │   - Library/share/joint_limits/cmake/ament_cmake_export_dependencies-extras.cmake
2025-01-23T14:34:46.8323736Z  │ │   - Library/share/joint_limits/cmake/ament_cmake_export_targets-extras.cmake
2025-01-23T14:34:46.8324313Z  │ │   - Library/share/joint_limits/cmake/export_joint_limitsExport-release.cmake
2025-01-23T14:34:46.8324873Z  │ │   - Library/share/joint_limits/cmake/export_joint_limitsExport.cmake
2025-01-23T14:34:46.8325374Z  │ │   - Library/share/joint_limits/cmake/joint_limitsConfig-version.cmake
2025-01-23T14:34:46.8325843Z  │ │   - Library/share/joint_limits/cmake/joint_limitsConfig.cmake
2025-01-23T14:34:46.8326304Z  │ │   - Library/share/joint_limits/environment/ament_prefix_path.bat
2025-01-23T14:34:46.8326765Z  │ │   - Library/share/joint_limits/environment/ament_prefix_path.dsv
2025-01-23T14:34:46.8327176Z  │ │   - Library/share/joint_limits/environment/path.bat
2025-01-23T14:34:46.8327555Z  │ │   - Library/share/joint_limits/environment/path.dsv
2025-01-23T14:34:46.8327921Z  │ │   - Library/share/joint_limits/joint_limiters.xml
2025-01-23T14:34:46.8328279Z  │ │   - Library/share/joint_limits/local_setup.bat
2025-01-23T14:34:46.8329327Z  │ │   - Library/share/joint_limits/local_setup.dsv
2025-01-23T14:34:46.8329683Z  │ │   - Library/share/joint_limits/package.dsv
2025-01-23T14:34:46.8330010Z  │ │   - Library/share/joint_limits/package.xml

The files:

2025-01-23T14:34:46.8308735Z  │ │   - Library/cmake/console_bridge-config-version.cmake
2025-01-23T14:34:46.8309498Z  │ │   - Library/cmake/console_bridge-config.cmake
2025-01-23T14:34:46.8310448Z  │ │   - Library/cmake/console_bridge-targets-release.cmake
2025-01-23T14:34:46.8311268Z  │ │   - Library/cmake/console_bridge-targets.cmake
2025-01-23T14:34:46.8312012Z  │ │   - Library/cmake/urdfdom_headers-config-version.cmake
2025-01-23T14:34:46.8312760Z  │ │   - Library/cmake/urdfdom_headers-config.cmake
2025-01-23T14:34:46.8313474Z  │ │   - Library/cmake/urdfdom_headersExport.cmake

are the problematic ones, as they are not installed by CMake (see the earliers lines that start with -- Installing:), but are instead part of urdfdom-headers, urdfdom and console_bridge packages.

In some local builds, I noticed the same problem with other files:

 │ │   - Library/CMake/yamlConfig.cmake
 │ │   - Library/CMake/yamlConfigVersion.cmake
 │ │   - Library/CMake/yamlTargets-release.cmake
 │ │   - Library/CMake/yamlTargets.cmake

In some builds (for example in https://github.com/traversaro/rattler-build-clobber-bomb-mre/actions/runs/12931317141/job/36064883960) no spurious files are included.

traversaro avatar Jan 23 '25 14:01 traversaro

A side-effect of this bug is that as the files spuriously included are actually part of the dependencies, as soon as the package is installed, this results in the problematic files being clobbered, and if you are building a deep enough graph of packages, you start seeing an "explosion" of clobbered files, see for example:

2025-01-22T00:08:18.6843107Z  │ │   - Library/CMake/console_bridge-config-version.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6844348Z  │ │   - Library/CMake/console_bridge-config-version.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6845562Z  │ │   - Library/CMake/console_bridge-config.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6847233Z  │ │   - Library/CMake/console_bridge-config.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6848450Z  │ │   - Library/CMake/console_bridge-targets-release.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6850263Z  │ │   - Library/CMake/console_bridge-targets-release.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6851758Z  │ │   - Library/CMake/console_bridge-targets.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6853317Z  │ │   - Library/CMake/console_bridge-targets.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6854479Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-moveit-resources-prbt-ikfast-manipulator-plugin
2025-01-22T00:08:18.6855924Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-01-22T00:08:18.6857379Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-pilz-industrial-motion-planner-testutils
2025-01-22T00:08:18.6859199Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-rviz-common
2025-01-22T00:08:18.6860522Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-moveit-resources-prbt-ikfast-manipulator-plugin
2025-01-22T00:08:18.6861649Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-01-22T00:08:18.6863388Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-pilz-industrial-motion-planner-testutils
2025-01-22T00:08:18.6864494Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-rviz-common
2025-01-22T00:08:18.6865888Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-moveit-resources-prbt-ikfast-manipulator-plugin
2025-01-22T00:08:18.6867355Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-01-22T00:08:18.6868699Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-pilz-industrial-motion-planner-testutils
2025-01-22T00:08:18.6869943Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-rviz-common

from https://github.com/RoboStack/ros-jazzy/pull/16 .

traversaro avatar Jan 23 '25 14:01 traversaro

I believe we haven't seen this issue in a relatively long while. Am I correct? If so, I think we can close it. We improved the "clobber" situation quite a bit to the point that I hope the most glaring issues are resolved.

wolfv avatar May 02 '25 16:05 wolfv

By looking at a recent PR build of Windows packages of robostack-jazzy with rattler-build 0.40.0 in https://github.com/RoboStack/ros-jazzy/actions/runs/14568672537/job/40861901679, it seems that the problem is still present, as there are outputs like:

2025-04-21T08:20:36.3907138Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-kdl-parser
2025-04-21T08:20:36.3907964Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-rviz2
2025-04-21T08:20:36.3908959Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-kdl-parser
2025-04-21T08:20:36.3909585Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-rviz2
2025-04-21T08:20:36.3910294Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-kdl-parser
2025-04-21T08:20:36.3910971Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-rviz2

and

2025-04-21T07:21:57.9208475Z  │ │ Files in package:
2025-04-21T07:21:57.9212526Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-kdl-parser__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-04-21T07:21:57.9213731Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-rviz2
2025-04-21T07:21:57.9215128Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-kdl-parser__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-04-21T07:21:57.9216154Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-rviz2
2025-04-21T07:21:57.9217224Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-kdl-parser__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-04-21T07:21:57.9218294Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-rviz2

if it ok for you, I would wait for a full Windows PR build of a robostack project on Windows without occurences before closing the issue. It could make sense to upgrade https://github.com/RoboStack/ros-jazzy/pull/54 to rattler-build 0.41.0 and monitor if the problem occurs again.

traversaro avatar May 02 '25 17:05 traversaro

Any "clobber" files should never be included in a package - so that is definitely a bug! We have to get on it

wolfv avatar May 03 '25 08:05 wolfv

Basically, when packaging "new" files we compare against the contents of the conda-meta folder. The JSON files in there should record all files that are installed in the environment.

It looks like, for some reason, some files or packages are not properly recorded. I have added a debug printout to the build script of your reproducer here: https://github.com/wolfv/rattler-build-clobber-bomb-mre

Let's see if that leads to something.

wolfv avatar May 28 '25 08:05 wolfv

Hmmm, ok, maybe this is a lead. I compared the urdfdom_headers package from conda-forge vs. the one from ros-humble.

One has the CMake as uppercase, and one as lowercase cmake folder. Windows file system is case-insensitive. So these paths are the same, but in Rust they are not. So I think what happens is something like this:

  • when we install ros-humble-urdfdom-headers first, we create the Library/CMake folder with uppercase
  • later, another package installs a cmake file to Library/cmake (equivalent path, different casing). On the filesystem this is installed to Library/CMake. We don't record the proper casing in the PrefixData record

When comparing filesystem vs. recorded files, we discover the old file as newly installed because it ended up in an upper case folder.

We should use case-insensitive comparison of files to prevent this.

wolfv avatar May 28 '25 08:05 wolfv

Good catch!

traversaro avatar May 28 '25 08:05 traversaro