gz-sim
gz-sim copied to clipboard
ign-gazebo3/main homebrew INTEGRATION test failures (segfaults)
Environment
- OS Version: macOS
- Source or binary build? 3465bbe
Description
Three new test failures over the last couple days: https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/113/testReport/
The follow two started today:
INTEGRATION_each_new_removed (fails with exception)
Start 67: INTEGRATION_each_new_removed67: Test command: /Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/build/bin/INTEGRATION_each_new_removed "--gtest_output=xml:/Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/build/test_results/INTEGRATION_each_new_removed.xml" 67: Test timeout computed to be: 240 67: Running main() from /Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/ign-gazebo/test/gtest/src/gtest_main.cc 67: [==========] Running 1 test from 1 test suite. 67: [----------] Global test environment set-up. 67: [----------] 1 test from EachNewRemovedFixture 67: [ RUN ] EachNewRemovedFixture.EachNewEachRemovedInSystem 67/150 Test #67: INTEGRATION_each_new_removed .........................Subprocess aborted***Exception: 1.08 sec
INTEGRATION_performer_detector (fails with segfault)
103: [ RUN ] PerformerDetectorTest.HandlesRemovedParentModel 103: [Err] [PerformerDetector.cc:104] Performer detector[detector1] has an emptywith an associated of [no_value]. This will be ignored. 103: [Err] [PerformerDetector.cc:98] Performer detector[detector1] has an empty with an associated of [no_key]. This will be ignored. 103/150 Test #103: INTEGRATION_performer_detector .......................***Exception: SegFault 1.85 sec
There are a two recent changes to the ign-gazebo3
branch, though I'm not sure if either of them is responsible for the recent failures:
- https://github.com/ignitionrobotics/ign-gazebo/pull/603
- https://github.com/ignitionrobotics/ign-gazebo/pull/574
This third failure started two days ago:
INTEGRATION_breadcrumbs (fails with segfault)
55: [ RUN ] BreadcrumbsTest.FuelDeploy 55: [Msg] Loading SDF world file[/Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/ign-gazebo/test/worlds/breadcrumbs.sdf]. 55: Warning [Model.cc:212] Non-unique names detected in XML children of model with name[vehicle_blue]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [2251689575469537287] / [ign_gazebo_components.World]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13994732549916512682] / [ign_gazebo_components.Name]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3630648173860223239] / [ign_gazebo_components.PhysicsEnginePlugin]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17459188283658606303] / [ign_gazebo_components.RenderEngineServerPlugin]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8705992680619689917] / [ign_gazebo_components.RenderEngineGuiPlugin]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8753193699724811771] / [ign_gazebo_components.Wind]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [12173050716021724529] / [ign_gazebo_components.WorldLinearVelocity]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [15943768124495574352] / [ign_gazebo_components.WorldLinearVelocitySeed]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17100615127981600159] / [ign_gazebo_components.Scene]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17605309075052480649] / [ign_gazebo_components.Atmosphere]. 55: [Dbg] [LevelManager.cc:195] Reading performer info 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [4425034272867445569] / [ign_gazebo_components.Performer]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11311330372155309806] / [ign_gazebo_components.PerformerLevels]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17121648710877364109] / [ign_gazebo_components.Geometry]. 55: [Msg] Created performer [3 / perf1] 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8064491505919932473] / [ign_gazebo_components.Level]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames]. 55: [Dbg] [Physics.cc:600] Loaded [ignition::physics::dartsim::Plugin] from library [/usr/local/Cellar/ignition-physics2/2.3.0_2/lib/ign-physics-2/engine-plugins/libignition-physics-dartsim-plugin.dylib] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Physics] for entity [1] 55: [Msg] Create service on [/world/breadcrumbs/create] 55: [Msg] Remove service on [/world/breadcrumbs/remove] 55: [Msg] Pose service on [/world/breadcrumbs/set_pose] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::UserCommands] for entity [1] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1] 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [4981278897826323946] / [ign_gazebo_components.WorldSdf]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [6687176221774458630] / [ign_gazebo_components.Model]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [6612894081701502240] / [ign_gazebo_components.Pose]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8546580419506082455] / [ign_gazebo_components.Static]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9712747055438129860] / [ign_gazebo_components.WindMode]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [5661073481138181711] / [ign_gazebo_components.SelfCollide]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11683062252779233161] / [ign_gazebo_components.SourceFilePath]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [5081358965268446661] / [ign_gazebo_components.Link]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8112400427272910195] / [ign_gazebo_components.Inertial]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [16454635107327670381] / [ign_gazebo_components.Visual]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13011964647677164955] / [ign_gazebo_components.CastShadows]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13440282432131634483] / [ign_gazebo_components.Transparency]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9853217982010720764] / [ign_gazebo_components.Material]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17938588655714334139] / [ign_gazebo_components.Collision]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9225962031573086509] / [ign_gazebo_components.CollisionElement]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [10522242218202596205] / [ign_gazebo_components.CanonicalLink]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [360894313363162387] / [ign_gazebo_components.Joint]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [4514298940308304267] / [ign_gazebo_components.JointType]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [16752981946841796574] / [ign_gazebo_components.JointAxis]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [6791213906290467105] / [ign_gazebo_components.ThreadPitch]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [14863055383328686534] / [ign_gazebo_components.ParentLinkName]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3888602368311486678] / [ign_gazebo_components.ChildLinkName]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11536476718181283925] / [ign_gazebo_components.ModelSdf]. 55: [Msg] DiffDrive subscribing to twist messages on [/model/vehicle_blue/cmd_vel] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::DiffDrive] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B1/deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B2/deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/fuel_deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B1_perf/deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B1_perf_large_volume/deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/no_rename_deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Msg] Breadcrumbs subscribing to deploy messages on [/rename_deploy] 55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5] 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3866641186784191835] / [ign_gazebo_components.Light]. 55: [Msg] Loaded level [4] 55: [Msg] Serving world controls on [/world/breadcrumbs/control] and [/world/breadcrumbs/playback/control] 55: [Msg] Serving GUI information on [/world/breadcrumbs/gui/info] 55: [Msg] World [breadcrumbs] initialized with [1ms] physics profile. 55: [Msg] Serving world SDF generation service on [/world/breadcrumbs/generate_world_sdf] 55: [Msg] Serving world names on [/gazebo/worlds] 55: [Msg] Resource path add service on [/gazebo/resource_paths/add]. 55: [Msg] Resource path get service on [/gazebo/resource_paths/get]. 55: [Msg] Resource paths published on [/gazebo/resource_paths]. 55: [Msg] Found no publishers on /stats, adding root stats topic 55: [Msg] Found no publishers on /clock, adding root clock topic 55: [Dbg] [SimulationRunner.cc:422] Creating PostUpdate worker threads: 4 55: [Dbg] [SimulationRunner.cc:435] Creating postupdate worker thread (0) 55: [Dbg] [SimulationRunner.cc:435] Creating postupdate worker thread (1) 55: [Dbg] [SimulationRunner.cc:435] Creating postupdate worker thread (2) 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [14423792621074650228] / [ign_gazebo_components.JointVelocityCmd]. 55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9436740692853731264] / [ign_gazebo_components.JointPosition]. 55: [Msg] Serving scene information on [/world/breadcrumbs/scene/info] 55: [Msg] Serving graph information on [/world/breadcrumbs/scene/graph] 55: [Msg] Serving full state on [/world/breadcrumbs/state] 55: [Msg] Serving full state (async) on [/world/breadcrumbs/state_async] 55: [Msg] Publishing scene information on [/world/breadcrumbs/scene/info] 55: [Msg] Publishing entity deletions on [/world/breadcrumbs/scene/deletion] 55: [Msg] Publishing state changes on [/world/breadcrumbs/state] 55: [Msg] Publishing pose messages on [/world/breadcrumbs/pose/info] 55: [Msg] Publishing dynamic pose messages on [/world/breadcrumbs/dynamic_pose/info] 55: [Msg] Deploying B2_0 at -2.2 2 5.525 -0 0 0 55: [Msg] Deploying B2_1 at -2.2 2 5.525 -0 0 0 55: [Msg] Deploying B2_2 at -2.2 2 5.525 -0 0 0 55: [Msg] Deploying B2_3 at -2.2 2 5.525 -0 0 -0[1 55/150 Test #55: INTEGRATION_breadcrumbs ..............................***Exception: SegFault 4.58 sec
Correlates to https://github.com/ignitionrobotics/ign-gazebo/pull/609, but I'm just speculating that they may be related.
I've detected some more today:
The following tests FAILED:
61 - INTEGRATION_breadcrumbs (SEGFAULT)
75 - INTEGRATION_each_new_removed (SEGFAULT)
143 - INTEGRATION_user_commands (SEGFAULT)
Current list of segfaults observed in main
branch:
The following tests FAILED:
61 - INTEGRATION_breadcrumbs (SEGFAULT)
73 - INTEGRATION_diff_drive_system (SEGFAULT)
103 - INTEGRATION_level_manager (SEGFAULT)
141 - INTEGRATION_scene_broadcaster_system (SEGFAULT)
151 - INTEGRATION_user_commands (SEGFAULT)
https://github.com/ignitionrobotics/ign-gazebo/pull/807 has a potential solution
More segfaulting tests from yesterday
59 - INTEGRATION_ackermann_steering_system (SEGFAULT)
119 - INTEGRATION_logical_audio_sensor_plugin (SEGFAULT)
159 - INTEGRATION_user_commands (SEGFAULT)
https://build.osrfoundation.org/job/ignition_gazebo-ci-main-homebrew-amd64/79
Grepping our last 30 days jobs at https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/ we find the following Integration tests flakily segfaulting on MacOS:
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|221|INTEGRATION_pose_publisher_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|219|INTEGRATION_diff_drive_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|219|INTEGRATION_velocity_control_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|212|INTEGRATION_pose_publisher_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|212|INTEGRATION_touch_plugin
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|211|INTEGRATION_altimeter_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|209|INTEGRATION_level_manager
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|208|INTEGRATION_diff_drive_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|208|INTEGRATION_level_manager
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|207|INTEGRATION_network_handshake
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|206|INTEGRATION_pose_publisher_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|205|INTEGRATION_logical_audio_sensor_plugin
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|205|INTEGRATION_performer_detector
I'm opening a PR to disable them on MacOS for now.