drake Address sporadic kcov segmentation faults

What happened?

The following failure occured in nightly production:

/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/_bazel_ubuntu/ca8cca4382e64e0e68b3df6a056070b5/execroot/drake/bazel-out/k8-dbg/bin/solvers/mosek_solver_internal_test.runfiles/drake/tools/dynamic_analysis/kcov: line 65: 75308 Segmentation fault      (core dumped) kcov "--include-path=/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/src" --verify --python-parse=python3 --exclude-pattern=third_party "--replace-src-path=/proc/self/cwd:/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/src" "/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/_bazel_ubuntu/ca8cca4382e64e0e68b3df6a056070b5/execroot/drake/bazel-out/k8-dbg/testlogs/solvers/mosek_solver_internal_test/test.outputs/kcov" solvers/mosek_solver_internal_test
Could not create "/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/_bazel_ubuntu/ca8cca4382e64e0e68b3df6a056070b5/execroot/drake/bazel-out/k8-dbg/testlogs/solvers/mosek_solver_internal_test/test.outputs/outputs.zip": zip not found or failed

//solvers:mosek_solver_internal_test                                     FAILED in 3.0s

Links:

https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/116/
https://drake-cdash.csail.mit.edu/test/1343352056

Jan 30 '24 13:01 liangfok

Closing since this is first occurrence.

Jan 30 '24 13:01 liangfok

We had another incidence today -- albeit in a different test.

https://drake-jenkins.csail.mit.edu/view/Weekly%20Production/job/linux-jammy-clang-bazel-weekly-everything-coverage/21/

Mar 04 '24 17:03 SeanCurtis-TRI

Three more failures, all in different tests:

https://drake-jenkins.csail.mit.edu/view/Weekly%20Production/job/linux-jammy-clang-bazel-weekly-everything-coverage/25/
https://drake-jenkins.csail.mit.edu/view/Weekly%20Production/job/linux-jammy-clang-bazel-weekly-mosek-coverage/26/
https://drake-jenkins.csail.mit.edu/view/Weekly%20Production/job/linux-jammy-clang-bazel-weekly-snopt-coverage/24/console

Mar 25 '24 16:03 BetsyMcPhail

3/29 segfault in //multibody/contact_solvers:minimum_degree_ordering_test in https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/175/

Mar 29 '24 13:03 BetsyMcPhail

4/1 segfault in //math:hopf_coordinate_test: linux-jammy-gcc-bazel-nightly-coverage

Apr 01 '24 14:04 BetsyMcPhail

Reopening the issue per the buildcop playbook. I tried reproducing the segfault locally but wasn't able to do it with a handful of runs.

Apr 02 '24 00:04 xuchen-han

4/9 //multibody/fem:dirichlet_boundary_condition_test linux-jammy-clang-bazel-nightly-coverage

Apr 09 '24 15:04 BetsyMcPhail

4/15: //multibody/parsing:package_map_remote_test https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/192/

Apr 15 '24 18:04 BetsyMcPhail

5/20: //common:text_logging_test failed in linux-jammy-gcc-bazel-weekly-everything-coverage

May 20 '24 14:05 BetsyMcPhail

5/21 //common:dummy_value_test failed in linux-jammy-clang-bazel-nightly-coverage

May 21 '24 14:05 BetsyMcPhail

9/19 //common:extract_double_test failed in linux-jammy-clang-bazel-nightly-coverage

Sep 19 '24 13:09 williamjallen

9/24 //manipulation/kuka_iiwa:iiwa_status_receiver_test failed in https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-nightly-coverage/355/consoleFull

Sep 24 '24 13:09 williamjallen

11/19 - linux-jammy-clang-bazel-nightly-coverage failed with a segfault in //systems/primitives:vector_log_test

Nov 19 '24 15:11 BetsyMcPhail

1/3/25 - //common:hash_test failed in linux-jammy-clang-bazel-nightly-coverage

Jan 03 '25 14:01 BetsyMcPhail

1/30/25 - //multibody/fem:linear_constitutive_model_data_test failed in linux-jammy-gcc-bazel-nightly-coverage

Jan 30 '25 17:01 SeanCurtis-TRI

2/5 //geometry:geometry_set_test failed in https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-gcc-bazel-nightly-coverage/496

Feb 05 '25 18:02 BetsyMcPhail

4/14 //common/test_utilities:eigen_matrix_compare_test failed in https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/558/

Apr 14 '25 16:04 BetsyMcPhail

4/23 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-nightly-coverage/567/

Apr 23 '25 16:04 BetsyMcPhail

4/23 https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-gcc-bazel-nightly-coverage/577/

Apr 23 '25 16:04 BetsyMcPhail