tracee
tracee copied to clipboard
aquasecurity
Sorting stage get locked during sendEvents
Description
After performing a stress test in Tracee and dumping the goroutine trace, me and @geyslan realized that around 1,097 goroutines were in the sync.Mutex.Lock state. All goroutines were created by the following logic: https://github.com/geyslan/tracee/blob/4d5cbbfaf1cb74d1c3a38bbac4d24072d7541d22/pkg/events/sorting/sorting.go#L189
case <-ticker.C:
sorter.updateSavedTimestamps()
if len(sorter.extractionSavedTimestamps) > sorter.intervalsAmountThresholdForDelay {
extractionTimestamp := sorter.extractionSavedTimestamps[0]
sorter.extractionSavedTimestamps = sorter.extractionSavedTimestamps[1:]
go sorter.sendEvents(met, out, extractionTimestamp)
}
The following trace shows the goroutines stucked in the locks:
Goroutine stack trace
goroutine 1 [chan receive, 1 minutes]:
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Run(0xc0000f9208, {0x31d2e78, 0xc000242480})
/tmp/tracee-master/pkg/ebpf/tracee.go:1435 +0x6da
github.com/aquasecurity/tracee/pkg/cmd.Runner.Run({{{0xc0022b1d40, 0xd, 0xd}, 0xc0006fcb60, 0xc000369640, 0xc001cf80f0, {0x0, 0x0}, {0x0, 0x2ab0, ...}, ...}, ...}, ...)
/tmp/tracee-master/pkg/cmd/tracee.go:112 +0x585
github.com/aquasecurity/tracee/cmd/tracee/cmd.init.func14(0x3e04d40, {0xc000142010?, 0x18?, 0x19?})
/tmp/tracee-master/cmd/tracee/cmd/root.go:64 +0x27a
github.com/spf13/cobra.(*Command).execute(0x3e04d40, {0xc000142010, 0x18, 0x19})
/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:989 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0x3e04d40)
/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/aquasecurity/tracee/cmd/tracee/cmd.Execute()
/tmp/tracee-master/cmd/tracee/cmd/root.go:381 +0x2a
main.main()
/tmp/tracee-master/cmd/tracee/main.go:11 +0x13
goroutine 88 [chan receive, 1 minutes]:
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).WaitForPipeline(0xc0000f9208, {0xc0019a7480?, 0xc000242480?, 0xc0006c8480?})
/tmp/tracee-master/pkg/ebpf/events_pipeline.go:792 +0x5c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).handleEvents(0xc0000f9208, {0x31d2e78, 0xc000242480}, 0xc00364cf00)
/tmp/tracee-master/pkg/ebpf/events_pipeline.go:133 +0x6cd
created by github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Run in goroutine 1
/tmp/tracee-master/pkg/ebpf/tracee.go:1409 +0x459
goroutine 100 [select]:
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start(0xc00052e8c0, 0xc0000f8f08, 0xc0000b2600, 0xc0000b2720, {0x31d2e78, 0xc000242480}, 0x23206f65726f7453?)
/tmp/tracee-master/pkg/events/sorting/sorting.go:177 +0x17f
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).StartPipeline in goroutine 88
/tmp/tracee-master/pkg/events/sorting/sorting.go:158 +0xf8
....
goroutine 6452 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x547785ce31?)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc00052e8d8)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:90
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc00052e8c0, 0xc0000f8f08, 0xc0000b2720, 0x182cbbf7ad06aeef)
/tmp/tracee-master/pkg/events/sorting/sorting.go:214 +0x74
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 100
/tmp/tracee-master/pkg/events/sorting/sorting.go:189 +0x2c5
goroutine 6350 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc00052e8d8)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:90
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc00052e8c0, 0xc0000f8f08, 0xc0000b2720, 0x182cbbf7b1891547)
/tmp/tracee-master/pkg/events/sorting/sorting.go:214 +0x74
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 100
/tmp/tracee-master/pkg/events/sorting/sorting.go:189 +0x2c5
goroutine 6364 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0xe4a?, 0x44?, 0x4b4400004b21?)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc00052e8d8)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
/home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:90
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc00052e8c0, 0xc0000f8f08, 0xc0000b2720, 0x182cbbf7b3e022f0)
/tmp/tracee-master/pkg/events/sorting/sorting.go:214 +0x74
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 100
/tmp/tracee-master/pkg/events/sorting/sorting.go:189 +0x2c5
It is possible to see that a parameter was added to sendEvents because @geyslan and I were using this Tracee commit to perform the tests.
Output of tracee version:
(paste your output here)
Output of uname -a:
(paste your output here)
Additional details
This is the pipeline summary of the running that @rscampos mentioned.
Field Average Max Max In File Last Diff Diff Since File
-------------------------------------------------------------------------------------------------------------------------------------------------------
go_goroutines 1_081 2_098 2025-03-14-17-37-43.prom 2_098
go_memstats_heap_inuse_bytes 944_145_808 1_519_616_000 2025-03-14-17-37-43.prom 1_519_616_000
process_cpu_seconds_total 89.01 126.31 2025-03-14-17-37-43.prom 126.31
process_max_fds 1_048_576 1_048_576 2025-03-14-17-35-54.prom 1_048_576
tracee_ebpf_decode_in_total 1_181_481 1_802_537 2025-03-14-17-37-43.prom 1_802_537
tracee_ebpf_decode_in_last 0 0 - 17:37:43.509072
tracee_ebpf_decode_out_total 1_147_139 1_753_495 2025-03-14-17-37-43.prom 1_753_495
tracee_ebpf_decode_out_last 0 0 - 17:37:43.509089
tracee_ebpf_decode_filtered_total 34_341 49_042 2025-03-14-17-37-35.prom 49_042
tracee_ebpf_decode_filtered_last 0 0 - 17:37:31.084305
tracee_ebpf_queue_in_total 0 0 - -
tracee_ebpf_queue_in_last 0 0 - -
tracee_ebpf_queue_out_total 0 0 - -
tracee_ebpf_queue_out_last 0 0 - -
tracee_ebpf_queue_filtered_total 0 0 - -
tracee_ebpf_queue_filtered_last 0 0 - -
tracee_ebpf_sort_in_total 1_147_120 1_753_495 2025-03-14-17-37-43.prom 1_753_495
tracee_ebpf_sort_in_last 0 0 - 17:37:43.509092
tracee_ebpf_sort_out_total 346_971 389_407 2025-03-14-17-36-20.prom 389_407 1364088 2025-03-14-17-37-43.prom
tracee_ebpf_sort_out_last 0 0 - 17:36:15.626680
tracee_ebpf_sort_filtered_total 0 0 - -
tracee_ebpf_sort_filtered_last 0 0 - -
tracee_ebpf_process_in_total 345_016 387_359 2025-03-14-17-36-20.prom 387_359
tracee_ebpf_process_in_last 0 0 - 17:36:15.626067
tracee_ebpf_process_out_total 345_015 387_358 2025-03-14-17-36-20.prom 387_358 1 2025-03-14-17-37-43.prom
tracee_ebpf_process_out_last 0 0 - 17:36:15.626067
tracee_ebpf_process_filtered_total 0 0 - -
tracee_ebpf_process_filtered_last 0 0 - -
tracee_ebpf_enrich_in_total 0 0 - -
tracee_ebpf_enrich_in_last 0 0 - -
tracee_ebpf_enrich_out_total 0 0 - -
tracee_ebpf_enrich_out_last 0 0 - -
tracee_ebpf_enrich_filtered_total 0 0 - -
tracee_ebpf_enrich_filtered_last 0 0 - -
tracee_ebpf_derive_in_total 327_855 367_388 2025-03-14-17-36-20.prom 367_388
tracee_ebpf_derive_in_last 0 0 - 17:36:15.584949
tracee_ebpf_derive_out_total 327_854 367_387 2025-03-14-17-36-20.prom 367_387 1 2025-03-14-17-37-43.prom
tracee_ebpf_derive_out_last 0 0 - 17:36:15.584948
tracee_ebpf_derive_filtered_total 0 0 - -
tracee_ebpf_derive_filtered_last 0 0 - -
tracee_ebpf_engine_in_total 314_738 352_053 2025-03-14-17-36-20.prom 352_053
tracee_ebpf_engine_in_last 0 0 - 17:36:15.572451
tracee_ebpf_engine_out_total 314_738 352_053 2025-03-14-17-36-20.prom 352_053
tracee_ebpf_engine_out_last 0 0 - 17:36:15.572452
tracee_ebpf_engine_filtered_total 0 0 - -
tracee_ebpf_engine_filtered_last 0 0 - -
tracee_ebpf_sink_in_total 314_738 352_053 2025-03-14-17-36-20.prom 352_053
tracee_ebpf_sink_in_last 0 0 - 17:36:15.574645
tracee_ebpf_sink_out_total 81_440 91_521 2025-03-14-17-36-20.prom 91_521
tracee_ebpf_sink_out_last 0 0 - 17:36:15.574641
tracee_ebpf_sink_filtered_total 233_298 260_532 2025-03-14-17-36-20.prom 260_532
tracee_ebpf_sink_filtered_last 0 0 - 17:36:15.574645
-------------------------------------------------------------------------------------------------------------------------------------------------------
Stressed locally two times, one printing to none and the other to a json file. None reproduced the deadlock.
The run which generated 351_999_998 events to sink stage:
dist/tracee --metrics --pyroscope --pprof -s tree=3868004 -e process_execute_failed -s comm=bash,taskset,mv -e sched_process_exec -e sched_process_fork -e sched_process_exit -e sig_ransomware -o none --no-containers --perf-buffer-size 2048 --blob-perf-buffer-size 2048 --pipeline-channel-size 20000 -o option:sort-events -o option:exec-env -o option:exec-hash=dev-inode -o option:parse-arguments --log debug --log file:tracee.log
Pipeline summary
Field Average Max Max In File Last Diff Diff Since File
-------------------------------------------------------------------------------------------------------------------------------------------------------
go_goroutines 353 356 2025-03-14-21-20-45.prom 353
go_memstats_heap_inuse_bytes 105_477_313 113_451_008 2025-03-14-23-29-57.prom 100_417_536
process_cpu_seconds_total 35651.18 71171.17 2025-03-14-23-43-03.prom 71171.17
process_max_fds 524_287 524_287 2025-03-14-20-17-14.prom 524_287
tracee_ebpf_decode_in_total 152_259_803 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_decode_in_last 0 0 - 02:42:55.313418
tracee_ebpf_decode_out_total 152_259_803 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_decode_out_last 0 0 - 02:42:55.313418
tracee_ebpf_decode_filtered_total 0 0 - -
tracee_ebpf_decode_filtered_last 0 0 - -
tracee_ebpf_queue_in_total 0 0 - -
tracee_ebpf_queue_in_last 0 0 - -
tracee_ebpf_queue_out_total 0 0 - -
tracee_ebpf_queue_out_last 0 0 - -
tracee_ebpf_queue_filtered_total 0 0 - -
tracee_ebpf_queue_filtered_last 0 0 - -
tracee_ebpf_sort_in_total 152_259_802 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_sort_in_last 0 0 - 02:42:55.313419
tracee_ebpf_sort_out_total 152_255_590 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_sort_out_last 0 0 - 02:42:55.505038
tracee_ebpf_sort_filtered_total 0 0 - -
tracee_ebpf_sort_filtered_last 0 0 - -
tracee_ebpf_process_in_total 152_255_548 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_process_in_last 0 0 - 02:42:55.505072
tracee_ebpf_process_out_total 152_255_547 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_process_out_last 0 0 - 02:42:55.505072
tracee_ebpf_process_filtered_total 0 0 - -
tracee_ebpf_process_filtered_last 0 0 - -
tracee_ebpf_enrich_in_total 0 0 - -
tracee_ebpf_enrich_in_last 0 0 - -
tracee_ebpf_enrich_out_total 0 0 - -
tracee_ebpf_enrich_out_last 0 0 - -
tracee_ebpf_enrich_filtered_total 0 0 - -
tracee_ebpf_enrich_filtered_last 0 0 - -
tracee_ebpf_derive_in_total 152_255_503 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_derive_in_last 0 0 - 02:42:55.505094
tracee_ebpf_derive_out_total 152_255_503 304_000_000 2025-03-14-23-42-58.prom 304_000_000
tracee_ebpf_derive_out_last 0 0 - 02:42:55.505094
tracee_ebpf_derive_filtered_total 0 0 - -
tracee_ebpf_derive_filtered_last 0 0 - -
tracee_ebpf_engine_in_total 176_295_836 351_999_998 2025-03-14-23-42-58.prom 351_999_998
tracee_ebpf_engine_in_last 0 0 - 02:42:55.505232
tracee_ebpf_engine_out_total 176_295_835 351_999_998 2025-03-14-23-42-58.prom 351_999_998
tracee_ebpf_engine_out_last 0 0 - 02:42:55.505232
tracee_ebpf_engine_filtered_total 0 0 - -
tracee_ebpf_engine_filtered_last 0 0 - -
tracee_ebpf_sink_in_total 176_295_832 351_999_998 2025-03-14-23-42-58.prom 351_999_998
tracee_ebpf_sink_in_last 0 0 - 02:42:55.505233
tracee_ebpf_sink_out_total 84_141_188 167_999_998 2025-03-14-23-42-58.prom 167_999_998
tracee_ebpf_sink_out_last 0 0 - 02:42:55.505234
tracee_ebpf_sink_filtered_total 92_154_643 184_000_000 2025-03-14-23-42-58.prom 184_000_000
tracee_ebpf_sink_filtered_last 0 0 - 02:42:55.505160
-------------------------------------------------------------------------------------------------------------------------------------------------------
Going to run it now loading all signatures and making scope global to increase the odds.
With all signatures I was able to reproduced it.
deadlocks
From 270 stack dumps (a dump every 10s):
found in 151 dumps:
goroutine 4649332 [sync.Mutex.Lock, 2 minutes]:
internal/sync.runtime_SemacquireMutex(0xc00078a690?, 0x5?, 0xc019651738?)
/home/gg/.goenv/versions/1.24.0/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0005ac748)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/sync/mutex.go:46
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc0005ac730, 0xc0004c0308, 0xc000668310, 0x182cfa99cfc59ed2)
/home/gg/code/tracee/pkg/events/sorting/sorting.go:212 +0x5f
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 148
/home/gg/code/tracee/pkg/events/sorting/sorting.go:187 +0x2a5
found in 188 dumps:
goroutine 4887664 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0xc0000e6524?, 0x8?, 0x735e110565b3?)
/home/gg/.goenv/versions/1.24.0/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0005ac748)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/sync/mutex.go:46
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc0005ac730, 0xc0004c0308, 0xc000668310, 0x182cfaba72d4f355)
/home/gg/code/tracee/pkg/events/sorting/sorting.go:212 +0x5f
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 148
/home/gg/code/tracee/pkg/events/sorting/sorting.go:187 +0x2a5
found in 184 dumps:
goroutine 3539690 [sync.Mutex.Lock, 5 minutes]:
internal/sync.runtime_SemacquireMutex(0xc06d8a0540?, 0x34?, 0x735e0000001e?)
/home/gg/.goenv/versions/1.24.0/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0005ac748)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/sync/mutex.go:46
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc0005ac730, 0xc0004c0308, 0xc000668310, 0x182cfa717179086d)
/home/gg/code/tracee/pkg/events/sorting/sorting.go:212 +0x5f
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 148
/home/gg/code/tracee/pkg/events/sorting/sorting.go:187 +0x2a5
found in 124 dumps:
goroutine 4742771 [sync.Mutex.Lock, 1 minutes]:
internal/sync.runtime_SemacquireMutex(0xc00078a690?, 0x45?, 0xc29538a738?)
/home/gg/.goenv/versions/1.24.0/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0005ac748)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
/home/gg/.goenv/versions/1.24.0/src/sync/mutex.go:46
github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).sendEvents(0xc0005ac730, 0xc0004c0308, 0xc000668310, 0x182cfaa5291199d5)
/home/gg/code/tracee/pkg/events/sorting/sorting.go:212 +0x5f
created by github.com/aquasecurity/tracee/pkg/events/sorting.(*EventsChronologicalSorter).Start in goroutine 148
/home/gg/code/tracee/pkg/events/sorting/sorting.go:187 +0x2a5
Pipeline summary
Field Average Max Max In File Last Diff Diff Since File
-------------------------------------------------------------------------------------------------------------------------------------------------------
go_goroutines 23_569 48_863 2025-03-15-10-21-55.prom 48_863
go_memstats_heap_inuse_bytes 45_211_704_182 52_310_114_304 2025-03-15-10-16-41.prom 51_908_214_784
process_cpu_seconds_total 9992.45 13983.2 2025-03-15-10-21-55.prom 13983.2
process_max_fds 524_287 524_287 2025-03-15-09-36-18.prom 524_287
tracee_ebpf_decode_in_total 64_908_123 80_675_720 2025-03-15-10-21-55.prom 80_675_720
tracee_ebpf_decode_in_last 0 0 - 13:21:55.369987
tracee_ebpf_decode_out_total 64_394_344 79_907_806 2025-03-15-10-21-55.prom 79_907_806 -12 2025-03-15-10-21-55.prom
tracee_ebpf_decode_out_last 0 0 - 13:21:55.368595
tracee_ebpf_decode_filtered_total 513_780 767_926 2025-03-15-10-21-55.prom 767_926
tracee_ebpf_decode_filtered_last 0 0 - 13:21:55.092158
tracee_ebpf_queue_in_total 0 0 - -
tracee_ebpf_queue_in_last 0 0 - -
tracee_ebpf_queue_out_total 0 0 - -
tracee_ebpf_queue_out_last 0 0 - -
tracee_ebpf_queue_filtered_total 0 0 - -
tracee_ebpf_queue_filtered_last 0 0 - -
tracee_ebpf_sort_in_total 64_394_261 79_907_806 2025-03-15-10-21-55.prom 79_907_806
tracee_ebpf_sort_in_last 0 0 - 13:21:55.368075
tracee_ebpf_sort_out_total 38_142_102 51_216_173 2025-03-15-10-21-55.prom 51_216_173 28691633 2025-03-15-10-21-55.prom
tracee_ebpf_sort_out_last 0 0 - 13:21:55.360168
tracee_ebpf_sort_filtered_total 0 0 - -
tracee_ebpf_sort_filtered_last 0 0 - -
tracee_ebpf_process_in_total 38_140_093 51_214_193 2025-03-15-10-21-55.prom 51_214_193
tracee_ebpf_process_in_last 0 0 - 13:21:55.369240
tracee_ebpf_process_out_total 38_140_090 51_214_162 2025-03-15-10-21-55.prom 51_214_162 31 2025-03-15-10-21-55.prom
tracee_ebpf_process_out_last 0 0 - 13:21:55.368355
tracee_ebpf_process_filtered_total 0 0 - -
tracee_ebpf_process_filtered_last 0 0 - -
tracee_ebpf_enrich_in_total 0 0 - -
tracee_ebpf_enrich_in_last 0 0 - -
tracee_ebpf_enrich_out_total 0 0 - -
tracee_ebpf_enrich_out_last 0 0 - -
tracee_ebpf_enrich_filtered_total 0 0 - -
tracee_ebpf_enrich_filtered_last 0 0 - -
tracee_ebpf_derive_in_total 38_137_886 51_211_791 2025-03-15-10-21-55.prom 51_211_791
tracee_ebpf_derive_in_last 0 0 - 13:21:54.919896
tracee_ebpf_derive_out_total 38_137_887 51_211_790 2025-03-15-10-21-55.prom 51_211_790 1 2025-03-15-10-21-55.prom
tracee_ebpf_derive_out_last 0 0 - 13:21:54.911679
tracee_ebpf_derive_filtered_total 0 0 - -
tracee_ebpf_derive_filtered_last 0 0 - -
tracee_ebpf_engine_in_total 40_867_118 54_881_202 2025-03-15-10-21-55.prom 54_881_202
tracee_ebpf_engine_in_last 0 0 - 13:21:55.127400
tracee_ebpf_engine_out_total 40_867_118 54_881_202 2025-03-15-10-21-55.prom 54_881_202
tracee_ebpf_engine_out_last 0 0 - 13:21:55.127400
tracee_ebpf_engine_filtered_total 0 0 - -
tracee_ebpf_engine_filtered_last 0 0 - -
tracee_ebpf_sink_in_total 40_867_109 54_881_202 2025-03-15-10-21-55.prom 54_881_202
tracee_ebpf_sink_in_last 0 0 - 13:21:55.127411
tracee_ebpf_sink_out_total 26_499_082 35_599_110 2025-03-15-10-21-55.prom 35_599_110
tracee_ebpf_sink_out_last 0 0 - 13:21:55.127411
tracee_ebpf_sink_filtered_total 14_368_026 19_282_092 2025-03-15-10-21-55.prom 19_282_092
tracee_ebpf_sink_filtered_last 0 0 - 13:21:54.932471
-------------------------------------------------------------------------------------------------------------------------------------------------------
@rscampos @geyslan a deadlock was recently fixed in the engine stage: https://github.com/aquasecurity/tracee/pull/4803
If engine is deadlocked, sorting stage will be deadlocked as well, so I don't think the issue was in the sorting stage itself, but since this stage starts many go routines, there will be a lot of its routines showing in the stack trace
