tracee icon indicating copy to clipboard operation
tracee copied to clipboard

Running./tracee has a huge impact on the performance of the system call 'write'

Open clove928 opened this issue 2 years ago • 3 comments

Description

Run the following commands, respectively, and monitor the performance impact of system calls 1、./tracee 2、./tracee --signatures-dir (Specify an empty folder here) 3、./tracee-ebpf The time taken for system calls when tracee is not running is used as a benchmark,The impact of the above three scenarios on system calls (‘write’)is 1、21.93 --This means that after running tracee, the latency of the write system call is 21.93 times higher than without running tracee 2、1.86 3、1.86

Output of tracee version:

0.17.1

Output of uname -a:

4.18.0-372.26.1

Additional details

When running./tracee, whether or not default signatures are loaded has such a big impact on the system call ‘’write‘. Why?

clove928 avatar Sep 06 '23 14:09 clove928

This is indeed a very high latency, and I want to first understand how to reproduce these results. If I have to guess, I would say the reason for this latency is one of the signature events that depends on vfs_write under the hood. For example, the dropped_executable signature depends on the magic_write event which is generated by the bpf program attached to vfs_write. This program extracts the full file path on every call, which can be expensive.

Can you please share with us how you measure the latency for the write syscall? Which tool do you use to measure? What is the load you run that generates the syscalls? What is the file path being written (especially interesting if the path has many components)? Are the writes being performed in some loop? Is it disk io or memory io? Any other details you can share about the environment?

yanivagman avatar Sep 06 '23 19:09 yanivagman

@AlonZivony were you able to look into this?

itaysk avatar Oct 17 '23 12:10 itaysk

@AlonZivony were you able to look into this?

No, I am focusing on the process tree currently. I volunteered to fix it, but don't have the tools to check performance. As I said when I took the task, I need someone to help me with this task.

AlonZivony avatar Oct 18 '23 07:10 AlonZivony

This has been resolved in #3899

yanivagman avatar Apr 18 '24 07:04 yanivagman