RFC: Scheduled traces

I'm exploring the idea to allow users to add scheduled traces to this project.

A scheduled trace is a trace that runs at a specific time in a similar way a Cronjob works in kubernetes.

It will leave the traces it creates in the system so that the user can inspect them using kubectl trace logs (#4).

Since most of bpftrace commands need the user to stop the profiling by sending a signal this will also need to support a way to have a configurable timeout so that the signal is automatically sent.

The user will be able to send the signal by attaching to the current job.

I really like this idea, this would be good for getting samples emitted on a regular basis to help with long-term patttern analysis.

Perhaps the current spec for the trace could be refactored so that it could be shared by a cron job, as well as an "on-demand" trace job?

@dalehamel I think that what we want is probably something like that, however I would like to find a way to save the output somewhere.

We also have to address the fact that maps and things like that need a sigint to plot their results, so probably that will need some kind of timer to act as a trigger.

find a way to save the output somewhere.

Logging to stdout already grants a lot of compatibilty here, as things like fluentd can pick up the logs.

Do you mean that the output should be "tee'd" to a file? If so, writing to a path on a persistent volume is one way to do it. bpftrace does both streaming output and reports of statistics/summaries, which have somewhat different characteristics. I think streaming output is a more natural fit for something like fluentd, and the reports benefit from a more traditional block-file storage style pattern.

some kind of timer to act as a trigger

Ah yeah that's a good point, and that adds the complexity of storing/configuring a value for how long to sample for before summarizing.

I don't think kubernetes has anything built-in to send anything other than SIGTERM and SIGKILL from the kubelet.

A quick-n-dirty thing that jumps to mind is to add another container to the pod with CAP_SYS_KILL (i think that's it, the one that let's you signal anyways), to signal the bpftrace process. The same container image could be used, just alter the entrypoint to a simple shell script that sleeps and then sends the kill command. In practice, this might get more complex as you have to account for either container failing to start or them not starting at the same time, or the bpftrace process dying, but this might not be a practical issue and I don't think there are that many states to cover.

Another approach might be to add a minimal init process and spawn a second process from the same container, but the pod-based approach "feels more kubernetes".

I think we should bump this again, it's similar to #80

The kubernetes CronJob is probably the way to do this, and i think that having an activeDeadlineSeconds set for that is absolutely essential.

I don't think this would be complicated to implement, the job code is already there and would just need some refactoring.

I could see a use in this for say, spawning trace jobs once an hour to collect sampling data for performance analysis in production, or even just a 'daily performence test'.

#88 should help with this as the output can be easily collected through stdout using preStop hooks along with kubernetes builtin deadlines