To meta-trace or not to meta-trace

[gomoripeti]

Nomral tracing in Erlang can be controlled in two parts:

• erlang:trace/3 controls which processes to trace (all/none/new/existing/Pid) and global flags (like timestamp format, arity, tracer module, and which events to trace like call/proc/gc)
• erlang:trace_pattern/3 controls MFA+match-spec

In XProf the former is controlled by the singleton xprof_core_tracer gen_server (for example to pause all tracing at once manually or automatically in case of overload) and the later by xprof_core_trace_handler per each MFA.

The problem is that we would like to set some of the global flags on a per MFA granularity. Eg use arity when we don't capture long calls. (This is worked around currently with some match-spec trick). With the dawn of our tracing NIF we would also like to set different state for the tracer module per MFA.

Meta-tracing traces all processes and does not care about the process trace flags set by erlang:trace/3, the trace flags are instead fixed to [call, timestamp]. The match specification function return_trace() works with meta-trace and sends its trace message to the same tracer.

Meta-tracing would allow to set a different tracer module with a different state per MFA. (This would be ideal for us). (It does not allow to use the arity flag, so this is out of question to use in the legacy solution without a tracer module.)

On the other hand meta-tracing would always trace all the processes. (Currently we only support tracing all the processes from the GUI anyway but tracing only 1 process from the shell works and this is used by some tests and profiling.) If we would like to implement something else than all-the-processes we need to implement it in the tracer module (enable function is a good candidate) and we cannot use the VM's tracing infrastructure.

More importantly the role of XProf processes would change. The main xprof_core_tracer would loose its global control of pausing all tracing at once. (It could do this by calling erlang:trace_pattern/3 for each monitored MFA.) It would also be possible to send trace messages directly to the per MFA gen_servers instead of the singleton proxy process which would have main role remaining to control starting/stopping monitoring MFAs. Overload protection would be moved to the per MFA servers too. This would be a somewhat big architectural change (To be investigated if removing the proxy is good or bad from safety perspective.)

An alternative is to pass a map as the global tracer module state which has one entry per MFA. Unfortunately this needs to be updated whenever there is a change in any of the MFA states. Also needs to be done by the singleton xprof_core_tracer while per MFA state is the responsibility of the per MFA gen_servers.

Another similar tracing is call-counting which is also valid for all processes and controlled only by erlang:trace_pattern/3.