filter_rules sometimes will casue troubles

[Nassue]

hello, i try use otter_span_mpdict_api to start a trace with some child spans, but if i use filter rules with {one_out_of, Num}, sometimes it will lose spans within a full trace. Maybe discard rules should be effective for a full trace?

[fholzhauser]

Hi,

That's tricky indeed. That particular condition has that handicap (it was already known when we implemented it), and I'm not sure if we can do anything about it. Perhaps we should add this to the description. Actually we made it for the request from our sponsor, and, as I can recall, the motivation behind it was to emit spans once in a while for "good" events that happen a lot, preferably not part of a complete trace. So indeed we probably don't want to use it for spans that are part of a larger trace.

Otter operates only on spans. It does not correlate them into a complete trace. Conceptually I don't think it should do it either. OpenTracing is a distributed tracing framework, where different nodes could send spans to the trace collector for the same trace, which then composes the complete trace from those.

If we really want to do such filtering on trace level, e.g. to "spare" the trace collector, and emitting all the spans from the nodes is not a concern (e.g. from network/system load point of view), then perhaps making a correlator proxy kind of thing could be an idea. That'd sit before the trace collector, get all the spans from all the nodes and forward only some to the collector, making sure that the complete trace stays intact. otter_srv (currently a unit test dependency of otter) on the receiving side, and otter for forwarding to the trace collector with some basic trace correlator (in ETS perhaps) in between could be a good start for such proxy.

I hope this clarifies a bit.

Regards, Ferenc

[Nassue]

Thank you very much for your reply，i will try to do something on collector or proxy.

[Nassue]

An idea, i can set a probability for calling otter_span_mpdict_api:start/1, then emit all the spans from the nodes. (o.o)

[fholzhauser]

Sorry, I don't really see how you'd do that ... could you elaborate ?

[Nassue]

Sorry, I am not good at English. I will show my codes to you:

%%%all functions are use "Multiple span process dictionary API"
trace_start(Name) ->
    case enable_trace(Name) of
        true ->
            case maybe_trace() of
                true ->
                    Span = otter_span_mpdict_api:start(Name),
                    trace_tag(Name, "trace_id", element(3, Span)),
                    Span;
                false ->
                    ignore
            end;
        false ->
            ignore
    end.

trace_start(_Name, undefined) ->
    ignore;
trace_start(Name, ParentSpan) ->
    case enable_trace(Name) of
        true ->
            Span = otter_span_mpdict_api:start(Name, ParentSpan),
            trace_tag(Name, "trace_id", element(3, Span)),
            Span;
        false ->
            ignore
    end.

trace_log(Name, Text) ->
    case enable_trace(Name) of
        true ->
            otter_span_mpdict_api:log(Name, Text);
        false ->
            ignore
    end.

trace_tag(Name, Key, Value) when is_list(Value) ->
    trace_tag(Name, Key, list_to_binary(Value));
trace_tag(Name, Key, Value) when is_atom(Value) ->
    trace_tag(Name, Key, atom_to_binary(Value, latin1));
trace_tag(Name, Key, Value) ->
    case enable_trace(Name) of
        true ->
            otter_span_mpdict_api:tag(Name, Key, Value);
        false ->
            ignore
    end.

trace_finish(Name) ->
    case enable_trace(Name) of
        true ->
            otter_span_mpdict_api:finish(Name),
            %it's necessary to erase spaninfo, because don't judge whether the span exists when log/tag
            erase({otter_span_information, Name});
        false ->
            ignore
    end.

get_span(Name) ->
    case enable_trace(Name) of
        true ->
            case get(trace_proc) of
                false ->
                    undefined;
                _ ->
                    Span = otter_span_mpdict_api:get_span(Name),
                    case element(4, Span) of
                        Name ->
                            Span;
                        _ ->
                            undefined
                    end
            end;
        _ ->
            undefined
    end.

%%%============================================
%%  Internal API
%%%============================================

enable_trace(Name) ->
    TraceList = application:get_env(myapp, opentracing_list, []),
    NameStr = atom_to_list(Name),
    lists:any(fun(Prefix) -> lists:prefix(Prefix, NameStr) end, TraceList).

maybe_trace() ->
    Probablity = application:get_env(msync, opentracing_probability, undefined),
    check(Probablity).

check({one_out_of, Nr}) ->
    case crypto:rand_uniform(0, Nr) of
        1 ->
            put(trace_proc, true),
            true;
        _ ->
            put(trace_proc, false),
            false
    end;
check(_) ->
    put(trace_proc, true),
    true.

tips: 1.set a probability for calling trace_start/1 2.erase spaninfo when the span finished

using: first.start major span trace_start/1 second.start child spans get_span/1 => trace_start/2

Could you give me some advice？