spigo
spigo copied to clipboard
adrianco
Extend message to include trace and span ids for zipkin like logging
Add two 64bit fields to message. One is set to indicate the overall context of a series of put or getrequest and getresponse messages making up a request from end to end. The other is unique for each span to be monitored. If both are set then they are logged along with message timestamp and other metadata. The result should be able to simulate enough detail to output zipkin or similar log files.
Added Context struct made up of two uint32, Request and Span. Decided that for the simulation 64bit wasn't needed. Easy to change implementation. Currently use an incrementing counter for Span and Request. Incrementer isn't thread-safe but seems good enough for simulation. Would have to have the same Request update two Spans simultaneously to be a real issue. Tried math.rand.Uint32 but it added a lot of overhead. Easy to swap in if needed.
Need to change terminology Request to Trace Zipkin thrift format https://github.com/openzipkin/zipkin/blob/master/zipkin-thrift/src/main/thrift/com/twitter/zipkin/zipkinCore.thrift Phosphor https://github.com/mattheath/phosphor/blob/master/domain/domain.go Zipkin in go https://github.com/go-kit/kit/tree/master/tracing/zipkin
I've implemented the trace propagation and collection, but don't yet have attributes being stored. Plan to try and interface to go-kit Zipkin in go packages.
Current code checked in writes flows to json_metrics/"arch"_flow.json but it doesn't write interesting data there yet and it's not in json format. So far only denominator and elb have been instrumented for flow collection and return path routing. Plan to make sure I have the pattern right before I instrument all the other packages.
if you can commit a couple of sample flow.json s so far, that will be awesome.. On Sat, Jul 25, 2015 at 5:07 PM Adrian Cockcroft [email protected] wrote:
Current code checked in writes flows to json_metrics/_flow.json but it doesn't write interesting data there yet and it's not in json format. So far only denominator and elb have been instrumented for flow collection and return path routing. Plan to make sure I have the pattern right before I instrument all the other packages.
— Reply to this email directly or view it on GitHub https://github.com/adrianco/spigo/issues/33#issuecomment-124918746.
Needs some more work. At present the only data collected in the trace is the instance name, and they are being collected in a map so it's being dumped in a random order. Need to add timestamps and sort output at least. Looking at interfacing via go-kit zipkin, need to understand how that works in detail to figure out where to hook in.
Decided that it was best to pass span parents around in the Context struct, so added that, which will make it easier to map the tree.
Should the response to a GetRequest have the same span id as the GetRequest? Wondering if I should reverse span and parent on the way back up. The way messages work in spigo is one way, no ack needed, so I have a GetRequest that causes a GetResponse to work its way back up the calling tree. For an http GET, does the outbound request have the same span id as the reply?
Borrowed ideas from Prometheus to use atomic increment for flow id's as described here https://github.com/gophercon/2015-talks/blob/master/Bj%C3%B6rn%20Rabenstein%20-%20Prometheus/slides.pdf