anonlink-entity-service icon indicating copy to clipboard operation
anonlink-entity-service copied to clipboard

Improve opentracing

Open hardbyte opened this issue 4 years ago • 5 comments

Misc issues when trialing jaeger tracing.

  1. I've seen the warning "invalid parent span" for the following tasks at various times (not always):
  • entityservice.tasks.solver.solver_task,
  • entityservice.tasks.mark_run_complete.mark_run_complete and
  • entityservice.tasks.run.prerun_check
  • entityservice.tasks.comparing.compute_filter_similarity

image

  1. Aggregate comparisons task doesn't include project or run id information:

image

  1. Uploading CLKS don't seem to be linked to the worker task that handles the uploaded data:

image

hardbyte avatar Feb 11 '20 04:02 hardbyte

We might not be initializing the tracer correctly due to the WSGI forking - https://github.com/jaegertracing/jaeger-client-python#wsgi-multi-processing-fork2

hardbyte avatar Feb 11 '20 22:02 hardbyte

More nice to haves:

  • we could show which deployment the service is from - e.g. a feature branch, or helm release.
  • we could link our client tools too - we can in theory trace by injecting HTTP_HEADERS from the request side too.

hardbyte avatar Feb 12 '20 04:02 hardbyte

During debugging I note that the parent_span of our tasks have an uber-trace-id

hardbyte avatar Mar 09 '20 00:03 hardbyte

opentracing-instrumentation natively supports celery since release 3.1. That might help...

wilko77 avatar May 28 '20 07:05 wilko77

I've been using opentelemetry, exporting to jaeger on another project. These are the dependencies I'm using:

jaeger-client = "^4.3.0"
opentelemetry-api = "^0.17b0"
opentelemetry-sdk = "^0.17b0"
opentelemetry-instrumentation = "^0.17b0"
opentelemetry-instrumentation-flask = "^0.17b0"
opentelemetry-instrumentation-celery = "^0.17b0"
opentelemetry-instrumentation-sqlalchemy = "^0.17b0"
opentelemetry-exporter-jaeger = "^0.17b0"
opentelemetry-instrumentation-redis = "^0.17b0"
opentelemetry-instrumentation-psycopg2 = "^0.17b0"
opentelemetry-instrumentation-requests = "^0.17b0"

hardbyte avatar Feb 22 '21 02:02 hardbyte