Support Instana as a Tracing Backend

[kdhamric]

As a User That is sending Tracing Information to Instana for observability I would like to be able to leverage Tracetest To ensure my system is functioning via a system wide trace-based test.

As I user, I would prefer a direct integration if possible So all I have to do is configure the Instana connection in Tracetest, But getting access via the OpenTelemetry Collector is ok if that is not possible.

[joshleecreates]

Looking forward to this, let me know how I can support!

[kdhamric]

@joshleecreates has provided @kdhamric with an account to use. Ken has invited @xoscar to the instance as an owner. Once we begin work, either Oscar or Ken should be able to add the dev to the account.

[joshleecreates]

Excellent. Our API documentation is available here in the meantime: https://instana.github.io/openapi/

[xoscar]

Thanks @joshleecreates I'll be taking a look today :D

[danielbdias]

hi @joshleecreates ! I created a small PoC using OTLP, and I was able to make it work with Instana, here: #3594

Now, I'm trying to do the same example with Tracetest, connecting directly on Instana to get traces and spans. Experimenting with Instana API using two APIs:

• POST /api/application-monitoring/analyze/traces: https://instana.github.io/openapi/#operation/getTraces
• GET /api/application-monitoring/v2/analyze/traces/{id}: https://instana.github.io/openapi/#operation/getTraceDownload

One thing that I noticed is that the second API only returns data about spans at a high level without tags:

{
    "items": [
        // ...
        {
            "id": "6ca00c8fa410fa0a",
            "timestamp": 1707177111058,
            "parentId": "0fce2fa8983cb58f",
            "foreignParentId": null,
            "name": "query SELECT",
            "duration": 1,
            "minSelfTime": 1,
            "networkTime": null,
            "callCount": 1,
            "errorCount": 0,
            "destination": {
                "service": {
                    "id": "UNKNOWN",
                    "label": "Unspecified"
                },
                "endpoint": {
                    "id": "F8uZbXdq3oXmscc5yADFItCR6kY",
                    "label": "",
                    "type": "OPENTELEMETRY"
                },
                "technologies": []
            },
            "cursor": {
                "@class": ".IngestionOffsetCursor",
                "ingestionTime": 1707178503000,
                "offset": 6
            }
        }
    ]
}

However, I can see them on Instana UI:

Do you know if there are other endpoints that we can use to fetch this data?

[danielbdias]

Folks, complementing our discussion on CNCF slack:

V1 Integration

Add Instana as an OTLP-based provider, as we did with Datadog, New Relic, etc., using OTel Collector to send OTLP traffic for both Instana and Tracetest.

flowchart LR
  Instana["Instana"]
  TracetestCore["Tracetest Core"]
  UserAPI["User API"]
  OTelCollector["Open Telemetry\nCollector"]
  
  UserAPI -- Send OTLP data --> OTelCollector
  OTelCollector-- Redirects data --> Instana
  OTelCollector -- Redirects data --> TracetestCore

The first implementation of this version using the OTLP provider is made on this PR: https://github.com/kubeshop/tracetest/pull/3610 .

V2 Integration

Add Instana as a direct integration, connecting on their APIs to grab Traces and Spans. We can use https://instana.github.io/openapi/#operation/getTraceDownload to do that. However, this API has only high-level data about the spans, which will not help users use selectors effectively.

flowchart LR
  Instana["Instana"]
  TracetestCore["Tracetest Core"]
  UserAPI["User API"]
  OTelCollector["Open Telemetry\nCollector"]
  
  UserAPI -- Send OTLP data --> OTelCollector
  OTelCollector-- Redirects data --> Instana
  TracetestCore -- Fetch traces from API --> Instana

Some questions that we need to figure out yet:

1. What is the time needed to emit a trace and see them on their API?
2. Can we have access to span attributes to build selectors?