GKE : trace work with post-forward but not if calling by https url

[raphaelauv]

I have a pod using CloudTraceSpanExporter inside a GKE auto-pilot with CLOUD NAT (private cluster)

If I call the api by the https://xxxxxx.com it do not produce a trace ( the api work and answer http request on the url without any problem )

but if I post-forward the pod on my host it produce a trace in gcp trace

I have activate the DEBUG logging but it do not produce any error when calling by the https URL

loggers = [logging.getLogger(name) for name in logging.root.manager.loggerDict]
for logger in loggers:
    logger.setLevel(logging.DEBUG)

the LB conf

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-XXXX-ingress
  namespace: api
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.allow-http: "false"
    ingress.gcp.kubernetes.io/pre-shared-cert: {{ .Values.val.cert }}
spec:
  rules:
    - http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: XXXXX-np-service
                port:
                  number: 60000

[raphaelauv]

I do not have the problem with

opentelemetry-exporter-gcp-trace==1.1.0
opentelemetry-propagator-gcp==1.1.0

but I have the problem with :

opentelemetry-exporter-gcp-trace==1.3.0
opentelemetry-propagator-gcp==1.3.0

[aabmass]

@raphaelauv is this still an issue? Sounds like maybe a regression in our propagator. Any idea what the x-cloud-trace-context header looks like here?

[aabmass]

@raphaelauv I'm closing this issue. If you encounter the problem again, please re-open.

[raphaelauv]

@aabmass I just did a test with

opentelemetry-exporter-gcp-trace==1.6.0
opentelemetry-propagator-gcp==1.6.0

problem is still the same

could you please re-open please

[raphaelauv]

reproducible example

import os

from fastapi import FastAPI

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.cloud_trace_propagator import CloudTraceFormatPropagator

TRACING_GCP_PROJECT_ID = os.getenv("TRACING_GCP_PROJECT_ID", None)

app = FastAPI()


@app.get('/test')
async def test():
    return {"ok"}


@app.get('/health')
async def test():
    return {"ok"}


set_global_textmap(CloudTraceFormatPropagator())
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(BatchSpanProcessor(CloudTraceSpanExporter(project_id=TRACING_GCP_PROJECT_ID)))
trace.set_tracer_provider(tracer_provider)

FastAPIInstrumentor.instrument_app(app=app, excluded_urls="/$,/health$,/openapi.json$")

FROM python:3.11-slim

ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV PIP_NO_CACHE_DIR=TRUE
ENV PIP_DISABLE_PIP_VERSION_CHECK=TRUE

RUN mkdir -p /opt/program
WORKDIR /opt/program

COPY requirements.txt /opt/program/
RUN python3 -m pip install -r /opt/program/requirements.txt

COPY src/api /opt/program/api

CMD ["uvicorn","--port" ,"8080","--host" ,"0.0.0.0","--no-access-log", "api.asgi:app"]

fastapi==0.85.0
uvicorn[standard]==0.18.3
opentelemetry-instrumentation-fastapi==0.43b0
opentelemetry-exporter-gcp-trace==1.6.0
opentelemetry-propagator-gcp==1.6.0