X-Grafana-User header not sent with Live View

[dani]

Describe the bug

I run victorialogs behind an nginx proxy (which handles adding extra_filters query arg depending on some authentication factors). When doing so, Live View from Grafana (using victorialogs datasource) is not working : logs are not streamd, I can only see logs up until the time I pressed the Live View button. Everything else is working (standard query from Grafana, query from VMUI, query or tailing using curl). Only Live View from Grafana is broken. As soon as I remove the nginx proxy (so Grafana talks to victoriametrics "directly"), everything is working, including Live View

To Reproduce

Configure Grafana victorialogs datasource to talk to victorialogs through an nginx reverse proxy Try to use Live View

My nginx conf

worker_processes        auto;
error_log               /dev/stderr warn;
pid                     /tmp/nginx.pid;

events {
  worker_connections    1024;
}

http {
  proxy_temp_path       /tmp/proxy_temp;
  client_body_temp_path /tmp/client_temp;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;
  include               /etc/nginx/mime.types;
  default_type          application/octet-stream;
  log_format            main  '$remote_addr - $remote_user [$time_local] "$request" '
                              '$status $body_bytes_sent "$http_referer" '
                              '"$http_user_agent" "$http_x_forwarded_for"';
  access_log            /dev/stdout main;
  sendfile              on;
  keepalive_timeout     65;

  set_real_ip_from 127.0.0.1;
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;
  # Detect if header X-Forwarded-Tls-Client-Cert-Info is present
  map $http_x_forwarded_tls_client_cert_info $has_x_forwarded_tls_client_cert_info {
    default 0;
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? 1;
  }
  # Detect if header X-Grafana-User is present
  map $http_x_grafana_user $has_x_grafana_user {
    default 0;
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? 1;
  }
  # map permissions to a header dependant (X-Forwarded-Tls-Client-Cert-Info) variable
  map $http_x_forwarded_tls_client_cert_info $extra_filters_x_forwarded_tls_client_cert_info {
    default "NOMAD_NAMESPACE%3A%3Dfake";
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? "";
  }
  # map permissions to a header dependant (X-Grafana-User) variable
  map $http_x_grafana_user $extra_filters_x_grafana_user {
    default "NOMAD_NAMESPACE%3A%3Dfake";
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? "";
  }
  server {
    listen 127.0.0.1:9427 default_server;
    server_tokens off;
    proxy_buffering off;
    client_max_body_size 0;
    proxy_http_version 1.1;
    proxy_request_buffering off;

    # Clear any extra_filters query arg
    if ($args ~ (.*)extra_filters=[^&]*(.*)){
      set $args $1$2;
    }
    # Set default extra_filters
    set $extra_filters "NOMAD_NAMESPACE%3A%3Dfake";

    # Set permissions for header X-Forwarded-Tls-Client-Cert-Info
    if ($has_x_forwarded_tls_client_cert_info != 0){
      set $extra_filters "$extra_filters_x_forwarded_tls_client_cert_info";
    }
    # Set permissions for header X-Grafana-User
    if ($has_x_grafana_user != 0){
      set $extra_filters "$extra_filters_x_grafana_user";
    }
    set $args "$args&extra_filters=$extra_filters";

    location ~ /(insert/|health) {
      access_log off;
      proxy_pass http://127.0.0.1:9428;
    }

    location /select/ {
      proxy_pass http://127.0.0.1:9428$uri$is_args$args;
    }

    location / {
      proxy_pass http://127.0.0.1:9428;
    }
  }
}

Version

victoria-logs-20241222-133402-tags-v1.4.0-victorialogs-0-g58f09fe3f8

Logs

No response

Screenshots

No response

Used command-line flags

-envflag.enable="true"
-envflag.prefix="VLOGS_"
-http.disableResponseCompression="true"
-httpListenAddr="127.0.0.1:9428"
-loggerOutput="stdout"
-loggerTimezone="Europe/Paris"
-retention.maxDiskSpaceUsageBytes="28252570KiB"
-retentionPeriod="90d"
-search.maxConcurrentRequests="16"
-search.maxQueryDuration="30s"
-search.maxQueueDuration="20s"
-storage.minFreeDiskSpaceBytes="500MB"
-storageDataPath="/data"

### Additional information

_No response_

[dani]

After some more debugging, the issue comes from the fact that the X-Grafana-User header is not sent by Grafana when using Live View mode (so it ended with a default extra_filters which filters everything out). Not sure if it's in the datasource plugin or in Grafana itself

[dani]

Just checked with the Loki datasource, and the X-Grafana-User header is correctly sent in Live View (when dataproxy -> send_user_header is true), so it looks like the problem is in VictoriaLogs datasource

[dmitryk-dk]

Hi @dani ! As far as I can see, the Grafana uses its own code to handle those headers, but when you are trying to use RunStream function, there are no headers present in the request, so in Live mode, we can't set correctly X-Grafana-User header. I used a debugger and checked all the middleware and in live mode middleware with setting headers like X-Grafana-.. skipped. I will try to find a better solution for this problem but need to check how it is better to get BackendContext from the request

[dmitryk-dk]

Hi @dani ! My issue was closed with new improvements from the plugin sdk side. So i will update the plugin version and will check how it will work with the headers for live mode https://github.com/grafana/grafana-plugin-sdk-go/issues/1218

[dmitryk-dk]

Hi @dani ! I have checked the changes and the Grafana needs to make changes in the code, because those headers are transport by the protobuf to the sdk and then to the plugin itself

message RunStreamRequest {
  PluginContext pluginContext = 1;

  // path part of a channel.
  string path = 2;
  // optional raw data. May be used as an extra payload supplied upon subscription.
  // For example, can contain JSON query object.
  bytes data = 3;
}

In the protobuf there were no changes. I asked in the issue, should I create the issue in the Grafana, or it was created already. So we need to wait until protobuf will change

[dmitryk-dk]

Hi @dani ! Have you tried Grafana v12 and the latest version of the datasource? Was the issue solved?

[dani]

I tried, but get very inconsistent results. Sometimes the live tailing is working, but ignoring my logsql filter (it just tails everything), and most of the time, tailing is not working (just as before)

[dmitryk-dk]

I tried, but get very inconsistent results. Sometimes the live tailing is working, but ignoring my logsql filter (it just tails everything), and most of the time, tailing is not working (just as before)

Hi @dani ! If the issue with headers was solved, could you close this issue as resolved? However, open a new one and describe the problem you are experiencing with Grafana v12. We will check it as the new issue