Log which upstream a request was sent to?

[ahhda]

We are using a number of upstreams (self-hosted + external providers) for our services. Sometimes we come across errors in our requests and would like to debug which particular node/service was causing the issue.

We can see the request and response if we enable access logs but we can't track which node actually served the request.

Would it be possible to add this to the access logs?

Dshackle config

host: 0.0.0.0
port: 2449
tls:
  enabled: false

cache:
  redis:
    enabled: false

proxy:
  host: 0.0.0.0
  port: 8545
  preserve-batch-order: true
  tls:
    enabled: false
  routes:
    - id: eth
      blockchain: ethereum

health:
  port: 8082
  host: 0.0.0.0
  path: /health
  blockchains:
    - chain: ethereum
      min-available: 1

monitoring:
  enabled: true
  prometheus:
    enabled: true
    bind: 0.0.0.0

accessLog:
  enabled: true
  filename: /var/log/access_log.jsonl
  include-messages: true

cluster:
  defaults:
    - chains:
        - ethereum
  upstreams:
    - id: cow-nethermind
      ...
    - id: alchemy
      ...

[splix]

@ahhda we have the same problem so it's our priority to add this feature asap. I mean, it's not available at this moment, but we're going to add additional log and tracing capabilities soon.

[ahhda]

For posterity, we implemented an Nginx proxy between our nodes and dshackle. This helps us log requests and responses sent to the nodes which we can then view in Elastic Search for debugging.

Nginx conf

load_module /usr/lib/nginx/modules/ndk_http_module.so;
load_module /usr/lib/nginx/modules/ngx_http_lua_module.so;

events {
    worker_connections 1024;
}

http {
    log_format bodylog escape=none '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

    lua_need_request_body on;

    server {
        listen 80;
        listen [::]:80;

        set $resp_body "";
        body_filter_by_lua '
            local resp_body = string.sub(ngx.arg[1], 1, 1000)
            ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
            if ngx.arg[2] then
            ngx.var.resp_body = ngx.ctx.buffered
            end
        ';
        access_log  /var/log/nginx/server.log bodylog;
        proxy_ssl_server_name on;

        location /alchemy {
            proxy_pass ${ALCHEMY_URL};
        }

        location /pokt {
            proxy_pass ${POKT_URL};
        }

        location /cow-nethermind {
            proxy_pass ${COW_URL};
        }
    }
}

Dshackle config

cluster:
  defaults:
    - chains:
        - ethereum
  upstreams:
    - id: cow-nethermind
      chain: ethereum
      connection:
        ethereum:
          rpc:
            url: "nginx/cow-nethermind"
    - id: pokt
      chain: ethereum
      role: fallback
      options:
        disable-validation: true
      connection:
        ethereum:
          rpc:
            url: "nginx/pokt"
    - id: alchemy
      chain: ethereum
      role: secondary
      options:
        disable-validation: true
      connection:
        ethereum:
          rpc:
            url: "nginx/alchemy"

Closing the issue as this is already part of the roadmap.