relay Cannot seem to scrub the user IP using relay's project config

Hi! I am trying to use relay to scrub the IP addresses of clients in several browser Javascript projects, before they hit Sentry's servers. But I have run into a lot of trouble getting this to work properly. For context, I am running the relay in the official docker container behind a reverse proxy that sets the X-Forwarded-For header to the IP address that I would like to scrub, and I am not setting the IP address in the user interface in the event. I have checked in the network view of my browser that there is no user.ip_address field in the event that was sent.

The documentation says that it should be enough to use a config like this under .relay/projects/<ID>.json to get it to work, but the IP still shows up in the Sentry UI as before.

{
  "publicKeys": [
    ...
  ],
  "config": {
    "allowedDomains": ["*"],
    "applications": {
      "$string": ["@ip:replace"]
    }
  }
}

I assume the relay cannot run in managed mode when these project config files are in use? I am currently running it in static mode.

I also dug through the relay source code, and found this test which has some dummy project config that seems to do what I want. So I appended that JSON under the config object in my project config JSON file, but it didn't work either. It seems like it's not just the event payload that is scrubbed, but also HTTP headers like X-Forwarded-For as well. Is this true?

I have observed through trial and error that if I set the user.ip_address to 127.0.0.1 in my Javascript project, then 127.0.0.1 shows up in the Sentry UI as expected. Otherwise, the IP address that I would like to scrub shows up, which means that the relay is somehow forwarding it upstream. At first I thought it was the X-Forwarded-For header that was copied by the relay, which seems to be what happens here in the source code. But then I tried running the relay locally in a docker container, which resulted in the IP address of the docker host (something like 10.15.0.1) ending up in the Sentry UI. Because the docker host runs some sort of NAT, this would be the remote client IP from the perspective of the relay. So it seems like the relay always forwards the client IP address upstream, and doesn't bother with the X-Forwarded-For header? This seems to be the case according to the user interface documentation. But this does not make sense when using the reverse proxy, since in that case it is the client's true IP (that I would like to scrub) that ends up in the Sentry UI and not the IP of the reverse proxy.

My questions are as follows:

Am I missing anything obvious in my project configs?
How exactly is the IP address shown in the Sentry UI derived from user.ip_address, X-Forwarded-For, the remote client IP, and other metadata? Will the relay forward the X-Forwarded-For header to Sentry? Will it set the user.ip_address to the value of X-Forwarded-For or the remote client IP before forwarding it upsteam?
I could hardcode user.ip_address to 127.0.0.1 in my JS projects. But this seems like a not so ideal solution if the X-Forwarded-For header is forwarded to Sentry regardless?
Can I obtain more debugging information from the relay? I.e. stuff like the event payloads it sends to Sentry, if it fails to load some project config, if it didn't find anything to scrub, etc.

Oct 19 '23 15:10 william-stacken

Hi! The IP address can be sent from SDKs in the event payload, and if not Relay can take the remote (not local) IP address. I don't know what your requirements are, but having the remote IP in Sentry may be alright. By default, Relay forwards X-Forwarded-For and user.ip_address fields but these can be scrubbed with data scrubbing, something like this should work (see docs):

"applications": {
  "$string": ["@ip:replace"],
  "$user.ip_address || $sdk.client_ip": ["@anything:remove"]
}

Relay should print some debug logs to stdout, this can be helpful for fetching project configs but not sure about data scrubbing. If you're on SaaS, linking to the event always helps to debug your specific problem.

Oct 20 '23 09:10 iker-barriocanal

I tried adding your suggestion to my project config and also tried adding some more filters just in case, but I am still not able to see that anything was removed or filtered. The event shows up in Sentry with the same metadata as when my relay was in managed mode and without scrubbing.

/etc/sentry-relay/projects/1.json

{
  "slug": "website",
  "publicKeys": [
    {
      "publicKey": "1111111",
      "isEnabled": true
    }
  ],
  "config": {
    "allowedDomains": ["*"],
    "applications": {
      "$string": ["@ip:replace"],
      "$user.ip_address || $sdk.client_ip || $http.headers.x-forwarded-for": ["@anything:remove"],
      "tags.os": ["@anything:remove"],
      "tags.transaction": ["@anything:remove"],
      "tags.url": ["@anything:remove"],
      "tags.mechanism": ["@anything:remove"],
      "contexts.**": ["@anything:remove"]
    }
  }
}

/etc/sentry-relay/config.yml

relay:
  mode: static
  upstream: https://o123.ingest.sentry.io/
  host: 0.0.0.0
  port: 3000
  tls_port: null
  tls_identity_path: null
  tls_identity_password: null
logging:
  level: "info"

relay stdout (using docker logs)

2023-10-20T12:57:19.973026Z  INFO relay::setup: launching relay from config folder /etc/sentry-relay
2023-10-20T12:57:19.975952Z  INFO relay::setup:   relay mode: static
2023-10-20T12:57:19.975989Z  INFO relay::setup:   relay id: ...
2023-10-20T12:57:19.976001Z  INFO relay::setup:   public key: 1111111
2023-10-20T12:57:19.976011Z  INFO relay::setup:   log level: INFO
2023-10-20T12:57:19.977136Z  INFO relay_server: relay server starting
2023-10-20T12:57:20.033109Z  INFO relay_server::actors::outcome: Configured to emit outcomes as client reports
2023-10-20T12:57:20.036543Z  INFO relay_server::actors::outcome: OutcomeProducer started.
2023-10-20T12:57:20.037388Z  INFO relay_server::actors::outcome_aggregator: outcome aggregator started
2023-10-20T12:57:20.038649Z  INFO relay_server::actors::processor: starting 2 envelope processing workers
2023-10-20T12:57:20.040697Z  INFO relay_server::actors::envelopes: envelope manager started
2023-10-20T12:57:20.041003Z  INFO relay_metrics::router: metrics router started
2023-10-20T12:57:20.041645Z  INFO relay_server::actors::project_cache: project cache started
2023-10-20T12:57:20.042562Z  INFO relay_server::actors::relays: key cache started
2023-10-20T12:57:20.044384Z  INFO relay_server::actors::project_upstream: project upstream cache started
2023-10-20T12:57:20.048478Z  INFO relay_server::actors::project_local: project local cache started
2023-10-20T12:57:20.056051Z  INFO relay_server::actors::server: spawning http server
2023-10-20T12:57:20.056295Z  INFO relay_server::actors::server:   listening on http://0.0.0.0:3000/

Then I run the command

curl -v "https://my-sentry-relay.com/api/1/envelope/?sentry_key=1111111&sentry_version=7&sentry_client=sentry.javascript.browser/7.68.0" \
     -H "Content-Type: text/plain;charset=UTF-8" \
      -H "X-forwarded-for: 4.4.4.4" \
    --data-binary @- << EOF
{"event_id":"12345678901234567890123456789012","sent_at":"2023-10-20T14:40:02.062Z","sdk":{"name":"sentry.javascript.browser","version":"7.68.0"},"trace":{"environment":"development","release":"debug","public_key":"1111111","trace_id":"12345678901234567890123456789012","sample_rate":"0.01","sampled":"false"}}
{"type":"event"}
{"exception":{"values":[{"type":"Error","value":"my message","mechanism":{"type":"generic","handled":true}}]},"level":"error","event_id":"12345678901234567890123456789012","platform":"javascript","timestamp":1697806682.061,"environment":"development","release":"debug","sdk":{"integrations":["InboundFilters","FunctionToString","TryCatch","Breadcrumbs","GlobalHandlers","LinkedErrors","Dedupe","HttpContext","BrowserTracing"],"name":"sentry.javascript.browser","version":"7.68.0","packages":[{"name":"npm:@sentry/browser","version":"7.68.0"}]},"contexts":{"trace":{"op":"pageload","span_id":"12345","trace_id":"12345678901234567890123456789012"}},"tags":{"transaction":"/"},"breadcrumbs":[{"timestamp":1697806682.057,"type":"debug","level":"info","message":"A message"}],"request":{"url":"https://example.com/","headers":{"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/118.0"}}}
EOF

And I get 4.4.4.4 as the user IP in Sentry. If I don't send the X-Forwarded-For header using curl, I get the unscrubbed client IP.

I have confirmed that the 1.json project config file is read, because if I enter invalid JSON then errors are printed to the relay stdout.

Oct 20 '23 14:10 william-stacken

This will require a bit more investigation. I'll add the ticket to the backlog and we'll get back at this soon!

Oct 23 '23 10:10 iker-barriocanal

Hi @william-stacken, sorry for not getting back to you sooner. Is this problem still occuring?

Feb 20 '24 22:02 TBS1996

Please re-open or create a new issue, if this is still a problem.

Dec 11 '24 13:12 Dav1dde