`@sentry/node` with `includeLocalVariables` is not showing anything about local variables on any errors on the Sentry dashboard

[ruohola]

Is there an existing issue for this?

• [X] I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• [X] I have reviewed the documentation https://docs.sentry.io/
• [X] I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

~~7.64.0~~ 7.73.0

Framework Version

Node 18.15.0

Link to Sentry event

[private]

SDK Setup

import * as Sentry from '@sentry/node'

Sentry.init({
  environment: ...,
  dsn: ...,
  tracesSampleRate: 1.0,
  includeLocalVariables: true,
  integrations: [
    new Sentry.Integrations.LocalVariables({
      captureAllExceptions: true,
    }),
  ],
  debug: true,
})

Steps to Reproduce

I'm trying to use @sentry/node's new includeLocalVariables (https://blog.sentry.io/local-variables-for-nodejs-in-sentry/). I'm using Sentry SDK 7.73.0 with Node 18.15.0 (TypeScript, without ESM modules), which should work fine. But my caught errors do not show any local variables in the Sentry Dashboard.

Expected Result

Each caught error would the values of show local variables as such on the Sentry dashboard (picture from the blog post):

Actual Result

The errors show up fine as before, but no local variables can be seen anywhere from the dashboard:

┆Issue is synchronized with this Jira Improvement by Unito

[timfish]

But my caught errors do not show any local variables in the Sentry Dashboard

The LocalVariables integration only includes local variables for uncaught exceptions by default. This default was chosen since there might be significant overhead capturing local variables for all caught exceptions in exception heavy code. Local variables are captured via the node inpector API and this has to suspend the event loop to capture the context.

You can enable local variable capture for all exceptions via the captureAllExceptions option passed to the integration constructor:

import * as Sentry from '@sentry/node'

Sentry.init({
  ...
  includeLocalVariables: true,
  integrations: [new Sentry.Integrations.LocalVariables({ captureAllExceptions: true })],
})

[martson77]

+1

[ruohola]

The LocalVariables integration only includes local variables for uncaught exceptions by default.

Interesting. And what about should/could local variables be included in exceptions caught by the Sentry Express integration, i.e. app.use(Sentry.Handlers.errorHandler())? That's my use case, and I am not seeing any local variables on exceptions handled by the integration.

[lforst]

@ruohola Hi, what node version are you using? Did you turn on the captureAllExceptions flag?

[ruohola]

@ruohola Hi, what node version are you using? Did you turn on the captureAllExceptions flag?

I'm using Node 18.15.0 as pointed out in opening. No, I did not yet enable captureAllExceptions. The performance hit does not entice me. My immediate question now is that should the exceptions caught by the Express integration include local variables? To me it would make sense if they did, since those are basically uncaught exceptions from my application's point of view and exceptions that propagate to Sentry.Handlers.errorHandler() are rare enough for any performance hit to be marginal.

[timfish]

since those are basically uncaught exceptions from my application's point of view

When talking about uncaught exceptions we are referring to those that would trigger uncaughtException or unhandledRejection events. These cause newer versions of node to terminate.

Caught exceptions are caused by any throw that is subsequentially caught by a try/catch. They may be caught in your code or they may be caught in some library code but they are still considered "caught" from the debuggers point of view.

The Chrome debugger has seperate options to pause on caught/uncaught:

exceptions that propagate to Sentry.Handlers.errorHandler() are rare enough for any performance hit to be marginal

In most cases, enabling captureAllExceptions should have negligible performance impact and we may even enable it by default in the future. However, I would still encourage you to test and profile it thouroughly before using it in a heavily loaded production app.

captureAllExceptions will pause execution and capture local variables every time a throw is encountered in code. For example, if you code uses something like throw new NotFoundError() to handle 404 status, every 404 will result in capturing local variables.

[ruohola]

since those are basically uncaught exceptions from my application's point of view

When talking about uncaught exceptions we are referring to those that would trigger uncaughtException or unhandledRejection events. These cause newer versions of node to terminate.

Caught exceptions are caused by any throw that is subsequentially caught by a try/catch. They may be caught in your code or they may be caught in some library code but they are still considered "caught" from the debuggers point of view.

The Chrome debugger has seperate options to pause on caught/uncaught:

Yes, I very much understand the difference between caught and uncaught exceptions. My question was that would it make sense for @sentry/node's captureException() (which is used inside Sentry.Handlers.errorHandler()) be altered so that one could include local variables on the error on a case-by-case basis. This would allow the Express integration to conveniently include local variables on top-level errors without the possible performance hit of enabling Sentry.Integrations.LocalVariables({ captureAllExceptions: true }).

 

captureAllExceptions will pause execution and capture local variables every time a throw is encountered in code. For example, if you code uses something like throw new NotFoundError() to handle 404 status, every 404 will result in capturing local variables.

Which does not sound at all something like I would want to enable.

[lforst]

My question was that would it make sense for @sentry/node's captureException() (which is used inside Sentry.Handlers.errorHandler()) be altered so that one could include local variables on the error on a case-by-case basis.

If it is possible, yes that would make sense, however, it is likely not. The idea is not bad.

Node does not provide a dedicated API to grab local variables in the current scope, but it is possible with Node.js' debugger/inspector API, which we are leveraging here. If you think it through, you can't retroactively tell the debugger to grab the local variable state as it was when an error was thrown. However, that's exactly what we would need to do, in order to make what you suggest happen. I scanned the debugger API and couldn't find a way (maybe I missed something though).

For the above reason, we need this all-or-nothing option. Either we can grab the variables in time, or we don't at all.

Which does not sound at all something like I would want to enable.

We thought people might not wanna have this on by default since this obviously impacts performance, so we added this option. Makes sense I think. This is addon functionality, not essential for the SDK to work. Feel free to leave it turned off.

In case you care about the technical details feel free to take a look at our code for this functionality. It is all within one file!

https://github.com/getsentry/sentry-javascript/blob/9dc0f1243625cd1bdb647f0427bc6adc06cba561/packages/node/src/integrations/localvariables.ts#L262

[AbhiPrasad]

Which does not sound at all something like I would want to enable.

The overhead caused by this really does vary from app to app, I would advise profiling in more detail if you're really interested in using this! Perhaps even with extra throwing the overhead is worthwhile to get extra debugging context.

[timfish]

I've opened a PR to add rate limiting for capturing caught exceptions. This should make it safe to enable for all: https://github.com/getsentry/sentry-javascript/pull/9102

[AbhiPrasad]

We've released changes to make capturing local variables for all errors (handled and unhandled) the default in 7.73.0 of the Node SDK. Tim's also done the great work of making sure we optimize the overhead in these scenarios so that impact is minimal. Please give it a try and let us know what you think!

https://github.com/getsentry/sentry-javascript/releases/tag/7.73.0

[ruohola]

After upgrading to 7.73.0 (and even explicitly enabling integrations: [new Sentry.Integrations.LocalVariables({ captureAllExceptions: true })]), I'm still not seeing anything about local variables in the stack traces shown for errors on my Sentry dashboard. Here are some screenshots from parts of my dashboard that don't show sensitive information:

[timfish]

You shouldn't need to manually set integrations since captureAllExceptions now defaults to true.

You want to enable the includeLocalVariables init option:

Sentry.init({
  dsn: "https://[email protected]/0",
  includeLocalVariables: true,
});

[ruohola]

You shouldn't need to manually set integrations since captureAllExceptions now defaults to true.

You want to enable the includeLocalVariables init option:

Sentry.init({
  dsn: "https://[email protected]/0",
  includeLocalVariables: true,
});

Yeah, I just had it there for good measure / documentation value. But, as shared in my opening, I do have includeLocalVariables: true. Here's my full init:

Sentry.init({
  environment: ...,
  dsn: ...,
  tracesSampleRate: 1.0,
  includeLocalVariables: true,
  integrations: [
    new Sentry.Integrations.LocalVariables({
      captureAllExceptions: true,
    }),
  ],
})

And all other functionalities of Sentry are working just fine for me.

[timfish]

There is rate-limiting for capturing of local variables for caught exceptions because each throw in your code causes the debugger to pause. The default limit is is 50 captures per second.

If you enable debug output, do you see any relevant messages? When rate limiting kicks in we log this.

Sentry.init({
  dsn: "https://[email protected]/0",
  debug: true,
}); 

[ruohola]

There is rate-limiting for capturing of local variables for caught exceptions because each throw in your code causes the debugger to pause. The default limit is is 50 captures per second.

If you enable debug output, do you see any relevant messages? When rate limiting kicks in we log this.

Sentry.init({
  dsn: "https://[email protected]/0",
  debug: true,
}); 

I'm not sure if I want to enable that in my non-local environments. Does the debug only add extra logging?

[timfish]

debug: true simply enables console logging from the Sentry SDK.

[ruohola]

Enabled debug: true, nothing from Sentry comes up in my logs. Still cannot see any local variables on the dashboard (screenshot cut not to reveal further code):

I'm 99% sure the rate-limiting is not causing this, my code is not exception heavy at all.

[timfish]

Hi @ruohola, Can you open a new issue detailing which SDKs you're using and the details to reproduce!

[ruohola]

Hi @ruohola, Can you open a new issue detailing which SDKs you're using and the details to reproduce!

It's the same exact issue.. Can we just reopen this?

[ruohola]

Updated the original description to include the attempted changes.

[lforst]

@ruohola We are actively trying to reproduce the issue you are facing but are not successful. I opened a PR with an E2E test (a test that tests the entire pipeline of ours - installing the SDK, building a TS express app, initializing the SDK, throwing an error uncaught and caught, checking the event in-flight and on Sentry) and it verifies the correct behavior.

Would you mind cross-checking your implementation with the test and let us know if there is a significant difference in implementation we should test for? https://github.com/getsentry/sentry-javascript/pull/9278/files#diff-7b8b9df985e616ee39a7c592ac07b74fe290a317311535fcad61a2a8c95194baR15

[klaa97]

I am experiencing a similar problem.

Environment:

• Sentry: 7.74.0
• Node: 20.10

To reproduce, I am capturing exceptions by passing the exception to a higher-level middleware, and doing:

const scope = new sentry.Scope();
// some scope stuff
sentry.captureException(exception, scope);

I enabled all the integrations and followed the discussion above, but it doesn't work. Do you have suggestions?

[AbhiPrasad]

Hi @klaa97, are you using esm or cjs? How are you configuring your Sentry.init call?

[klaa97]

@AbhiPrasad

I am using CJS. Also, Sentry.init is like this

sentry.init({
  environment: "myEnvironment",
  includeLocalVariables: true,
  normalizeDepth: 5,
  maxValueLength: 2000,
  integrations: [
    new RewriteFrames({ root: path.resolve("./dist") }),
    new ExtraErrorData({ depth: 5 }),
  ],
})

There is no need to manually add new Sentry.Integrations.LocalVariables, as the SDK should automatically add it. Moreover, captureAllExceptions is by default True in the new versions...

[timfish]

We have integration tests that validate that local variables are correctly attached but they are only basic use cases and these might not cover everything.

We're fetching the local variables via the inspector API and it sounds like there might be some cases where the debugger objects don't match our expectations. A minimal code snippet that reproduces the issue would help confirm this and can be added to the test cases.

Another thing worth considering is that local variable capture is rate limited due to the potential performance impact of debugger pausing. If you init Sentry with debug: true, rate limiting events are logged to the console to help highlight this.

[klaa97]

I already checked with debug: true, the issue is not the rate-limiting :( . I will try to add a test-case in the integration tests, thank you for the context and explanation!

[kantuni]

+1

[Lms24]

If anyone can provide a minimal reproduction (e.g. GH repo/stackblitz) we can debug this further. Thank you!

[maximlopin2]

Maybe someone has an example of code that works with includeLocalVariables?