Dev console errors are thrown to sentry

[markivancho]

Is there an existing issue for this?

• [X] I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• [X] I have reviewed the documentation https://docs.sentry.io/
• [X] I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which package are you using?

@sentry/react

SDK Version

6.19.7

Framework Version

No response

Link to Sentry event

https://sentry.io/organizations/matic/issues/3311552008/events/1599a2d013f14bb7a76d1f4cb9028cbc/?project=5376741

Steps to Reproduce

Only Chrome 102, older chrome versions don't trigger it, other browsers uncheked

1. Go to website with initialized sentry
2. Open dev console
3. Run copy('any string')

Expected Result

"any string" is copied to clipboard, no errors triggered to sentry

Actual Result

"any string" is copied to clipboard, few errors triggered to sentry

Generally, all the errors that appear in console is triggered to sentry(

[Lms24]

Hi @markivancho and thanks for reporting! I was able to reproduce this in the SDK version 7.0.0 with Chrome 102, too. I'll bring it up with the team and we'll look into it.

[markivancho]

@Lms24 thanks for the follow-up! Looking forward for the fix, either from chromium or you :)

[Lms24]

Hi @markivancho, we found the reason why these errors are suddenly thrown. It's Chromium related starting with 102 (as you reported) and was possibly introduced here. It was fixed a few days later but the fix isn't yet in a stable chromium version which is why we're gonna get quite a lot of events like these (there's a couple more events we found related to Chromium eagerly parsing console input). If you install the latest chromium canary version and run your reproduction example, you shouldn't get those errors anymore.

We're currently investigating if there's a way of automatically filtering out those errors.

[Lms24]

Hey @markivancho (and everyone else who's getting these errors),

we're still discussing how to best filter such browser-caused errors in the future automatically. In the meantime, you have a couple of options on how to filter these events out manually:

Using beforeSend

You can add a filter criterion for beforeSend in Sentry.init. Something along these lines:

1. Filter out all events from the buggy browser version:

import { Event as SentryEvent } from "@sentry/<yourSDK>";

beforeSend: (event: SentryEvent) => {
  const isBuggyBrowser = navigator.userAgent?.includes("Chrome/102");

  if (isBuggyBrowser) {
    return null;
  }
  return event;
}

2. Be more specific and filter for exception type and values:

import { Event as SentryEvent } from "@sentry/<yourSDK>";

beforeSend: (event: SentryEvent) => {
  const isBuggyBrowser = navigator.userAgent?.includes("Chrome/102");
  const error = event.exception?.values?.at(0);
  const isKnownBrowserBug =
    isBuggyBrowser &&
    (error?.type === "EvalError" &&
      error?.value?.includes("Possible side-effect in debug-evaluate")) ||
    (error?.type === "SyntaxError" &&
      (error?.value?.includes("Unexpected end of input") ||
        error?.value?.includes("Invalid or unexpected token") ||
        error.value?.includes("missing ) after argument list")));
  
  if (isKnownBrowserBug) {
    return null;
  }
 
  return event;
}

Using ignoreErrors

Alternatively, if you do not care about browser versions, you can go a simpler route and use the ignoreErrors property in Sentry.init. Note though, that this does not take the user agent into account at all and will thus ignore errors from all browsers:

ignoreErrors: [
  "Possible side-effect in debug-evaluate",
  "Unexpected end of input",
  "Invalid or unexpected token",
  "missing ) after argument list",
]

Using Inbound Filters in Sentry UI

We recognize that these approaches require a redeployment of your app with the changed SDK config. If this is not an option, you can do something similar to ignoreErrors above in the Sentry UI: In your project's settings, go to "Inbound Filters" and at the bottom you can configure a filter for error messages.

Again, this option filters out errors of all browsers, so similariy to ignoreErrors it cannot be applied to a specific browser (version).

[markivancho]

@Lms24 thanks for providing possible solutions, very much appreciated

[firasdib]

I don't think this is a bug, but rather a "feature": https://bugs.chromium.org/p/chromium/issues/detail?id=1295750

This behavior is still exhibited in version 103.

[Lms24]

Admittedly it's been a while since I looked into this but I think the "feature" is that errors from the console are thrown to window.onerror. The "bug" though is that errors during the eager evaluation of expressions while users are typing in the console are also thrown to window.onerror. This should be fixed in version 104.

[Lms24]

I'm going to close this issue for the time being, since it seems as if the provided workaround works well enough. Feel free to reach out/comment here if you have additional questions or concerns.

[firasdib]

@Lms24 Agreed, the eager evaluation part is solved, but the larger issue is that errors from input in the dev console trigger the onerror handler of sentry. I have 99% errors in my sentry list from users trying things out in their console while on my website, and has practically rendered Sentry completely useless for me. I have to sift through thousands of errors in order to find a "real" one.

If this can't be resolved, I'll just have to skip Sentry altogether, which would be a shame, since its been immensely useful.

Edit: The provided solutions do not work for me, as the errors thrown are not deterministic. Some are syntax errors, some are variables being redefined, etc etc.

[Lms24]

I have 99% errors in my sentry list from users trying things out in their console

This is a pretty good reason to reopen this issue ;) Thanks for letting us know about it. Okay, so I think we need to think about strategies to filter out console errors thrown to window.onerror. Will reopen and backlog this again and bring your case up with our team. Let's see what we can do here!

To everyone else reading this message: If you would like to see this implemented, please react to this answer. Also, PRs are more than welcome!

[firasdib]

@Lms24 Thank you.

I'll dump some of my knowledge here in case it helps anyone. It is not possible to (reliably) detect if the dev console is open, so I would not explore that avenue any further. And, as far as I know, the errors are not tagged/differentiated in any particular way when they come from input in the dev console.

The stack trace does not necessarily relay any meaningful information: 'Error: foo\n at <anonymous>:1:1'

[kubajmarek]

Just checked this SO solution and it appeared to be working in all 3 Chrome/Safari/Firefox and trigger switching to Console tab, but I didn't explore the reliability.

Checking if console is open before every event might not be enough, so I'm considering some way track it and toggle Sentry off entirely, even if Console gets closed. 🤔

[AbhiPrasad]

Since the errors are not deterministic, there’s no good way for us to tell what is an actual error and what is not.

One thing you can choose to do to cut the bleeding for now is to disable instrumenting window.onerror on browsers that produce the error or maybe if the console is open - or both!

[lobsterkatie]

Chatted with the team about this and we agreed it's worth doing a little more research here (to see if we can hack a way to tell which errors are which), and if not, we'll file an issue with the Chrome dev team and see what they say. We've pulled it out of the backlog and put it on our active to-do list.

If this can't be resolved, I'll just have to skip Sentry altogether, which would be a shame, since its been immensely useful.

As has regex101 for any number of us! We'll see what we can do here.

[Lms24]

Hi @firasdib @Irevall and everyone else,

we investigated console errors a little more and we mostly came to the same conclusions. Unfortunately, there is no bullet-proof way of detecting if an error came from the dev console.

While toying around with the console, we detected some patterns that basic console errors have in common. For example, they mostly have a stacktrace with only one frame that is anonymous or whose file name is equal to the URL of the current page. It does however depend a lot on the type of error (syntax vs. type and reference errors). So with this information, we went ahead and hacked together a filter that is likely to filter out simple console errors:

Sentry.init({
  // ...
  beforeSend: (event: SentryEvent) => filterConsoleErrors(event),
});

function filterConsoleErrors(event: SentryEvent): SentryEvent | null {
  const originalException = event.exception?.values?.[0];

  // Console errors appear to always bubble up to `window.onerror` and to be unhandled.
  // So if, we don't have the original exception or the mechanism looks different,
  // we can early return the event.
  // (Note, this might change depending on the used framework, so feel free to remove this check.)
  if (
    !originalException ||
    !originalException.mechanism ||
    originalException.mechanism.type !== "onerror" ||
    originalException.mechanism.handled
  ) {
    return event;
  }

  const stackframes = originalException.stacktrace?.frames;
  const errortype = originalException.type?.toLowerCase();

  // If we don't have any information on error type or stacktrace, we have no information about the error
  // this is unlikely to happen but it doesn't appear to happen in console errors.
  // Hence, we can early return here as well.
  if (!stackframes || !errortype) {
    return event;
  }

  // For simple console errors (e.g. users just typing a statement they want evaluated)
  // the stacktrace will only have one frame.
  // This condition will not catch errors that would be thrown if users type in multi-line
  // statemets. For example, if they define a multi-line function.
  // You can try experimenting with this number but there's little guarantee that the other
  // conditions will work. Nevertheless, the checks below also work with multi-frame stacktraces.
  const hasShortStackTrace = stackframes.length <= 1;

  if (!hasShortStackTrace) { 
    return event;
  }

  switch (errortype) {
    case "typeerror":
    case "referenceerror":
      const hasOnlySuspiciousTypeRefFrames = stackframes.every(
        (frame) =>
          frame.function === "?" &&
          frame.lineno === 1 &&
          frame.filename === "<anonymous>"
      );
      if (hasOnlySuspiciousTypeRefFrames) {
        console.log("I found a console type or ref error. Dropping it!");
        return null;
      }
      break;

    case "syntaxerror":
      // For some reason, syntax errors have the current URL as a filename
      const url = window.location.href;
      const hasOnlySuspiciousSyntaxFrames = stackframes.every(
        (frame) =>
          frame.function === "?" && frame.lineno === 1 && frame.filename === url
      );
      if (hasOnlySuspiciousSyntaxFrames) {
        console.log("I found a console syntax error. Dropping it!");
        return null;
      }
      break;
  }

  return event;
}

Please note that this is not a perfect solution. There might be some weird edge cases in which we'd filter out actual non-console errors but given the conditions it's rather unlikely I'd say. Also, these filters don't work well for multi-line console statements (where the error might not be in the first line). However, our best guess is that users in the console usually just try out simple stuff but this is only an assumption ofc.

If you want, feel free to give this a try. If it turns out to work really well and you get rid of the noise, we might create an intergation from this code. However, let's first see if it's worth doing so. In case it doesn't help with the noise, would you mind sharing a few errors so that we can get a sense of what else to filter on?

Regardless, the best way forward would be to get some sort of flag or indication from the browser that the error came from the console. We'll therefore open a Chromium issue to ask for this because it is a real problem for our users. Let's see if the team gets back to us.

Hope this helps a little!

[firasdib]

@Lms24 Thank you for the reply, you arrived at the same conclusion (and similar solution) as I.

I will integrate this into regex101 and deploy to see if it helps reduce the noiise.

Here's a couple of typical errors I'm getting (raw stacktrace attached):

SyntaxError: Unexpected end of input
  at ? (https://regex101.com/:1:63)

EvalError: Possible side-effect in debug-evaluate
  at ? (https://regex101.com/:0:0)

SyntaxError: missing ) after argument list
  at ? (https://regex101.com/:1:41)

SyntaxError: Invalid regular expression: missing /
  at ? (https://regex101.com/:1:13)

SyntaxError: Unexpected identifier
  at ? (https://regex101.com/:1:3)

SyntaxError: Unexpected string
  at ? (https://regex101.com/:1:3)

SyntaxError: Unexpected token ':'
  at ? (https://regex101.com/:13:10)

SyntaxError: Unexpected token '/'
  at ? (https://regex101.com/:1:12)

SyntaxError: Unexpected token '}'
  at ? (https://regex101.com/:3:42)

SyntaxError: Unexpected token '.'
  at ? (https://regex101.com/:1:15)

ReferenceError: login is not defined
  at ? (<anonymous>:2:27)

I would support raising an issue with the Chromium team.

[Lms24]

Thanks for the examples. I think most of them should be caught by the solution above. The EvalError: Possible side-effect in debug-evaluate could still be from the eager input bug that was fixed in Chromium 104 (workaround for this one is above). That leaves us with the errors having a higher line number than 1. You could experiment with loosening the filter criterion on the line number in the solution above but I can't guarantee that this won't cause false positives.

I would support raising an issue with the Chromium team.

Just opened up an issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1350066

[firasdib]

Great work! I'll publish a modified version of your code and see how it works out for me.

[firasdib]

The filter has been running for about 24h, and the issue count has dropped from 500+ to <20. The reported issues, for the most part, seem reasonable and related to my application. So for what its worth, it seems to work well.

Here is my modified version currently running, in case someone wants to use/modify it:

function filterConsoleErrors(event) {
  const originalException = event.exception?.values?.[0];

  // Console errors appear to always bubble up to `window.onerror` and to be unhandled.
  // So if, we don't have the original exception or the mechanism looks different,
  // we can early return the event.
  // (Note, this might change depending on the used framework, so feel free to remove this check.)
  if (
    !originalException ||
    !originalException.mechanism ||
    originalException.mechanism.type !== 'onerror' ||
    originalException.mechanism.handled
  ) {
    return event;
  }

  const stackFrames = originalException.stacktrace?.frames;
  const errorType = originalException.type?.toLowerCase();

  // If we don't have any information on error type or stacktrace, we have no information about the error
  // this is unlikely to happen but it doesn't appear to happen in console errors.
  // Hence, we can early return here as well.
  if (!stackFrames || !errorType) {
    return event;
  }

  // For simple console errors (e.g. users just typing a statement they want evaluated)
  // the stacktrace will only have one frame.
  // This condition will not catch errors that would be thrown if users type in multi-line
  // statements. For example, if they define a multi-line function.
  // You can try experimenting with this number but there's little guarantee that the other
  // conditions will work. Nevertheless, the checks below also work with multi-frame stacktraces.
  const hasShortStackTrace = stackFrames.length <= 2;

  if (hasShortStackTrace && isSuspiciousError(errorType) && hasSuspiciousFrames(stackFrames)) {
    console.warn('Dropping error due to suspicious stack frames.');

    return null;
  }

  return event;
}

function isSuspiciousError(errorType) {
  return ['syntaxerror', 'referenceerror', 'typeerror'].includes(errorType);
}

function hasSuspiciousFrames(stackFrames) {
  const allSuspicious = stackFrames.every(isSuspiciousFrame);

  // Certain type errors will include the thrown error message as the second stack frame,
  // but the first will still follow the suspicious pattern.

  const firstSuspicious = stackFrames.length === 2 && isSuspiciousFrame(stackFrames[0]);

  return allSuspicious || firstSuspicious;
}

function isSuspiciousFrame(frame) {
  const url = window.location.href;

  return frame.function === '?' && (frame.filename === '<anonymous>' || frame.filename === url);
}

export default filterConsoleErrors;

[Lms24]

Awesome, really glad to hear that this is working reasonably well for you! Also, thanks for providing your modified version. We'll keep an eye on the Chromium issue for now and see how things go there. In case we get a reliable way to filter out console errors, we'll integrate it into our SDK.

[jameshoward]

It looks like this is fixed in Chrome 106. I updated and now can't recreate sending errors to Sentry from the dev tools console.

[AbhiPrasad]

Awesome, closing this issue as a result, but please reach out if there is anything else.