Issue: Source Maps Not Correctly Mapping Stack Trace in Sentry

[haseeb-lab]

CLI Version

2.33.1

Operating System and Architecture

• [ ] macOS (arm64)
• [ ] macOS (x86_64)
• [ ] Linux (i686)
• [x] Linux (x86_64)
• [ ] Linux (armv7)
• [ ] Linux (aarch64)
• [ ] Windows (i686)
• [ ] Windows (x86_64)

Operating System Version

Amazon Linux 2/5.9.4

Link to reproduction repository

No response

CLI Command

sentry-cli sourcemaps upload

Exact Reproduction Steps

Steps to Reproduce

Initialize Sentry:

I initialized Sentry in my Node.js project on top of index file by import sentry-instrument.js.

import * as Sentry from '@sentry/node';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  integrations: [
    Sentry.httpIntegration({ tracing: true }),
    Sentry.rewriteFramesIntegration({
      iteratee: (frame) => {
        frame.filename = frame.filename.replace(/^\//, '');
        return frame;
      }
    })
    // enable Express.js middleware tracing
  ],

  // Set tracesSampleRate to 1.0 to capture 100%
  // of transactions for performance monitoring.
  // We recommend adjusting this value in production
  tracesSampleRate: 1.0,
  // Set sampling rate for profiling - this is relative to tracesSampleRate
  profilesSampleRate: 1.0
});

Configure CodeBuild:

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 18
    commands:
      - n 18.17.1
  pre_build:
    commands:
      - echo Starting npm install
      - npm install --legacy-peer-deps && npm install --only=dev --legacy-peer-deps
      - npm i -g sequelize-cli
      - npm install -g @sentry/cli
  build:
    commands:
      - echo Starting to run migrations
      - echo Starting npm build
      - npm run build --loglevel verbose
      - npm prune --production
      - sentry-cli sourcemaps inject --org numu-04 --project numu-node-api ./dist
  post_build:
    commands:
      - echo "Uploading source maps to Sentry"
      - sentry-cli sourcemaps upload --org numu-04 --project numu-node-api ./dist --url-prefix '~/var/app/current/src'
artifacts:
  files:
    - package.json
    - package-lock.json
    - dist/**/*
    - node_modules/**/*
    - .ebextensions/**/*
    - .platform/**/*
    - .npmrc

Push the Code:

I pushed the code to the repository, triggering the pipeline in AWS CodePipeline. The pipeline runs, and the build specifications are executed in CodeBuild.

Deploy to Sentry and Elastic Beanstalk: build code was injected with debug-id and uploaded to sentry.

I created an endpoint in my application that intentionally throws an exception to test Sentry's error reporting functionality. This is how I verified that the errors are being sent to Sentry.

  /** Express Error Handler */
  api.use(
    Sentry.expressErrorHandler({
      shouldHandleError(error) {
        if (error.status > 499) {
          Sentry.captureException(error);
          return true;
        }

        return false;
      }
    })
  );

Expected Results

I expect the Sentry stack trace to accurately map to the original source code, displaying the exact line of code where the error occurred, based on the uploaded source maps. This would help in pinpointing the issue directly in the original code rather than in the build artifacts.

Actual Results

When I throw an error to Sentry via API, Stack trace only shows line number of the error correctly. However, it does not display corresponding code from original source file. Source maps are not being utilized as expected to reveal specific line of code where error occurred.

and this event's json has no debug-id (i am assuming here is the problem)

Uploaded source maps on Sentry do include this file.

I also checked uploaded code on the server, and it does have Sentry debug ID injected:

"use strict";
!function(){try{var e="undefined"!=typeof window?window:"undefined"!=typeof global?global:"undefined"!=typeof self?self:{},n=(new Error).stack;n&&(e._sentryDebugIds=e._sentryDebugIds||{},e._sentryDebugIds[n]="18a58218-****-****-****-9fc9262b7d65")}catch(e){}}();


var _interopRequireDefault = require("@babel/runtime/helpers/interopRequireDefault");
Object.defineProperty(exports, "__esModule", {
  value: true
});
exports["default"] = void 0;
var _regenerator = _interopRequireDefault(require("@babel/runtime/regenerator"));
var _asyncToGenerator2 = _interopRequireDefault(require("@babel/runtime/helpers/asyncToGenerator"));
var _classCallCheck2 = _interopRequireDefault(require("@babel/runtime/helpers/classCallCheck"));
var _createClass2 = _interopRequireDefault(require("@babel/runtime/helpers/createClass"));
var _possibleConstructorReturn2 = _interopRequireDefault(require("@babel/runtime/helpers/possibleConstructorReturn"));
var _getPrototypeOf2 = _interopRequireDefault(require("@babel/runtime/helpers/getPrototypeOf"));
var _inherits2 = _interopRequireDefault(require("@babel/runtime/helpers/inherits"));
var _utils = require("../../utils");
var _base = require("../base/base.service");
var _country = require("../country");
function _callSuper(t, o, e) { return o = (0, _getPrototypeOf2["default"])(o), (0, _possibleConstructorReturn2["default"])(t, _isNativeReflectConstruct() ? Reflect.construct(o, e || [], (0, _getPrototypeOf2["default"])(t).constructor) : o.apply(t, e)); }
function _isNativeReflectConstruct() { try { var t = !Boolean.prototype.valueOf.call(Reflect.construct(Boolean, [], function () {})); } catch (t) {} return (_isNativeReflectConstruct = function _isNativeReflectConstruct() { return !!t; })(); }
var SentryService = /*#__PURE__*/function (_BaseService) {
  function SentryService() {
    (0, _classCallCheck2["default"])(this, SentryService);
    return _callSuper(this, SentryService, arguments);
  }
  (0, _inherits2["default"])(SentryService, _BaseService);
  return (0, _createClass2["default"])(SentryService, [{
    key: "fetchSentryCountry",
    value: // this service has error in it
    // so we can throw error on sentry
    // to test its functionality
    function () {
      var _fetchSentryCountry = (0, _asyncToGenerator2["default"])( /*#__PURE__*/_regenerator["default"].mark(function _callee() {
        return _regenerator["default"].wrap(function _callee$(_context) {
          while (1) switch (_context.prev = _context.next) {
            case 0:
              _context.prev = 0;
              _context.next = 3;
              return _country.CountryService.findByPk({
                id: 0
              }).then(function (data) {
                data.map(function (country) {
                  return country.name;
                });
              })["catch"](function (error) {
                throw new _utils.AppError({
                  error: error
                });
              });
            case 3:
              _context.next = 8;
              break;
            case 5:
              _context.prev = 5;
              _context.t0 = _context["catch"](0);
              throw new _utils.AppError({
                error: _context.t0
              });
            case 8:
            case "end":
              return _context.stop();
          }
        }, _callee, null, [[0, 5]]);
      }));
      function fetchSentryCountry() {
        return _fetchSentryCountry.apply(this, arguments);
      }
      return fetchSentryCountry;
    }()
  }]);
}(_base.BaseService);
var _default = exports["default"] = new SentryService();
//# sourceMappingURL=sentry.service.js.map
//# debugId=18a58218-****-****-****-9fc9262b7d65

Sentry source map code

"use strict";
!function(){try{var e="undefined"!=typeof window?window:"undefined"!=typeof global?global:"undefined"!=typeof self?self:{},n=(new Error).stack;n&&(e._sentryDebugIds=e._sentryDebugIds||{},e._sentryDebugIds[n]="18a58218-****-****-****-9fc9262b7d65")}catch(e){}}();


var _interopRequireDefault = require("@babel/runtime/helpers/interopRequireDefault");
Object.defineProperty(exports, "__esModule", {
  value: true
});
exports["default"] = void 0;
var _regenerator = _interopRequireDefault(require("@babel/runtime/regenerator"));
var _asyncToGenerator2 = _interopRequireDefault(require("@babel/runtime/helpers/asyncToGenerator"));
var _classCallCheck2 = _interopRequireDefault(require("@babel/runtime/helpers/classCallCheck"));
var _createClass2 = _interopRequireDefault(require("@babel/runtime/helpers/createClass"));
var _possibleConstructorReturn2 = _interopRequireDefault(require("@babel/runtime/helpers/possibleConstructorReturn"));
var _getPrototypeOf2 = _interopRequireDefault(require("@babel/runtime/helpers/getPrototypeOf"));
var _inherits2 = _interopRequireDefault(require("@babel/runtime/helpers/inherits"));
var _utils = require("../../utils");
var _base = require("../base/base.service");
var _country = require("../country");
function _callSuper(t, o, e) { return o = (0, _getPrototypeOf2["default"])(o), (0, _possibleConstructorReturn2["default"])(t, _isNativeReflectConstruct() ? Reflect.construct(o, e || [], (0, _getPrototypeOf2["default"])(t).constructor) : o.apply(t, e)); }
function _isNativeReflectConstruct() { try { var t = !Boolean.prototype.valueOf.call(Reflect.construct(Boolean, [], function () {})); } catch (t) {} return (_isNativeReflectConstruct = function _isNativeReflectConstruct() { return !!t; })(); }
var SentryService = /*#__PURE__*/function (_BaseService) {
  function SentryService() {
    (0, _classCallCheck2["default"])(this, SentryService);
    return _callSuper(this, SentryService, arguments);
  }
  (0, _inherits2["default"])(SentryService, _BaseService);
  return (0, _createClass2["default"])(SentryService, [{
    key: "fetchSentryCountry",
    value: // this service has error in it
    // so we can throw error on sentry
    // to test its functionality
    function () {
      var _fetchSentryCountry = (0, _asyncToGenerator2["default"])( /*#__PURE__*/_regenerator["default"].mark(function _callee() {
        return _regenerator["default"].wrap(function _callee$(_context) {
          while (1) switch (_context.prev = _context.next) {
            case 0:
              _context.prev = 0;
              _context.next = 3;
              return _country.CountryService.findByPk({
                id: 0
              }).then(function (data) {
                data.map(function (country) {
                  return country.name;
                });
              })["catch"](function (error) {
                throw new _utils.AppError({
                  error: error
                });
              });
            case 3:
              _context.next = 8;
              break;
            case 5:
              _context.prev = 5;
              _context.t0 = _context["catch"](0);
              throw new _utils.AppError({
                error: _context.t0
              });
            case 8:
            case "end":
              return _context.stop();
          }
        }, _callee, null, [[0, 5]]);
      }));
      function fetchSentryCountry() {
        return _fetchSentryCountry.apply(this, arguments);
      }
      return fetchSentryCountry;
    }()
  }]);
}(_base.BaseService);
var _default = exports["default"] = new SentryService();
//# sourceMappingURL=sentry.service.js.map
//# debugId=18a58218-****-****-****-9fc9262b7d65

I initially tried using Sentry SDK version 7, but it did not work as expected. I then migrated to SDK version 8, yet I am encountering the same issue with stack trace mapping.

Logs

I'll share the full log output.

[szokeasaurusrex]

and this event's json has no debug-id (i am assuming here is the problem)

Yes, missing Debug ID in the event can certainly explain the behavior you are observing here.

Usually this occurs when the code you are running when testing your app does not have the Debug IDs injected. From the snippet you provided, though, it does indeed look like the Debug IDs are there.

Are you absolutely certain that the code you are running when testing the app out is the code that has the Debug IDs injected? And, did you inject Debug IDs into all of your JavaScript code, not just this one file?

You could also try uploading sourcemaps using one of our bundler plugins. These might be easier to setup than the Sentry CLI

[haseeb-lab]

Thank you for your prompt reply.

I want to confirm that I am 100% sure that the debug ID has been injected by the CLI, and the same files were uploaded to both Sentry and our servers.

I suspect that the issue might be related to pm2-runtime, which we are using to run our server. pm2-runtime utilizes source-map-support for logging purposes, and unfortunately, it does not have a flag to disable source-map-support. According to the documentation, the --disable-source-map-support flag is available for pm2, but not for pm2-runtime.

Here is how I start my server:

{
"start": "./node_modules/pm2/bin/pm2-runtime ./dist/index.js -i max && npm run pm2:logs",
"pm2:logs": "node ./node_modules/pm2/bin/pm2 logs"
}

I attempted to run the server with a simple node command and it worked !!!!!!!! , but I want to use pm2 for process management.

Additionally, I found a related issue in the PM2 GitHub repo that describes a similar problem: https://github.com/Unitech/pm2/issues/5003

It seems that the stack trace is being altered by pm2-runtime, and Sentry is unable to process it correctly, causing issues with attaching debug_meta in the event.

One more thing: I’ve read every Sentry blog and checked replies from the Sentry team on GitHub issues related to this problem. In most cases, the response from you guys is always same that the source maps uploaded to Sentry are not the same as those on the server. Why is there such certainty about this being the issue? The problem could be something else, just like in my case.

[lforst]

@haseeb-numu In case pm2 messes with Error.stack in any way, source mapping with Sentry will not work. The SDK relies on the stack trace to be correct and original. If it is altered in a meaningful way, source maps in Sentry will break.

[haseeb-lab]

Have you guys considered rethinking the logic of attaching debug_id to events? It seems there are ongoing issues with integrations, such as with Cordova and now PM2 runtime.

[lforst]

@haseeb-numu Not really no. Messing with Error.stack is bound to run into problems because it alters native behaviour that other tools like Sentry depend on. I think it is better to raise this issue with the maintainers PM2 and so on.

[szokeasaurusrex]

From the discussion above, it seems that this issue is originating from outside Sentry CLI, so I am closing this issue.

[net]

You can disable pm2's source map support even when using pm2-runtime by using an ecosystem.config.js file and setting disable_source_map_support: true.