Sourcecode not visible when using PyInstaller

[jaymegordo]

Important Details

How are you running Sentry? Frozen python app created with PyInstaller

On-Premise w/ Docker, version x.y.z

Saas (sentry.io)

Other, version x.y.z, [briefly describe your environment]

Description

When my app is frozen with pyinstaller, the error context lines are not captured.

Added a couple print statements in sentry_sdk/utils.py:

def get_lines_from_file(
    filename,  # type: str
    lineno,  # type: int
    loader=None,  # type: Optional[Any]
    module=None,  # type: Optional[str]
):
    # type: (...) -> Tuple[List[Annotated[str]], Optional[Annotated[str]], List[Annotated[str]]]
    context_lines = 5
    source = None
    if loader is not None and hasattr(loader, "get_source"):
        try:
            source_str = loader.get_source(module)  # type: Optional[str]
        except (ImportError, IOError):
            source_str = None
        if source_str is not None:
            source = source_str.splitlines()

    print(f'filename: {filename}, module: {module}')
    print(f'source is None: {source is None}')
    if source is None:
        try:
            source = linecache.getlines(filename)
            print(f'source: {source}')
        except (OSError, IOError):
            return [], None, []

and at the end of sentry_sdk\client.py at line 216 in _prepare_event, i just printed print(event['exception']) to see the final event exception passed out. (trimmed some extra non imporant stuff off this)

App not frozen (working):

filename: Y:\OneDrive\Python\SMS\smseventlog\gui\gui.py, module: smseventlog.gui.gui
source is None: False

{'filename': 'smseventlog\\gui\\gui.py', 
'abs_path': 'Y:\\OneDrive\\Python\\SMS\\smseventlog\\gui\\gui.py', 
'function': 'test_error', 
'module': 'smseventlog.gui.gui', 
'lineno': 257, 
'pre_context': ['', '    def test_error(self):', '        a = 5', '        b = 6', ''], 
'context_line': '        return a / 0', 
'post_context': ['', '    def create_actions(self):', '        # Menu/shortcuts', '        t = self.active_table_widget', ''], 
'vars': {'self': '<smseventlog.gui.gui.MainWindow object at 0x000002CD9D0DFAF0>', 'a': '5', 'b': '6'}, 
'in_app': True}]}

App frozen (not working):

filename: smseventlog\gui\gui.py, module: smseventlog.gui.gui
source is None: True
source: []

 {'filename': 'smseventlog\\gui\\gui.py', 
'abs_path': 'C:\\Users\\Jayme\\Documents\\smseventlog\\dist\\smseventlog_win\\smseventlog\\gui\\gui.py', 
'function': 'test_error', 
'module': 'smseventlog.gui.gui', 
'lineno': 257,
 'pre_context': [], 
'context_line': None, 
'post_context': [], 
'vars': {'self': '<smseventlog.gui.gui.MainWindow object at 0x0000022638C4A280>', 'a': '5', 'b': '6'}, 
'in_app': True}]}}]}

the difference seems to be that in the non frozen app, the full filename is recognized Y:\OneDrive\Python\SMS\smseventlog\gui\gui.py, while in the frozen app its only recoginzed as smseventlog\gui\gui.py and the module can't be propery read to find the lines specified in the traceback.

Steps to Reproduce

What you expected to happen

Pre and post context should be captured, even in a frozen app.

Possible Solution

[If you have an idea on how this could be solved include that detail here.]

[untitaker]

That's a known issue in the Python SDK. Moving there.

[jaymegordo]

Thanks @untitaker, in the meantime do you know the best way to include extra data say a stacktrace or whatever else by using eg capture_exception(exception, extra_data='extra stuff here' ?

[untitaker]

Please refer to the documentation: https://docs.sentry.io/platforms/python/enriching-error-data/additional-data/#adding-context

[jaymegordo]

Any progress on this one? would be super helpful to have working thx!!! :)

[becurrie]

Seconding @jaymegordo, having a fix/solution for this would be very beneficial :)

[becurrie]

@jaymegordo @untitaker After reviewing the source-code, I don't think this will be very easy to handle... The file itself in a frozen executable is in relation to the directory it's in (see relevant pyinstaller docs) rather than the absolute path (which is required to grab source code lines). Additionally, even if we can find the file and open it for viewing, it's going to be a .pyc file which will not contain the same exact lines as our normal .py files we want to send stack trace information about (I suppose some de-compilation could be performed if we have the file to get at least something similar to our Python code?) I think a proper fix for this would end up being a pretty big change to the source code and how it handles this edge case.

The best I was able to come up with for the time being is to use the before_send hook (docs) in conjunction with the traceback module in Python, I'm able to do something like this:

def before_send(event, hint):
    event["extra"]["exception"] = ["".join(
        traceback.format_exception(*hint["exc_info"])
    )]
    return event

And my sdk init call:

sentry_sdk.init(
    dsn=__SENTRY_DSN__,
    release=__SENTRY_RELEASE__,
    before_send=before_send,
)

This works well enough so that I at least have some information available in the Sentry logs.

Edit The photo above is actually from a normal .py file sentry event exception being captured. Using a frozen executable results in a Sentry event more like this:

@untitaker Is this a appropriate enough solution for the issue? Or is there something else kicking about in Sentry that could handle this? I only started using Sentry the other day, so a lot of it is still new to me!

[untitaker]

I'm surprised to see that there's any information at all reachable from within the Python process. We use our own "formatting function" on exception, I wonder where format_exception gets this extra info from?

[becurrie]

@untitaker Actually, I just realized I took my screenshot from my .py file sentry event, rather than my frozen executable... The information provided in the additional data "exception" looks like this. I'll update my photo above as well.

Frozen executables do display some useful information, notably though, it removes the line that actually caused the exception being sent along in the event.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[antonpirker]

There is also some information about PyInstaller in this issue: https://github.com/getsentry/sentry-python/issues/711

[antonpirker]

I will add this to the internal low priority list. In the next couple of weeks we will do proper roadmap planing and decide if we can implement this or not. But I think this is not very far up our very long list of things to implement.

So @jaymegordo or @becurrie is one of you motivated to look into this and maybe provide a PR with improvements?

[rcoup]

Coming at this from a slightly different angle... In Sentry JS there's source maps, and the Sentry server can go from the compiled traceback to the original source... is there a way that approach could be applied with respect to Python .pyc content?

• at build/release time, upload source .py files to sentry
• distribute .pyc files to users either via pyinstaller or other mechanism
• exception happens; sentry-python gets source module; line number; etc which are in the .pyc
• when the error gets to the sentry server the associated source .py code is matched and displayed in the UI

[sentrivana]

Since this hasn't had any activity for a while and it's kind of low on our priority list, I've closed the issue. If someone feels like tackling it themselves feel free to reopen.

[felixbrucker]

I just stumbled upon this issue, is it possible to link the source code of a python app on github to show relevant source code based on the tagged release of the code?

For example:

• i have linked my github repo and created a source mapping on sentry
• i created a tag on github named 1.2.3
• a pyinstaller packaged app with release set to [email protected] crashes and generates a sentry issue
• as the file and line number are there, could it not pull in the source from github for that release?

i was hoping this would work out of the box, but alas im either doing something wrong or its not possible

[sentrivana]

Hey @felixbrucker, I don't know how exactly the GH code linking works since this happens outside of the SDK in Sentry itself, but I can imagine based on this thread that errors sent from a PyInstaller packaged app might be missing something we require server-side for matching to the correct GH code context. If you followed https://docs.sentry.io/product/integrations/source-code-mgmt/github/#stack-trace-linking and it doesn't work, that'd probably be the most likely reason.

I'll reopen this, but please everyone note that the priority has unfortunately not changed and we will not get to this anytime in the near future. PRs are, as always, most welcome.

[felixbrucker]

Yeah, seems like it just does not work

[antonpirker]

Yes, this does not work right now. @rcoup idea would be the correct solution. But we will not support source maps in the Python SDK for now. Sorry.