External Packages with `data_files` are not correctly working (eg. with `pydevd-pycharm`)

[timrid]

Describe the bug

To debug a packaged application with PyCharm, the package pydevd-pycharm must be used. I can use it via briefcase dev without any problems. However, when I run briefcase run I get the following error:

Traceback (most recent call last):
  File "\app\helloworld\__main__.py", line 4, in <module>
    main().main_loop()
    ~~~~^^
  File "\app\helloworld\app.py", line 194, in main
    start_debugger()
    ~~~~~~~~~~~~~~^^
  File "\app\helloworld\app.py", line 133, in start_debugger
    import pydevd_pycharm
  File "\app_packages\pydevd_pycharm.py", line 3, in <module>
    from pydevd import settrace, stoptrace
  File "\app_packages\pydevd.py", line 16, in <module>
    from _pydevd_bundle.pydevd_collect_try_except_info import collect_return_info
ModuleNotFoundError: No module named '_pydevd_bundle.pydevd_collect_try_except_info'

If you compare the _pydevd_bundle folder of the virtual environment (.venv\Lib\site-packages\_pydevd_bundle) used by briefcase dev with the one from the packaged application (build\helloworld\windows\app\src\app_packages\_pydevd_bundle) used by briefcase run, you can see that many .py files are missing in the packaged application. There is only the data in the packaged application that is available in the virtual environment under (.venv\_pydevd_bundle), which is only the data_files.

This is due to a bug in the combination of the pip parameters --target= in combination with --upgrade, which are used here by briefcase. There is already a ticket in the pip repo for this.

I have temporary removed --upgrade at the position in briefcase and then it works as desired. Maybe it is a possibility to remove the --upgrade option from briefcase completely?

Steps to reproduce

1. Follow the BeeWare tutorial
2. Add pydevd-pycharm~=243.23654.177 to requires in pyproject.toml
3. Add the following to your code:

import pydevd_pycharm
pydevd_pycharm.settrace("localhost", port=5678, stdoutToServer=True, stderrToServer=True, suspend=False)

4. Run briefcase run

Expected behavior

Importing pydevd_pycharm should work.

Screenshots

No response

Environment

• Operating System: Windows 11
• Python version: 3.13
• PyCharm 2024.3.2 (Professional Edition)
• Software versions:
  • Briefcase: 0.3.21

Logs

Additional context

No response

[freakboy3742]

I've taken an initial look at this - and... wow Pycharm's debugger package is a hot mess.

They're distributing a package that has binary components... as a .tar.gz that contains the pre-compiled binaries, instead of as a wheel. So installing on macOS involves downloading linux and windows binary libraries.

Ok; so that's just some extra binaries that don't need to exist. Why is that a problem?

Well, on macOS, if those binaries are in an app, and they look like libraries, they need to be signed for the app to be able to run. Linux binaries have headers that make them look like valid macOS binaries, so macOS needs to be able to sign them - but they're not macOS binaries, so macOS signing tooling is unable to sign them.

It won't be possible to install the package at all on iOS. You can't use setup.py Extension compiler tooling on iOS, because the installation target (the iOS device) doesn't have a compiler. Because of this, iOS requires the use of wheels for distribution of binary content.

Your comment on #2147 suggests this works on Android... and frankly, I'm amazed. It should be choking on compile the extensions as well. If it's getting through somehow (and based on a quick read of the code, I'm not sure how), it's because Python 3.12 reports sys.platform == "Linux. That won't work on Python 3.13 or later.

With all that said - I can confirm I see the same underlying behavior on macOS as you're reporting on Windows - the _pydevd_bundle folder contains binaries, but not Python source.

Finding the --upgrade bug in pip is a nice catch - but unfortunately, removing that flag isn't an option. briefcase update -r needs to be able to upgrade packages.

To me, this is looking very much like a PyCharm issue. If they packaged their binary package as a binary package, I imagine this would work a whole lot better - and it would be a lot smaller to boot.

[mhsmith]

Finding the --upgrade bug in pip is a nice catch - but unfortunately, removing that flag isn't an option. briefcase update -r needs to be able to upgrade packages.

Doesn't it delete the directory and start from scratch every time?

[freakboy3742]

Finding the --upgrade bug in pip is a nice catch - but unfortunately, removing that flag isn't an option. briefcase update -r needs to be able to upgrade packages.

Doesn't it delete the directory and start from scratch every time?

... of course it does. 🤦

Ok, maybe it is a viable solution then.