VLC 3.0.12.1 (silicon) & libvlccore.dylib issues

[ben-hearn-sb]

Hello!

So I have a software project I am developing that uses this fabulous python-vlc module. I am packaging the application up using PyInstaller.

When some users try to boot the app they get the following exception in the logs:

Uncaught exception
Traceback (most recent call last):
  File "PyInstaller/loader/pyimod04_ctypes.py", line 53, in __init__
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 374, in __init__
OSError: dlopen(/Applications/VLC.app/Contents/MacOS/lib/libvlccore.dylib, 6): no suitable image found.  Did find:
	/Applications/VLC.app/Contents/MacOS/lib/libvlccore.dylib: mach-o, but wrong architecture
	/Applications/VLC.app/Contents/MacOS/lib/libvlccore.9.dylib: mach-o, but wrong architecture

At first I thought this was Monterrey 12.0.1 being the problem or an update to Big Sur perhaps but I have test cases on the following architectures which indicate that it is VLC itself and the M1 that is the problem:

Big Sur 11.6, VLC 3.0.16 Vetinari (Intel 64bit) - works fine Big Sur 11.4, VLC 3.0.12.1 Vetinari (Apple Silicon) - throws above exception Monterrey 12.0.1, VLC 3.0.12.1 Vetinari (Apple Silicon) - throws above exception

Do we know if VLC are aware of these issues or is this something the python-vlc module can address?

Cheers,

Ben

[mrJean1]

Before investigating this further, one question. Does python-VLC work properly with VLC in all 3 cases without PyInstaller? If so, it is possibly a PyInstaller or ctypes issue.

[mrJean1]

After installing PyInstaller 4.8 into Python 3.10.2 (from Python.org) on a M1 Macbook Air running macOS 12.1 Monterey the cocoavlc.py example was used to build a single-file, windowed application cocoavlc.app as shown further below.

The cocoavlc.app runs without any trouble on the same Macbook Air, just like the cocoavlc.py script, plays the same videos, has the same menus and shortcuts, shows the same dialogs, etc. Note, cocoavlc[.py] has one dependency and needs PyCocoa to be installed.

Notice the versions of vlc.py 3.0.12119 and libVLC 3.0.16 and that Python 3.10.2 is running on arm64, meaning Apple Silicon natively. Both (lib)VLC and Python 3.10.2 and so-called universal2 builds for macOS.

However, it is unclear from the build log whether the cocoavlc.app contains the libVLC.dylib library in its bundle. From the plugins: ... line below, it seems to suggest that the cocoavlc.app loads that library from the /Applications/VLC/Contents/MacOS/... directory, just like the cocoavlc.py script. And if so, that would be another, separate issue, for example when moving the cocoavlc.app to a different macOS system or to one with a different /Application/VLC/... release installed.

% ./dist/cocoavlc -v   # this is the command line version
cocoavlc.py: 21.11.02 (pycocoa 21.11.04 macOS 12.1 arm64)
vlc.pyc: 3.0.12119 (Mon May 31 18:25:17 2021 3.0.12)
libVLC: 3.0.16 Vetinari (0x3001000)
plugins: /Applications/VLC/Contents/MacOS/plugins
Python: 3.10.2 (64bit) macOS 12.1 arm64

For comparison:

% python3 ./cocoavlc.py -v
cocoavlc.py: 21.11.02 (pycocoa 21.11.04 macOS 12.1 arm64)
vlc.py: 3.0.12119 (Mon May 31 18:25:17 2021 3.0.12)
libVLC: 3.0.16 Vetinari (0x3001000)
plugins: /Applications/VLC/Contents/MacOS/plugins
Python: 3.10.2 (64bit) macOS 12.1 arm64

Building the cocoavlc application:

% pyinstaller ./cocoavlc.py --onefile --windowed
28 INFO: PyInstaller: 4.8
29 INFO: Python: 3.10.2
34 INFO: Platform: macOS-12.1-arm64-arm-64bit
34 INFO: wrote .../PyCocoa/cocoavlc.spec
36 INFO: UPX is not available.
36 INFO: Extending PYTHONPATH with paths
['.../PyCocoa']
97 INFO: checking Analysis
102 INFO: checking PYZ
103 INFO: EXE target arch: arm64
103 INFO: Code signing identity: None
104 INFO: checking PKG
104 INFO: Building because toc changed
104 INFO: Building PKG (CArchive) cocoavlc.pkg
1862 INFO: Building PKG (CArchive) cocoavlc.pkg completed successfully.
1863 INFO: Bootloader /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/PyInstaller/bootloader/Darwin-64bit/runw
1863 INFO: checking EXE
1863 INFO: Building because name changed
1863 INFO: Building EXE from EXE-00.toc
WARNING: The output directory ".../PyCocoa/dist/cocoavlc" and ALL ITS CONTENTS will be REMOVED! Continue? (y/N)y
On your own risk, you can use the option `--noconfirm` to get rid of this question.
8990 INFO: Removing dir .../PyCocoa/dist/cocoavlc
9001 INFO: Copying bootloader EXE to .../PyCocoa/dist/cocoavlc
9003 INFO: Converting EXE to target arch (arm64)
9019 INFO: Removing signature(s) from EXE
9034 INFO: Appending PKG archive to EXE
9038 INFO: Fixing EXE headers for code signing
9044 INFO: Re-signing the EXE
9083 INFO: Building EXE from EXE-00.toc completed successfully.
9084 INFO: checking BUNDLE
9084 INFO: Building BUNDLE because BUNDLE-00.toc is non existent
9085 INFO: Building BUNDLE BUNDLE-00.toc
9087 INFO: Moving BUNDLE data files to Resource directory
9087 INFO: Signing the BUNDLE...
9107 INFO: Building BUNDLE BUNDLE-00.toc completed successfully.

[ben-hearn-sb]

I had suspected at first it was a PyInstaller issue due to the traceback that the error I posted actually triggers but I jave never had to package the vlc dylib with the app and the path that it is looking for is a general system path.

I am on PyInstaller 4.7 (I will upgrade and test this weekend) I do not bundle any plugins or binaries related to vlc when running the build (never needed to thus far) Python 3.9.? (not sure the exact version)

Could this be an issue of building the app within the actual M1 architecture? I am using an older OS on the intel chipset.

EDIT: Another user reports similar issues with another modules dylibs via PyInstaller , I will get a friend to build the app on his M1 and see if that works https://github.com/pyinstaller/pyinstaller/issues/6410

File "/Users/benhearn/Documents/etrax/venv/lib/python3.9/site-packages/vlc.py", line 210, in <module>
  File "/Users/benhearn/Documents/etrax/venv/lib/python3.9/site-packages/vlc.py", line 182, in find_lib
  File "PyInstaller/loader/pyimod04_ctypes.py", line 55, in __init__
pyimod04_ctypes.install.<locals>.PyInstallerImportError: Failed to load dynlib/dll '/Applications/VLC.app/Contents/MacOS/lib/libvlccore.dylib'. Most likely this dynlib/dll was not found when the application was frozen.

Here is my build script:

import PyInstaller.__main__
import os
import etrax.config as cfg

current_dir = os.path.dirname(os.path.dirname(__file__))

PyInstaller.__main__.run([
    '--name=%s' % f'_eTrax_mac {cfg.VERSION}',
    '--distpath=%s' % '/Users/benhearn/Documents/etrax_build/mac/dist',
    '--workpath=%s' % '/Users/benhearn/Documents/etrax_build/mac/build',
    '--paths=%s' % '/Users/benhearn/Documents/etrax/etrax/etrax',
    '--add-data=%s' % '/Users/benhearn/Documents/etrax/etrax/etrax/icons/*.png:etrax/icons',
    '--noconfirm',
    '--onedir',
    '--debug=all',
    '--windowed',
    '--osx-bundle-identifier=%s'%'com.ioproject.software.eTrax',
    '--manifest=None',
    '--icon=%s' % '/Users/benhearn/Documents/etrax/etrax/etrax/icons/app_icon.icns',
    os.path.join(current_dir+'/etrax/', 'main.py')
])

[mrJean1]

AFAIK, by default Pyinstaller builds an app bundle only for one architecture. The cocoavlc.app mentioned above did run on the M1 Macbook Air, but not on an Intel Macbook (with macOS 11.6.1 Big Sur).

After rebuilding the cocoavlc.app on the M1 Macbook Air (with macOS 12.1 Monterey) with additional command line option --target-arch=universal2, the resulting windowed app runs just fine on Intel (running macOS 11.6.1 Big Sur) and M1. Same for the non-windowed cocoavlc app.

Both Macbooks have standard VLC 3.0.16 installed, but the -arm64 release on the M1 and the -intel64 one in the Intel-based machine. Also, this is cocoavlc.py 21.11.02, PyCocoa 21.11.04, vlc.py 3.0.12119, PyInstaller 4.8 and this Python 3.10.2 macOS 64-bit universal2, all on M1.

Obviously, building universal2 apps is only needed if the app is intended to run on both Apple Silicon and Intel-based machines.

[mrJean1]

Three comments on your build script.

1. Make sure all 3 machines have the same, preferably the latest PyInstaller, Python, VLC and the compatible python-vlc vlc.py releases.

2. The error message ...

File "/Users/.../python3.9/site-packages/vlc.py", line 210, in <module>
  File "/Users/.../python3.9/site-packages/vlc.py", line 182, in find_lib
  File "PyInstaller/loader/pyimod04_ctypes.py", line 55, in __init__ ...

is a concern. It looks like PyInstaller's own ctypes version is unable to find the libVLC library. This is most likely a macOS issue not handled in older versions of python-vlc. Use vlc.py release 3.0.6 or any later one (with the corresponding VLC app).

3. The earlier, uncaught exception ...

  File "PyInstaller/loader/pyimod04_ctypes.py", line 53, in __init__
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 374, in __init__
OSError: dlopen(/Applications/VLC.app/Contents/MacOS/lib/libvlccore.dylib, 6): no suitable image found.  Did find:
	/Applications/VLC.app/Contents/MacOS/lib/libvlccore.dylib: mach-o, but wrong architecture
	/Applications/VLC.app/Contents/MacOS/lib/libvlccore.9.dylib: mach-o, but wrong architecture

points to bad VLC.app or one built for the wrong architecture. For example, if Python 3.9.2_1 is a Homebrew build for Apple Silicon and if the VLC.App is the Intel version, each will run just fine on an M1 machine. But building a PyStaller app with that Python 3.9.2_1 version will (try to create) an M1 version by default, no problem. But that app will not run since there's no matching VLC library. Btw, that same Python code should also fail when run from Python 3.9.2_1.

[ben-hearn-sb]

So it appears the universal2 flag wont work for my OS, however I am running 10.14.6 Mojave which according to some digging through the folliwing PyInstaller issue could be why it cannot find the FAT binaries. https://github.com/pyinstaller/pyinstaller/issues/5315

AssertionError: /usr/local/Cellar/[email protected]/3.10.1/Frameworks/Python.framework/Versions/3.10/Python is not a fat binary

During actual compilation though, the libvlccore.dylib has warnings issues about it (with or without universal2 flag). Is this a concern during the PyInstaller build process? Should I specifically include the .dylibs in the package somewhere?

Build machine, version etc. (python-vlc lates

224 INFO: PyInstaller: 4.8
224 INFO: Python: 3.10.1
234 INFO: Platform: macOS-10.14.6-x86_64-i386-64bit

VLC warning:

18661 INFO: Looking for ctypes DLLs
18736 WARNING: Library libvlccore.dylib required via ctypes not found
18736 WARNING: Library libvlc.so.5 required via ctypes not found
18736 WARNING: Library libvlc.dylib required via ctypes not found

[mrJean1]

If "fat binary" means the same as Apple term universal2 Apps, that Python 3.10.1 and PyInstaller can only create intel bundles.

PyInstaller can build universal2 bundles only with a Python release built as universal2. It is not a macOS issue, it depends solely on the Python release used to run PyInstaller. Similarly, if Python is an intel-only release, you can build PyInstaller bundles for intel, but not for M1 and not universal2 ones.

It is unlikely that including libvlccore.dylib (plus libvlc.dylib!) will work right away. The vlc.py script looks for those files using a fully qualified path, something like /Applications/VLC.App/Contents/MacOS/lib/.... That may have to be changed to make vlc.py find those libs inside the PyInstaller bundle. Or vlc.py has to be enhanced to handle running frozen**. That can be probably done, but will require some tinkering. Plus again, to build a universal2 PyInstaller bundle, the VLC.App (where both libs are taken from) must be built as universal2.

One more complication, a PyInstaller intel bundle will run on an M1 machine, no problem. However if the libs included with that are intel only, it will not. It may work if those libs are universal2, but that needs to be tested too.

HTH __ **)

    import sys
    if getattr(sys, 'frozen', False):
        # running as PyInstaller bundle
    ...

[ben-hearn-sb]

Thats strange as python 3.9.1 started shipping universal2 bundles and as of 3.10.1 it is universal2 by default. However, a lot of issues floating around the web make mention of Xcode 12 which is not supported on my OS which could be the biggest issue here. I will need to upgrade and get this tested, for now I am going to build on an M1 machine and test that out.

I have included in program a try except for vlc that disables the audio player if it errors out, so far the only lib that not working on M1 is python-vlc. I will get back with some more info once I have got an M1 build and upgraded my OS & Xcode

[mrJean1]

After tinkering with vlc.py and the PyInstaller spec to include the libvlccore.dylib and libvlc.dylib VLC files in an --onefile and --onedir bundle, the resulting apps do not work. Either PyInstaller complains about fat binaries (like you found eaerlier) or the --windowed bundle can't find the VLC files.

A non--windowed bundle does find and load the VLC files if built as --onedir, but it is unable to create a VLC media_player_new instance. Perhaps other things are needed, other than those VLC files, to make such PyInstaller bundles work. Not sure.

Another issue may be that macOS restricts running unverified or non-signed libraries. Also not sure.

In any case, the only option left is to build PyInstaller bundles without VLC libs and instead depend on VLC being installed on the underlying system.

[ben-hearn-sb]

A requirement to play audio is to have VLC installed on the system as per your comment above with regards to fixed system pathing for the VLC libraries.

My app is packaged as onedir for macOS and onefile for Windows.

[mrJean1]

Using PyInstaller on an M1 to build --windowed --target-arch=universal2 bundles with universal2 Python 3.10.2 works just fine for two python-vlc examples cocoavlc.py and tkvcl.py, both as --onedir and --onefile. All bundles run just fine on the same M1 without any problem**.

However, all bundles fail when copied to an intel Macbook. The issue is macOS can't verify the app or complains that the app is damaged. Both have probably the same root cause: missing or bad code signatures, more about that here. __ **) Except menu item snapshot does not work in cocoavlc.py on M1 before VLC 3.0.16+ and macOS 12.6.

[ben-hearn-sb]

Ok, could a signed certificate solve this problem or are we stuck for now building on an M1 for M1 machines?

[mrJean1]

A) Possibly, but code signing is not needed if the bundle only runs on the machine where it is built.

B) An intel bundle built on an intel machine may work on an M1 machine with an intel or universal2 VLC.App, but requires code signing to be tested.

[ben-hearn-sb]

Unfortunately the M1 machine has presented issues in other libraries I use for my software so I have not been able to actually build this on an M1 to test the library currently. Were you able to test the code signing solution for the library at all?

[mrJean1]

No. Code signing thru PyInstaller fails and I haven’t investigated this beyond that.