non-x86 issue (cmsis-pack-manager dependency)

[patrislav1]

Trying to run pyocd on Raspberry Pi.

Tried installing dependencies manually and installed pyocd with --no-deps as suggested in installing_on_non_x86.md. pip3 succeeds installing, but pyocd throws DistributionNotFound at runtime, even when invoked with python3 -m pyocd (see below).

Current workaround is to install from source after removing cmsis-pack-manager from install_requires in setup.py.

$ python3 -m pyocd
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/__init__.py", line 21, in <module>
    from . import gdbserver
  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/gdbserver/__init__.py", line 17, in <module>
    from .gdbserver import GDBServer
  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/gdbserver/gdbserver.py", line 41, in <module>
    from ..rtos import RTOS
  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/rtos/__init__.py", line 30, in <module>
    load_plugin_classes_of_type('pyocd.rtos', RTOS, ThreadProvider)
  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/core/plugin.py", line 88, in load_plugin_classes_of_type
    plugin = entry_point.load()()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2410, in load
    self.require(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2433, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'cmsis-pack-manager>=0.2.7' distribution was not found and is required by the application

[tbandtg]

I am having this same problem. python -mpyocd gdbserver --pack=Cypress.PSoC6_DFP.1.1.0.pack --no-deps Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 163, in _run_module_as_main mod_name, _Error) File "/usr/lib/python2.7/runpy.py", line 111, in _get_module_details import(mod_name) # Do not catch exceptions initializing package File "/home/pi/.local/lib/python2.7/site-packages/pyocd/init.py", line 21, in from . import gdbserver File "/home/pi/.local/lib/python2.7/site-packages/pyocd/gdbserver/init.py", line 17, in from .gdbserver import GDBServer File "/home/pi/.local/lib/python2.7/site-packages/pyocd/gdbserver/gdbserver.py", line 39, in from ..rtos import RTOS File "/home/pi/.local/lib/python2.7/site-packages/pyocd/rtos/init.py", line 30, in load_plugin_classes_of_type('pyocd.rtos', RTOS, ThreadProvider) File "/home/pi/.local/lib/python2.7/site-packages/pyocd/core/plugin.py", line 88, in load_plugin_classes_of_type plugin = entry_point.load()() File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2410, in load self.require(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 2433, in require items = working_set.resolve(reqs, env, installer, extras=self.extras) File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 786, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'cmsis-pack-manager>=0.2.7' distribution was not found and is required by the application

[cederom]

This could be either pyOCD problem because it depends on CPM (CMSIS Pack Manager) [1] when it should not, or CMP problem because it still does not provides PyPI precompiled binary [2] for ARM target [3] :-)

[1] https://github.com/pyocd/pyOCD/issues/674 [2] https://pypi.org/project/cmsis-pack-manager/#files [3] https://github.com/pyocd/cmsis-pack-manager/issues/108

[quissj]

Same problem, could you please add a note on top of that page (installing_on_non_x86.md) until the issue resolved?

As patrislav1 said, installing from source (with removing cmsis-pack-manager from install_requires ) is ok.

[cederom]

Guys have seen my answer above? Have you tried installing or building the CPM by hand?

Have you tried pip install cmsis-pack-manager ? My ARM build is below:

[patrislav1]

Guys have seen my answer above? Have you tried installing or building the CPM by hand?

Have you tried pip install cmsis-pack-manager ? My ARM build is below:

Can you use your knowledge to create a PR on the documentation installing_on_non_x86.md? The information there seems to be incorrect or at least outdated.

[tbandtg]

I followed the guide, to install wihtout cmsis package manager and it no longer works.

[cederom]

@flit do you think if a binary build could be provided for ARM platforms? One build would save a lot of people :-) If you need external support around pyOCD I am open for contracts right now :-)

[cederom]

@patrislav1 thanks for the suggestion, in a free moment I could try to update the documentation, for now I am a bit pushed by deadlines with my own works, sorry, please take a look at the video, there is a step by step instruction what I did to make it work - basically you need to obtain the source code from github, build and install by hand CPM in the first place, then pyOCD, on the target ARM platform. You need some system packages on the target platform in order to build - i.e. rust - also the information provided in the documentation is true for some platforms as there were problems with the build.

Providing binary build over PyPI looks like a best solution - one build to help them all - again this may still require some OS dependency packages to be available (like rust) and this needs deeper investigation :-)

[pfalcon]

The issue also confirmed on Debian 10/aarch64 (raspi4) with Python 3.7.3 and pyocd 0.29.0. The backtrace is exactly like in the initial description above.

Now, given that https://github.com/pyocd/pyOCD/blob/master/docs/installing_on_non_x86.md was last (and first too) updated on 2019-07-17, and given the content of the backtrace, I wonder if the breakage happened on Python setuptools side. Specifically:

  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/rtos/__init__.py", line 30, in <module>
    load_plugin_classes_of_type('pyocd.rtos', RTOS, ThreadProvider)
  File "/home/huesmann/.local/lib/python3.7/site-packages/pyocd/core/plugin.py", line 88, in load_plugin_classes_of_type
    plugin = entry_point.load()()
...
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)

Here, pyocd makes innocent request to query entrypoints in the group 'pyocd.rtos', but pkg_resources (a module in setuptools) instead of doing what it was asked, goes to poke its nose into unsatisfied dependencies and crashes on that.

Note that on raspi4, it's possible to build cmsis-pack-manager, it's just neither it or its rust dependencies are really needed, and dealing with them is slow (especially if you CI-like setup and need to deal with that again and again).

[pfalcon]

Note that on raspi4, it's possible to build cmsis-pack-manager

I was too fast saying that...

There was another recent Rust dependencies drama: https://github.com/pyca/cryptography/issues/5771.

Seeing cases like that, I decided that it's not worthy investment of time to see workarounds, and better to just embrace Rust becoming a dependency of various packages. After all, again, it all seemed to build ok, even if required installing dependencies beforehand and then extra build time.

So, today I replayed my steps from a month ago or so, preparing instructions for pyocd deployment on production systems. And I see that I no longer can install cmsis-pack-manager on RasPi4/Debian10. I get Rust build errors now:

    error[E0658]: subslice patterns are unstable
       --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/object-0.23.0/src/read/mod.rs:162:41
        |
    162 |             [0x7f, b'E', b'L', b'F', 1, ..] => FileKind::Elf32,
        |                                         ^^
        |
        = note: for more information, see https://github.com/rust-lang/rust/issues/62254

As can be seen, it doesn't happen in the cmsis-pack-manager code per se, but in one of dependencies. And the only explanation would be that it was upgraded in the meantime, and started to break the build.

So, welcome to the brave new absolutely memory-safe Rust dependency hell!

[pfalcon]

Ok, taking hints from links embedded in the error message quoted above (the issue is now closed), I figured maybe that was too old Rust, I had 1.41 from Debin10 package repos. For comparison, cryptography quoted above cites 1.45 as the minimum required version. I used https://rustup.rs/ to install newer toolchain (no problem with aarch64 packages availability there). That installed 1.49 as of now.

With that, I was able to install pyocd 0.29.0 on RasPi4, the process (99% of which is building Rust code of course) took 15min.

[cederom]

Guys you are just discovering what mess Linux is when it comes to maintenance, not to mention quick wins are rewarded over long term stable working environment :-) This is why around 15 years ago I switched to FreeBSD for good and avoid Linux just as Windows :-) You can too :-) Open-Source is not only Linux. Linux is almost always tainted with binary drivers and other stuff that sure gives you quickly working stuff but sooner or later will leave you pants down. Linux is a kind of Windows in Unix world :-)

My yesterday experience with Linux emulation when my home directory was mounted:

root@0xCFMX4:/# adduser user
Adding user `user' ...
Adding new group `user' (1000) ...
Adding new user `user' (1000) with group `cd' ...
Creating home directory `/home/user' ...
Stopped: Couldn't create home directory `/home/user': File exists.

Removing directory `/home/user' ...
Removing user `user' ...
Removing group `user' ...
groupdel: group 'user' does not exist
adduser: `groupdel user' returned error code 6. Exiting.

And you can say goodbye to all of your files, documents, work, projects, etc. Because when Linux finds a file then surely it must be something to remove with no questions asked :-( Linux Sux Ass. Really.

[AGlass0fMilk]

Still seeing this as of today...

As mentioned above, the only way to completely remove the cmsis packs requirement is to delete the line from the setup.py script and install pyocd from source.

[cederom]

Hey @AGlass0fMilk do you have problem with dependencies for CMSIS-PACK or dependencies are fine but still the build fails? If the second option how much RAM do you have on your ARM board? I heard that in case of GCC build on ARM board it may fail because of RAM shortage. Can you confirm that this is not the problem for you? Do you have 8GB+ swap on your board?

I would like to identify possible build problems of bigger packages with low RAM boards.. and if adding swap would help?

[AGlass0fMilk]

Hey @AGlass0fMilk do you have problem with dependencies for CMSIS-PACK or dependencies are fine but still the build fails? If the second option how much RAM do you have on your ARM board? I heard that in case of GCC build on ARM board it may fail because of RAM shortage. Can you confirm that this is not the problem for you? Do you have 8GB+ swap on your board?

I would like to identify possible build problems of bigger packages with low RAM boards.. and if adding swap would help?

Hi @cederom, it's the rust build that causes the installation process to fail. It runs the machine out of memory.

I'm not on a Raspberry Pi, I'm running on a Samsung Artik 710S I had laying around. It's no longer supported but I'm not doing anything too fancy -- just trying to host a remote debug server so I can test with a K64F running Mbed-OS remotely (my router is downstairs and I didn't want to run a cable up to my office, nor depend on wifi).

The Artik is pretty powerful -- Octa-core Cortex A53 @ 1.4Ghz with 1GB of DDR3 RAM. It only has a 4GB eMMC that is nearly full. I have my python environment and other non-system files on an external 16GB SD card. The Artik still runs out of RAM while building the rust portions of the code.

I am not building anything on the Artik, just using it to host a GDB server so I can debug the K64F target attached via USB.

Removing the cmsis packs requirement from the setup.py script let me finish the install and I'm not up and running. It's just frustrating that there's (somewhat obscure) documentation about this exact issue and the commands don't even work.

[patrislav1]

Current workaround is to install from source after removing cmsis-pack-manager from install_requires in setup.py.

Update: since the packaging changed, the workaround is now to remove the cmsis-pack-manager line from the install_requires section in setup.cfg.