import libarchive.public error AttributeError: dlsym(0xXXXXXXX, archive_read_support_filter_all): symbol not found

[ramitwadhwa]

Hi All,

I have installed libarchive using pip on Mac OS 10.11.6.

My Code:

import libarchive.public

When I run this I get this error

Traceback (most recent call last): File "Test.py", line 1, in import libarchive.public File "/Library/Python/2.7/site-packages/libarchive/public.py", line 1, in from libarchive.adapters.archive_read import
File "/Library/Python/2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in import libarchive.calls.archive_read File "/Library/Python/2.7/site-packages/libarchive/calls/archive_read.py", line 17, in c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/init.py", line 378, in getattr func = self.getitem(name) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/init.py", line 383, in getitem func = self._FuncPtr((name_or_ordinal, self)) AttributeError: dlsym(0x7fb1d1e47f60, archive_read_support_filter_all): symbol not found

I have installed libarchive also using brew.

Is this library supported on Mac ?

[rocky]

I solved this by compiling libarchive from source https://github.com/libarchive/libarchive rather than via brew.

[SkyLeach]

@rocky I hear that is dangerous. That's why homebrew doesn't fix the issue: it breaks other things on the mac.

This needs to be elevated to bug, because the osx libarchive is out of date and this module ignores the LDFLAGS and CPPFLAGS required to point it to a newer version of the library (/usr/local/opt/libarchive for instance, the location used by homebrew).

[dsoprea]

This library just imports whatever is available. How are compiler flags supposed to remedy that? We're not compiling anything.

[SkyLeach]

@dsoprea I flagged this for looking into it this evening. I figured it was a dlopen call, not a compiler call.

[dsoprea]

@SkyLeach In the case of OSX having such an old version, I'm not sure there's a better way to point to a different version than to just set LA_LIBRARY_FILEPATH in the environment.

Thanks for bringing that reality to my attention.

[SkyLeach]

@dsoprea I wasn't aware of that environment variable (grepped for the more common LD_LIBRARY_PATH).

Any reason why modding library.py to examine both might be a bad idea? In my experience, everyone just assumes the common environment variables.

[dsoprea]

It was an oversight. Would you be able add a check for LD_LIBRARY_PATH after the existing one?

On Dec 15, 2016 6:18 PM, "Matt Gregory" [email protected] wrote:

@dsoprea https://github.com/dsoprea I wasn't aware of that environment variable (grepped for the more common LD_LIBRARY_PATH).

Any reason why modding library.py to examine both might be a bad idea? In my experience, everyone just assumes the common environment variables.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dsoprea/PyEasyArchive/issues/16#issuecomment-267473295, or mute the thread https://github.com/notifications/unsubscribe-auth/AArranq7DC0PmRpNG87g88-5n3Gnyr5Sks5rIcrXgaJpZM4KP7q8 .

[SkyLeach]

Ok, so I started looking at this. I'll issue a pull request when done if necessary, but I suspect the fault is actually in ctypes.

[SkyLeach]

Still looking at this, but the simple fix didn't work.

[96imranahmed]

Having the issue as well, let me know if you manage to find a fix @SkyLeach !

[SkyLeach]

I have a fix, working the PR now

[SkyLeach]

PR #21 up.

Please test. I did and it seems to work very well.

[dsoprea]

I didn't see any specific reasons why we can't just check for LD_LIBRARY_PATH (the conventional variable) in addition to the current variable. This was my original suggestion. Wouldn't the correct solution be to add the correct location to this in the environment so that anything that seeks to use it will find the correct one (a global solution rather than one localized to this project)?

[SkyLeach]

If I'm not mistaken, this ONLY works during setup. testing:

running install_egg_info
Copying libarchive.egg-info to /Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive-0.4.3-py2.7.egg-info
running install_scripts
(PyEasyArchive-skyleach) ________________________________________________________________________________
| ~/src/PyEasyArchive @ MAGREGOR-M-W0XX (magregor)
| => python
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import libarchive
>>>
(PyEasyArchive-skyleach) ________________________________________________________________________________
| ~/src/PyEasyArchive @ MAGREGOR-M-W0XX (magregor)
| => python2.7 ~/tmp/test_lafh.py
Traceback (most recent call last):
  File "/Users/magregor/tmp/test_lafh.py", line 8, in <module>
    import libarchive.public as la
  File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/public.py", line 1, in <module>
    from libarchive.adapters.archive_read import \
  File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
    import libarchive.calls.archive_read
  File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
    c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 375, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 380, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fda33d15d20, archive_read_support_filter_all): symbol not found

So I'm going to move the function out of the setup.py

[dsoprea]

@SkyLeach Yes. If we're all convinced that this is the way to go, then it'd have to go in library.py.

[SkyLeach]

@dsoprea done This will slow down library loading a tiny bit for homebrew users (only once when first imported though)

[dsoprea]

@SkyLeach See my other inquiry from thirty-five minutes ago.

[SkyLeach]

@dsoprea sure we can, but it'll still only work for PyEasyArchive. LD_LIBRARY_PATH wasn't finding libarchive.13.dylib (which is what goes to /usr/local/opt/libarchive) The only difference would be that the same code has to run, and then copen would have to search the new path for libarchive.dylib. Two searches instead of just the one, and no difference in overhead.

If we tried to check LD_LIBRARY_PATH then we'd have to worry about it being set for everything else, but not libarchive since it doesn't get installed in the normal place.

[dsoprea]

I need you to walk me through this. As I understand it, there is already a system-global libarchive and, if anyone wants a newer one, they'd install it via Brew. Obviously this one isn't Brew-linked because then it'd conflict with the system-global one. However, why can't you just prepend/add the Brew one to LD_LIBRARY_PATH/LA_LIBRARY_FILEPATH in the system environment or right above where you import this project?

I'm not concerned about discussing how libarchive affects the rest of the system. It's sufficient to know just that there's already a system-global one that has to remain in-place for OS X.

Since this is an exclusive Brew/Mac problem, it's not appropriate to put conditional logic in the project that doesn't help anyone else, especially when that logic depends on third-party tooling. Rather, the proper method should be identified and then be added as a footnote in the documentation.

[SkyLeach]

@dsoprea this is an OS-specific issue. On OSX the libarchive shipped with the OS is required and old (doesn't work with PyEasyArchive and/or newer archives). You can't just replace it. Most mac users use homebrew. If not, then they will most likely have LD_LIBRARY_PATH set since they are doing something very custom. Simply setting LD_LIBRARY_PATH with a mac (default) or a mac (with homebrew) won't work, because of the above issues.

Certainly it is possible to add to the documentation that all mac users need to do some esoteric stuff (basically what I just did) in order to support libarchive and PyEasyArchive. It can be done in .bash_profile (or .bashrc) using similar commands, although much more complicated with bash to strip the version number. Not using the specific installed version (that I pull with brew info --json) means it will not be found by the find_library function.

Is this technically specific to mac + homebrew users? Yes. That crowd just happens to be 90% of mac users who do development, thus making it the lions share of anyone wishing to install PyEasyArchive on a mac.

[dsoprea]

@SkyLeach (and everyone else)

I'm suggesting that the affected individuals set the value into LD_LIBRARY_PATH using one of the three ways:

1. Prepending the environment change to the command-line that they're executing the command-line from.
2. Doing a call to os.environ above the import.
3. Setting it into their user-profile (which should only affect their terminal sessions, but I'm totally sure).

I added a clause to the "Notes" section of the document as well as a script, based on your code, to print the path.

https://github.com/dsoprea/PyEasyArchive/pull/22

Check it and then I'll merge.

Everyone else: Does this help? Is this or isn't this sufficient?

[SkyLeach]

Looks good, but I'll need to test to be sure that the system library isn't found before/instead-of the environment library (which was one issue).

[dsoprea]

Please do. Does this seem like a practical solution for the affected population?

On Wed, Feb 8, 2017 at 9:37 AM, Matt Gregory [email protected] wrote:

Looks good, but I'll need to test to be sure that the system library isn't found before/instead-of the environment library (which was one issue).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dsoprea/PyEasyArchive/issues/16#issuecomment-278345360, or mute the thread https://github.com/notifications/unsubscribe-auth/AArranOtUwfgifZZYCs6iU55-VIsl5T8ks5radMXgaJpZM4KP7q8 .

[SkyLeach]

@dsoprea it seems so, but doesn't work as intended. We may still need to tell users to use LA_LIBRARY_FILEPATH:

(PyEasyArchive-skyleach) ________________________________________________________________________________
| ~/src/PyEasyArchive @ MAGREGOR-M-W0XX (magregor)
| => LD_LIBRARY_PATH=/usr/local/Cellar/libarchive/3.2.2/lib python2.7 ~/tmp/test_lafh.py
Traceback (most recent call last):
  File "/Users/magregor/tmp/test_lafh.py", line 8, in <module>
    import libarchive.public as la
  File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/public.py", line 1, in <module>
    from libarchive.adapters.archive_read import \
  File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
    import libarchive.calls.archive_read
  File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
    c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 375, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 380, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fc5e4c6f640, archive_read_support_filter_all): symbol not found

LA_LIBRARY_FILEPATH=/usr/local/Cellar/libarchive/3.2.2/lib/libarchive.dylib python2.7 ~/tmp/test_lafh.py works fine though

[SkyLeach]

My test script btw (which is pretty simple):

FILE="dcnm-installer.6.1.2.iso"
import sys, os
import pprint
import libarchive.public as la
with open(FILE, 'rb') as fh:
    with la.file_reader(fh.name) as lafh:
        entries = False
        for entry in lafh:
            print str(entry)
            pprint.pprint(dir(entry.filetype))

please disregard the open fh then using fh.name, I was using it to test for os filelocks

[dsoprea]

It goes back to my original suggestion. ctypes doesn't use LD_LIBRARY_PATH. I've updated the library-load logic to implement it. Do an update and retry. Note that you can run "./run.py" from the "tests/" subdirectory to run a couple of tests to test the loading, too.

[dsoprea]

I just went ahead and merged. It's already been tested.

Everyone: Please see if this makes it easier for you to cope with running under OS X.

[devssh]

I tried running it and didn't work on mac High Sierra, libarchive v0.4.4 I just ran pip install libarchive and import libarchive.public gives error, same on python 2 as well

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/envs/py36/lib/python3.6/site-packages/libarchive/public.py", line 1, in <module>
    from libarchive.adapters.archive_read import \
  File "/usr/local/anaconda3/envs/py36/lib/python3.6/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
    import libarchive.calls.archive_read
  File "/usr/local/anaconda3/envs/py36/lib/python3.6/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
    c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
  File "/usr/local/anaconda3/envs/py36/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/anaconda3/envs/py36/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fee5f551c10, archive_read_support_filter_all): symbol not found

[kklochkov]

@devssh, did you try something like this?

os.environ['LA_LIBRARY_FILEPATH'] = '/usr/local/Cellar/libarchive/3.3.2/lib/libarchive.dylib' import libarchive.public import libarchive.constants

[devssh]

It did not work, although I am unable to find the lib/libarchive.dylib as I am using Anaconda which might be another complication on top of MacOSX High Sierra

Requirement already satisfied: libarchive in /usr/local/anaconda3/envs/py36/lib/python3.6/site-packages
Requirement already satisfied: nose in /usr/local/anaconda3/envs/py36/lib/python3.6/site-packages (from libarchive)

I'll get back to you if I find it