Problem with installing banditPAM

[retzerjj]

I've been following some discussions about install issues however I still can't seem to successfully install banditPAM. I'm getting a long error message that seems to indicate that building the wheel for banditPAM failed (full error attached). I'm working with a MAC intel using Monterrey. I successfully install llvm and armadillo. To answer some questions asked previously: When I try pip install ., I get: ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

When I execute gcc --version I get: Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include- dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1 Apple clang version 13.0.0 (clang-1300.0.29.30) Target: x86_64-apple-darwin21.2.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

When I try pip3 -V I get: pip 21.3.1 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip (python 3.10)

I've tried multiple install commands (e.g. pip, pip3 etc.) but with no luck. Any suggestions would be greatly appreciated, Thanks!

banditPAMError.txt

[motiwari]

Hi @retzerjj , thanks for filing an issue. Could you try pip install banditpam? This should pull version 2.0.0 from PyPI.

The error message you surfaced (ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.) also suggests that you may be trying to install the repository from the wrong root directory. If you'd like to install the code from source, could you run pip install --no-cache-dir -e . (note the trailing period) from the repo's root directory, e.g. /BanditPAM?

[motiwari]

I also just took a closer look at your full error output and it looks like your LLVM installation is having trouble finding the C++ standard library headers. Did you try to install LLVM from source? If so, could you install it instead via brew install llvm?

More helpful details are here: https://github.com/ThrunGroup/BanditPAM/blob/main/docs/install_mac.md

[retzerjj]

Thanks very much for your response. I did install llvm using brew. When executing the command again (brew install llvm) I get the following message:

Warning: llvm 13.0.0_2 is already installed and up-to-date. To reinstall 13.0.0_2, run: brew reinstall llvm

I next downloaded and extracted banditPAM and ran

pip install --no-cache-dir -e .

in the downloaded directory. It does indeed look like there is a problem finding the stdio.h header file (new error message text attached) but since the llvm installation looks ok, I'm not sure what to do next.

Thanks again very much

Note: not sure if this plays a role however armadillo appears to have been installed correctly too:

brew install armadillo 

   Warning: armadillo 10.7.5 is already installed and up-to-date.

To reinstall 10.7.5, run: brew reinstall armadillo

banditPAMError2.txt

[motiwari]

Thanks @retzerjj ! Just to confirm, did you also install the xcode command line tools via the Mac installation guide? This post suggests that might be the issue

[retzerjj]

Hi @motiwari, yes I did. When I try

xcode-select --install

I get

xcode-select: error: command line tools are already installed, use "Software Update" to install updates

Thanks again.

[motiwari]

Interesting -- could you update the Xcode tools by opening the "App Store" app and Choosing "Updates" on the left-hand sidebar?

Also, does your /usr/local/ directory exist?

[retzerjj]

Hello @motiwari

I checked and I do have a /usr/local directory. I also checked and there was no update for xcode in the app store app.

I tried to update xcode command line tools (which were installed). I deleted them and re-installed from the command line. I also noted that when I opened the xcode app it asked to install additional software and I did so.

I then re-booted the computer and after trying to install banditPAM (this time version 3) and it did in fact work!!

Many thanks for all your help!

However, I next went to install banditPAM on my macbook (apple silicon). I made sure I had armadillo/llvm installed and xcode tools was present and updated. I now get a slightly different error saying, I believe, that it cant find armadillo. I checked and it appears llvm is installed under homebrew, unlike on my iMac, but my path appears to be correctly indicating where to look:

echo $PATH /opt/homebrew/opt/llvm/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin

I've attached the error message to this note. Again, any suggestions would be greatly appreciated, your help is invaluable!!

P.S. Two additional notes (not sure if they are material) 1. Apple released an update for command line tools which was installed on both machines 2. The iMac (where banditPAM installed successfully) is Intel based using a bash shell. The MacBook Air (where the error is now being produced) is Apple M1 Silicon and uses a zsh shell.

Thanks again very much! banditPAM_error_m1.txt

[motiwari]

Hey @retzerjj , that's great to hear! Thanks for reporting back with the fix -- I've added this to the install guide to help future generations.

Regarding the new error on the Macbook Air, could you confirm you've run brew install armadillo and provide the output of brew --prefix armadillo?

Also, is there a file called armadillo in your /usr/local/include? If not, we may need to create a symlink there to your brew-installed armadillo (perhaps the brew link step did not complete successfully and raised an error when installing armadillo?). For example, here is the output on my machine:

>>>brew --prefix armadillo
/usr/local/opt/armadillo
>>>ls -alh /usr/local/include/armadillo
lrwxr-xr-x 1 motiwari admin 44 Nov 11 09:04 /usr/local/include/armadillo -> /usr/local/Cellar/armadillo/10.7.3/include/armadillo

By the way, the PATH variable is only used to search for executables and so won't help the linking problem. Instead, we may need to edit your LD_LIBRARY_PATH, which tells the linker where to look for libraries it needs, like armadillo.

[motiwari]

Oh, it also looks like M1 Macs have changed some important directories. Thanks for surfacing this bug, I'll work on a fix!

In the meantime, you can likely get it to work by creating a symlink in a (new) /usr/local/include directory for armadillo, pointing to where brew has installed it

[motiwari]

@retzerjj could you try the following command and let me know if it works? You may need a sudo:

sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

[retzerjj]

Hello @motiwari , I tried your suggestions and my results are listed below

1. Here is the path I get when I execute "brew --prefix armadillo" /opt/homebrew/opt/armadillo

2. I checked and there is no file called armadillo in "/usr/local/include"

3. I tried adding to my path in ~/.zshrc (based on recommendations from the link regarding M1 Mac important directory changes): export HOMEBREW_PREFIX="/opt/homebrew" export HOMEBREW_CELLAR="/opt/homebrew/Cellar" export HOMEBREW_NO_ANALYTICS=1

4. I also tried creating a symlink with the command "sudo ln -s /opt/homebrew/opt /usr/local/include" (which may or may not be correct). I was unable to compile banditPAM after this as well so I unlinked it.

5. I executed the command "sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam" The output in attached file "newInstallAttempt.txt"

Thanks again for all your help, Happy New Year!

newInstallAttempt.txt

[motiwari]

Happy New Year, @retzerjj !

Could you try running the command below again? I just updated something.

sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

If that still doesn't work, could you try running the following command

mkdir -p /usr/local/include && ln -s /opt/homebrew/opt/armadillo/include/armadillo /usr/local/include/armadillo

and then running pip install banditpam again?

If neither of those work, could you provide the output of the following command?

tree /opt/homebrew/opt/armadillo

A few tips that might help:

• The problem is that on M1 Macs, the installation location of packages like armadillo has changed
• If it is easier for you, you can provide your terminal output in pastes from PasteBin
• You can use single backtick (the key to the left of the 1) for code formatting and triple backticks for

code block

[retzerjj]

Hello @motiwari I am still unable to install banditPAM. It seems like the error is not related to not finding armadillo in /usr/local/include but rather maybe that an error is generated once it is found:

#include "armadillo_bits/config.hpp" ^~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 error generated. error: command '/opt/homebrew/opt/llvm/bin/clang' failed with exit code 1

I also ran the tree command, output is in the attached txt file. Thanks!

treeOutput.txt

[motiwari]

@retzerjj We are making progress! Can you try one more time:

sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

[retzerjj]

@motiwari it looks like its not getting past the setup.py step (error attached newError.txt )

[motiwari]

Oops sorry, can you try this command instead?

sudo python3 -m pip install --no-cache-dir -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

[retzerjj]

Looks like a similar error newError2.txt .

[motiwari]

Sorry for the churn here, @retzerjj . Let me get my hands on an M1 Mac so I can debug locally and save you the time. I'll get back to you shortly

[retzerjj]

Thanks very much @motiwari, I very much appreciate it.

[motiwari]

Hey @retzerjj , just letting you know I haven't forgotten about this. It turns out building for M1 Macs is much harder than I initially thought, but I'm making progress and should have an update soon

[retzerjj]

Fantastic! Thanks so much for doing this, I’ll look forward to your update, Take care, Joe

On Jan 10, 2022, at 11:12 PM, Mo Tiwari @.***> wrote:

Hey @retzerjj https://github.com/retzerjj , just letting you know I haven't forgotten about this. It turns out building for M1 Macs is much harder than I initially thought, but I'm making progress and should have an update soon

— Reply to this email directly, view it on GitHub https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1009605548, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP5KCRTCH5GGT243C33ICTUVO325ANCNFSM5K4PG5MA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

[motiwari]

Hi @retzerjj ; I think I was able to build the Python wheel for M1 Macs; could you install banditpam via pip install -i https://test.pypi.org/simple/ banditpam==3.0.3a1 and let me know if it works?

The code in Example 1 should run.

A caveat: you'll have to use CPython 3.8+. I had a friend with an M1 Mac test it and it appears to work!

In case you get an error like RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe, you may need to run pip install --upgrade numpy

[retzerjj]

Hello @motiwari

It appears to install correctly:

However I’m getting errors when I try running the code:

from banditpam import KMedoids TypeError: object of type 'module' has no len() import numpy as np import matplotlib.pyplot as plt

Generate data from a Gaussian Mixture Model with the given means:

np.random.seed(0) n_per_cluster = 40 means = np.array([[0,0], [-5,5], [5,5]]) X = np.vstack([np.random.randn(n_per_cluster, 2) + mu for mu in means])

Fit the data with BanditPAM:

kmed = KMedoids(n_medoids=3, algorithm="BanditPAM") NameError: name 'KMedoids' is not defined kmed.fit(X, 'L2') NameError: name 'kmed' is not defined

I suspect this is because I’m not installing using CPython however I can’t seem to find instructions for downloading / installing CPython on a Mac. Is there a GitHub site that would have the instructions/code?

Thanks Joe

On Jan 25, 2022, at 3:45 PM, Mo Tiwari @.***> wrote:

Hi @retzerjj https://github.com/retzerjj ; I think I was able to build the Python wheel for M1 Macs; could you install the appropriate wheel (attached) for your python version and let me know if it works?

E.g., download the wheel from the attached zip and run pip install pip install banditpam-3.0.3a1-cp39-cp39-macosx_10_14_universal2.whl

Then run the code in Example 1 https://github.com/ThrunGroup/BanditPAM#example-1-synthetic-data-from-a-gaussian-mixture-model?

A caveat: you'll have to use CPython 3.8+

banditpam-3.0.3a1-cp38-cp38-macosx_10_14_universal2.whl.zip https://github.com/ThrunGroup/BanditPAM/files/7937399/banditpam-3.0.3a1-cp38-cp38-macosx_10_14_universal2.whl.zip banditpam-3.0.3a1-cp39-cp39-macosx_10_14_universal2.whl.zip https://github.com/ThrunGroup/BanditPAM/files/7937404/banditpam-3.0.3a1-cp39-cp39-macosx_10_14_universal2.whl.zip banditpam-3.0.3a1-cp310-cp310-macosx_10_14_universal2.whl.zip https://github.com/ThrunGroup/BanditPAM/files/7937409/banditpam-3.0.3a1-cp310-cp310-macosx_10_14_universal2.whl.zip — Reply to this email directly, view it on GitHub https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1021638296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP5KCW3I2ONRMFWRADDRK3UX4KWVANCNFSM5K4PG5MA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

[motiwari]

Thanks @retzerjj ! It's strange; pip install -i https://test.pypi.org/simple/ banditpam==3.0.3a1 should not have succeeded unless you're using CPython. In the pip output, did it somehow fall back to a previous version? What's the output if you run python -c "import platform; print(platform.python_implementation())"?

CPython should be the default Python implementation when using Anaconda, which I highly recommend!

If you're running CPython, the version is >=3.8, and you're still getting this error, could you provide the output of the following commands?

a) the output when you open a python shell:

>>> python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:50:38)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.

b) the output of pip -V

[retzerjj]

Hello @motiwari The results of pip -V and python are below. Also the message I get when trying to install banditPAM from the terminal is shown. Note that when the install did appear to work I was installing from the “terminal” tab inside of RStudio. Thanks Joe

Hello

It appears to install correctly:

However I’m getting errors when I try running the code:

from banditpam import KMedoids TypeError: object of type 'module' has no len() import numpy as np import matplotlib.pyplot as plt

Generate data from a Gaussian Mixture Model with the given means:

np.random.seed(0) n_per_cluster = 40 means = np.array([[0,0], [-5,5], [5,5]]) X = np.vstack([np.random.randn(n_per_cluster, 2) + mu for mu in means])

Fit the data with BanditPAM:

kmed = KMedoids(n_medoids=3, algorithm="BanditPAM") NameError: name 'KMedoids' is not defined kmed.fit(X, 'L2') NameError: name 'kmed' is not defined

I suspect this is because I’m not installing using CPython however I can’t seem to find instructions for downloading / installing CPython on a Mac. Is there a GitHub site that would have the instructions/code?

Thanks Joe

On Jan 26, 2022, at 9:35 AM, Joseph Retzer @.***> wrote:

Begin forwarded message:

From: Mo Tiwari @.*** @.>> Subject: Re: [ThrunGroup/BanditPAM] Problem with installing banditPAM (Issue #167) Date: January 25, 2022 at 3:45:14 PM CST To: ThrunGroup/BanditPAM @. @.>> Cc: Joseph Retzer @. @.>>, Mention @. @.>> Reply-To: ThrunGroup/BanditPAM @. @.***>>

Hi @retzerjj https://github.com/retzerjj ; I think I was able to build the Python wheel for M1 Macs; could you install the appropriate wheel (attached) for your python version and let me know if it works?

E.g., download the wheel from the attached zip and run pip install pip install banditpam-3.0.3a1-cp39-cp39-macosx_10_14_universal2.whl

Then run the code in Example 1 https://github.com/ThrunGroup/BanditPAM#example-1-synthetic-data-from-a-gaussian-mixture-model?

A caveat: you'll have to use CPython 3.8+

banditpam-3.0.3a1-cp38-cp38-macosx_10_14_universal2.whl.zip https://github.com/ThrunGroup/BanditPAM/files/7937399/banditpam-3.0.3a1-cp38-cp38-macosx_10_14_universal2.whl.zip banditpam-3.0.3a1-cp39-cp39-macosx_10_14_universal2.whl.zip https://github.com/ThrunGroup/BanditPAM/files/7937404/banditpam-3.0.3a1-cp39-cp39-macosx_10_14_universal2.whl.zip banditpam-3.0.3a1-cp310-cp310-macosx_10_14_universal2.whl.zip https://github.com/ThrunGroup/BanditPAM/files/7937409/banditpam-3.0.3a1-cp310-cp310-macosx_10_14_universal2.whl.zip — Reply to this email directly, view it on GitHub https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1021638296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP5KCW3I2ONRMFWRADDRK3UX4KWVANCNFSM5K4PG5MA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

On Jan 26, 2022, at 12:28 PM, Mo Tiwari @.***> wrote:

Thanks @retzerjj https://github.com/retzerjj ! It's strange; pip install -i https://test.pypi.org/simple/ banditpam==3.0.3a1 should not have succeeded unless you're using CPython. In the pip output, did it somehow fall back to a previous version? What's the output if you run python -c "import platform; print(platform.python_implementation())"?

CPython should be the default Python implementation when using Anaconda https://www.anaconda.com/, which I highly recommend!

If you're running CPython, the version is >=3.8, and you're still getting this error, could you provide the output of the following commands?

a) the output when you open a python shell:

python Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:50:38) [Clang 11.1.0 ] on darwin Type "help", "copyright", "credits" or "license" for more information. b) the output of pip -V

— Reply to this email directly, view it on GitHub https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1022474790, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP5KCSNHHJYFRGSBK5TTHLUYA4MBANCNFSM5K4PG5MA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

[retzerjj]

Hello Sudhir, Thats great. Could you let me know how you installed it through Anaconda? I tried adding the plpi channel but it does not show up. I also tried to install the .whl file downloaded from GitHub but I’m not able to do that (not sure how to access the file). Thanks & take care, Joe

On Feb 1, 2022, at 7:18 AM, Sudhir Kumar Suman @.***> wrote:

@retzerjj https://github.com/retzerjj I'm having the same problem when trying to install banditpam on my Mac M1[unresolved]. For the time being, I've tried Anaconda and installed banditpam there, and it's working.

— Reply to this email directly, view it on GitHub https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1026833899, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP5KCRDQ4UD5OBHPTAKYLDUY7MQ7ANCNFSM5K4PG5MA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

[motiwari]

Hmmm it seems Sudhir deleted his comment, but we owe a thank you to him!

@retzerjj you can install Python from Anaconda here; it should provide instructions on how to activate the appropriate Anaconda environment after you've installed it. By the way, did you copy-paste the wrong information in this comment?

Probably it works on Anaconda because I did at least something right when creating the M1 Mac wheels; but I've also heard from others that the wheels don't work with Miniconda or other Python installations. I'll leave this issue open until the package is working everywhere

[SumanSudhir]

Yes, Banditpam was successfully installed, but I was unable to use it because it displayed an error while importing it.

"ImportError: dlopen(/Users/sudhirwork/anaconda3/lib/python3.9/site-packages/banditpam.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '__ZTWN4arma19mt19937_64_instanceE'"

[retzerjj]

It appears my .png attachments of my error messages are not visible in my email, is there another way I could send them?

I have Anaconda installed with multiple versions of Python (3.9 and 3.10 included). I also added pypi as a channel. However, bandiPAM does not appear in the “Not Installed” file list and I’m not sure how to install the .whl file I downloaded using Anaconda.

On Feb 1, 2022, at 3:57 PM, Mo Tiwari @.***> wrote:

Hmmm it seems Sudhir deleted his comment, but we owe a thank you to him!

@retzerjj https://github.com/retzerjj you can install Python from Anaconda here https://www.anaconda.com/; it should provide instructions on how to activate the appropriate Anaconda environment after you've installed it. By the way, did you copy-paste the wrong information in this comment https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1026161508?

Probably it works on Anaconda because I did at least something right when creating the M1 Mac wheels; but I've also heard from others that the wheels don't work with Miniconda or other Python installations. I'll leave this issue open until the package is working everywhere

— Reply to this email directly, view it on GitHub https://github.com/ThrunGroup/BanditPAM/issues/167#issuecomment-1027325733, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP5KCQWHQPLRAWYT23BGXDUZBJN7ANCNFSM5K4PG5MA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

[motiwari]

Sorry for the late response @retzerjj . I'm not sure how to attach pictures by email-replying to Github email notifications, but if you sign into Github and view this link, you can attach an image in the comment box.

At the moment, I'm not sure what to do for this because I can't reproduce your or Sudhir's error locally without an M1 Mac. Two of my friends with M1 Mac where able to use the M1 wheel successfully, but another one surfaced an error like this:

ImportError: dlopen(/Users/frieda/miniconda3/envs/behavior/lib/python3.8/site-packages/banditpam.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_omp_get_max_threads' 

This looks like this is different from Sudhir's error message, which seems to be complaining about an armadillo symbol.

I ordered an M1 Mac on 12/29/21, but it still won't be delivered until 3/10/22 (presumably due to existing supply chain issues). If you provide your error message then I can investigate it when I receive the M1 Mac