FIt-SNE
FIt-SNE copied to clipboard
Published
20 hours ago •
KlugerLab
KlugerLab
Error in pip install fitsne
Hi, here.
I tried to install Fit-SNE by the command line "pip install fitsne". However, there seemed to be somthing wrong with gcc.
pip install fitsne
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting fitsne
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/92/d4/e19298eb37d1b56d5b0675daf0443a42a17000258ab55797035b8ece2aeb/fitsne-1.1.1.tar.gz (151 kB)
|████████████████████████████████| 151 kB 1.6 MB/s
Requirement already satisfied: numpy in ./anaconda2/envs/pegasus/lib/python3.7/site-packages (from fitsne) (1.18.5)
Requirement already satisfied: cython in ./anaconda2/envs/pegasus/lib/python3.7/site-packages (from fitsne) (0.29.21)
Building wheels for collected packages: fitsne
Building wheel for fitsne (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /cluster/home/zfli/anaconda2/envs/pegasus/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-yunl5jsw/fitsne/setup.py'"'"'; __file__='"'"'/tmp/pip-install-yunl5jsw/fitsne/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-9yw0krtm
cwd: /tmp/pip-install-yunl5jsw/fitsne/
Complete output (33 lines):
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/fitsne
copying fitsne/__init__.py -> build/lib.linux-x86_64-3.7/fitsne
copying fitsne/_version.py -> build/lib.linux-x86_64-3.7/fitsne
copying fitsne/cywrap.py -> build/lib.linux-x86_64-3.7/fitsne
running build_ext
building 'fitsne.cppwrap' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/fitsne
creating build/temp.linux-x86_64-3.7/fitsne/src
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/cppwrap.cpp -o build/temp.linux-x86_64-3.7/fitsne/cppwrap.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/src/nbodyfft.cpp -o build/temp.linux-x86_64-3.7/fitsne/src/nbodyfft.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from fitsne/src/nbodyfft.cpp:4:
fitsne/src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
7 | #include <fftw3.h>
| ^~~~~~~~~
compilation terminated.
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running bdist_wheel
running build
running build_py
running build_ext
building 'fitsne.cppwrap' extension
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/cppwrap.cpp -o build/temp.linux-x86_64-3.7/fitsne/cppwrap.o -std=c++11 -O3 -pthread -lfftw3 -lm -stdlib=libc++
gcc: error: unrecognized command-line option ‘-stdlib=libc++’
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for fitsne
Running setup.py clean for fitsne
Failed to build fitsne
Installing collected packages: fitsne
Running setup.py install for fitsne ... error
ERROR: Command errored out with exit status 1:
command: /cluster/home/zfli/anaconda2/envs/pegasus/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-yunl5jsw/fitsne/setup.py'"'"'; __file__='"'"'/tmp/pip-install-yunl5jsw/fitsne/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-obe2addf/install-record.txt --single-version-externally-managed --compile --install-headers /cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m/fitsne
cwd: /tmp/pip-install-yunl5jsw/fitsne/
Complete output (33 lines):
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/fitsne
copying fitsne/__init__.py -> build/lib.linux-x86_64-3.7/fitsne
copying fitsne/_version.py -> build/lib.linux-x86_64-3.7/fitsne
copying fitsne/cywrap.py -> build/lib.linux-x86_64-3.7/fitsne
running build_ext
building 'fitsne.cppwrap' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/fitsne
creating build/temp.linux-x86_64-3.7/fitsne/src
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/cppwrap.cpp -o build/temp.linux-x86_64-3.7/fitsne/cppwrap.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/src/nbodyfft.cpp -o build/temp.linux-x86_64-3.7/fitsne/src/nbodyfft.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from fitsne/src/nbodyfft.cpp:4:
fitsne/src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
7 | #include <fftw3.h>
| ^~~~~~~~~
compilation terminated.
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running install
running build
running build_py
running build_ext
building 'fitsne.cppwrap' extension
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/cppwrap.cpp -o build/temp.linux-x86_64-3.7/fitsne/cppwrap.o -std=c++11 -O3 -pthread -lfftw3 -lm -stdlib=libc++
gcc: error: unrecognized command-line option ‘-stdlib=libc++’
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /cluster/home/zfli/anaconda2/envs/pegasus/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-yunl5jsw/fitsne/setup.py'"'"'; __file__='"'"'/tmp/pip-install-yunl5jsw/fitsne/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-obe2addf/install-record.txt --single-version-externally-managed --compile --install-headers /cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m/fitsne Check the logs for full command output.
version information: gcc version 10.2.0 (GCC)
Try to remove the problematic flag -stdlib=libc++ from the compiler arguments. I add that flag for Mac compilers, but I think it isn't necessary with GCC.
remove the problematic flag
Sorry to disturb you again. How to remove the problematic flag -stdlib=libc++ from the compiler arguments when installing fitsne? Should I download the sources and install locally?
You should probably clone the repository https://github.com/KlugerLab/pyFIt-SNE and change thesetup.py file accordingly. However, I'm not sure how up-to-date that repository is. Whenever I use this C++ implementation of FIt-SNE, I follow the instructions here and don't bother with the PyPI package. It's kind of a pain to set up.
Sorry to reopen this issue again. I tried to remove the problematic flag -stdlib=libc++ from the compiler arguments, there was still something wrong with C++. Is my setup.py codes wrong?
My all steps were as follows.
git clone https://github.com/KlugerLab/pyFIt-SNE.git
Then I changed the setup.py codes.
from setuptools import setup, find_packages # from distutils.core import setup
from setuptools.extension import Extension
from Cython.Build import cythonize
from os import path
from os import environ
from sys import platform
__version__ = "0.0.0"
exec(open('fitsne/_version.py').read())
here = path.abspath(path.dirname(__file__))
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
long_description = f.read()
author="George Linderman, Gioele La Manno"
author_email="[email protected], [email protected]"
url="https://github.com/KlugerLab/FIt-SNE"
download_url="https://github.com/KlugerLab/pyFIt-SNE/archive/%s.tar.gz"%__version__
keywords=["tSNE", "embedding"]
description="Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE)"
license="BSD3"
#Try...except because for some OS X setups, the compilation fails without -stdlib=libc++
try:
if platform == "darwin":
extensions = [Extension("fitsne.cppwrap",
["fitsne/cppwrap.pyx", "fitsne/src/nbodyfft.cpp", "fitsne/src/sptree.cpp", "fitsne/src/tsne.cpp"],
language="c++",
extra_compile_args=["-std=c++11", "-O3", '-pthread', "-lfftw3", "-lm"],
extra_link_args=['-lfftw3', '-lm',"-mmacosx-version-min=10.9"])]
else:
extensions = [Extension("fitsne.cppwrap",
["fitsne/cppwrap.pyx", "fitsne/src/nbodyfft.cpp", "fitsne/src/sptree.cpp", "fitsne/src/tsne.cpp"],
language="c++",
extra_compile_args=["-std=c++11", "-O3", '-pthread', "-lfftw3", "-lm"],
extra_link_args=['-lfftw3', '-lm'])]
extensions = cythonize(extensions, language="c++", include_path=[])
# package_data = {}
__version__ = "0.0.0"
exec(open('fitsne/_version.py').read())
setup(name="fitsne",
version=__version__,
packages=find_packages(),
install_requires=['numpy', 'cython'],
ext_modules=extensions,
# package_data=package_data,
# metadata
author=author,
author_email=author_email,
url=url,
download_url=download_url,
keywords=keywords,
description=description,
long_description=long_description,
license=license)
except:
extensions = [Extension("fitsne.cppwrap",
["fitsne/cppwrap.pyx", "fitsne/src/nbodyfft.cpp", "fitsne/src/sptree.cpp", "fitsne/src/tsne.cpp"],
language="c++",
extra_compile_args=["-std=c++11", "-O3", '-pthread', "-lfftw3", "-lm"],
extra_link_args=['-lfftw3', '-lm'])]
extensions = cythonize(extensions, language="c++", include_path=[])
setup(name="fitsne",
version=__version__,
packages=find_packages(),
install_requires=['numpy', 'cython'],
ext_modules=extensions,
author=author,
author_email=author_email,
url=url,
download_url=download_url,
keywords=keywords,
description=description,
long_description=long_description,
license=license)
Then I run setup.py in python.
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Obtaining file:///cluster/home/zfli/apps/pyFIt-SNE
Requirement already satisfied: numpy in /cluster/home/zfli/anaconda2/envs/pegasus/lib/python3.7/site-packages (from fitsne==1.1.0) (1.18.5)
Requirement already satisfied: cython in /cluster/home/zfli/anaconda2/envs/pegasus/lib/python3.7/site-packages (from fitsne==1.1.0) (0.29.21)
Installing collected packages: fitsne
Running setup.py develop for fitsne
ERROR: Command errored out with exit status 1:
command: /cluster/home/zfli/anaconda2/envs/pegasus/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/cluster/home/zfli/apps/pyFIt-SNE/setup.py'"'"'; __file__='"'"'/cluster/home/zfli/apps/pyFIt-SNE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
cwd: /cluster/home/zfli/apps/pyFIt-SNE/
Complete output (51 lines):
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running develop
running egg_info
creating fitsne.egg-info
writing fitsne.egg-info/PKG-INFO
writing dependency_links to fitsne.egg-info/dependency_links.txt
writing requirements to fitsne.egg-info/requires.txt
writing top-level names to fitsne.egg-info/top_level.txt
writing manifest file 'fitsne.egg-info/SOURCES.txt'
reading manifest file 'fitsne.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'LICENSE'
writing manifest file 'fitsne.egg-info/SOURCES.txt'
running build_ext
building 'fitsne.cppwrap' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/fitsne
creating build/temp.linux-x86_64-3.7/fitsne/src
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/cppwrap.cpp -o build/temp.linux-x86_64-3.7/fitsne/cppwrap.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/src/nbodyfft.cpp -o build/temp.linux-x86_64-3.7/fitsne/src/nbodyfft.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from fitsne/src/nbodyfft.cpp:4:
fitsne/src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
7 | #include <fftw3.h>
| ^~~~~~~~~
compilation terminated.
Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running develop
running egg_info
writing fitsne.egg-info/PKG-INFO
writing dependency_links to fitsne.egg-info/dependency_links.txt
writing requirements to fitsne.egg-info/requires.txt
writing top-level names to fitsne.egg-info/top_level.txt
reading manifest file 'fitsne.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'LICENSE'
writing manifest file 'fitsne.egg-info/SOURCES.txt'
running build_ext
building 'fitsne.cppwrap' extension
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/cppwrap.cpp -o build/temp.linux-x86_64-3.7/fitsne/cppwrap.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /cluster/home/zfli/anaconda2/envs/pegasus/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ifitsne -I/cluster/home/zfli/anaconda2/envs/pegasus/include/python3.7m -c fitsne/src/nbodyfft.cpp -o build/temp.linux-x86_64-3.7/fitsne/src/nbodyfft.o -std=c++11 -O3 -pthread -lfftw3 -lm
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from fitsne/src/nbodyfft.cpp:4:
fitsne/src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
7 | #include <fftw3.h>
| ^~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /cluster/home/zfli/anaconda2/envs/pegasus/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/cluster/home/zfli/apps/pyFIt-SNE/setup.py'"'"'; __file__='"'"'/cluster/home/zfli/apps/pyFIt-SNE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
Well, now it seems like it doesn't find FFTW. I don't really know how to solve that without diving into it. Installation and distribution problems like this are exactly why I developed openTSNE. If you're okay with a marginal decrease in speed, you can use that, and it's completely painless to install.
fitsne/cppwrap.py need this at the beginning to satisfy cython:
# distutils: language=c++
Modify PyPi package and install it
# Get fitsne pip package.
wget https://files.pythonhosted.org/packages/92/d4/e19298eb37d1b56d5b0675daf0443a42a17000258ab55797035b8ece2aeb/fitsne-1.1.1.tar.gz
# Extract tar ball.
tar xzf fitsne-1.1.1.tar.gz
cd fitsne-1.1.1
Create file ../fitsne-pypi.patch with content:
--- fitsne-1.1.1/setup.py 2018-11-17 12:42:11.000000000 +0100
+++ fitsne-1.1.1-patched/setup.py 2021-01-27 16:14:57.148976000 +0100
@@ -24,16 +24,14 @@
if platform == "darwin":
extensions = [Extension("fitsne.cppwrap",
["fitsne/cppwrap.pyx", "fitsne/src/nbodyfft.cpp", "fitsne/src/sptree.cpp", "fitsne/src/tsne.cpp"],
- language="c++",
extra_compile_args=["-std=c++11", "-O3", '-pthread', "-lfftw3", "-lm"],
- extra_link_args=['-lfftw3', '-lm',"-mmacosx-version-min=10.9"])]
+ extra_link_args=['-lfftw3', '-lm', "-mmacosx-version-min=10.9"])]
else:
extensions = [Extension("fitsne.cppwrap",
["fitsne/cppwrap.pyx", "fitsne/src/nbodyfft.cpp", "fitsne/src/sptree.cpp", "fitsne/src/tsne.cpp"],
- language="c++",
extra_compile_args=["-std=c++11", "-O3", '-pthread', "-lfftw3", "-lm"],
extra_link_args=['-lfftw3', '-lm'])]
- extensions = cythonize(extensions, language="c++", include_path=[])
+ extensions = cythonize(extensions, language_level="3", include_path=[])
# package_data = {}
__version__ = "0.0.0"
@@ -58,10 +56,9 @@
except:
extensions = [Extension("fitsne.cppwrap",
["fitsne/cppwrap.pyx", "fitsne/src/nbodyfft.cpp", "fitsne/src/sptree.cpp", "fitsne/src/tsne.cpp"],
- language="c++",
extra_compile_args=["-std=c++11","-stdlib=libc++", "-O3", '-pthread', "-lfftw3", "-lm"],
extra_link_args=['-lfftw3', '-lm'])]
- extensions = cythonize(extensions, language="c++", include_path=[])
+ extensions = cythonize(extensions, language_level="3", include_path=[])
setup(name="fitsne",
version=__version__,
--- fitsne-1.1.1/fitsne/cppwrap.pyx 2020-01-07 22:11:12.000000000 +0100
+++ fitsne-1.1.1-patched/fitsne/cppwrap.pyx 2021-01-27 16:11:46.431525000 +0100
@@ -1,3 +1,5 @@
+# distutils: language=c++
+
import cython
from libcpp cimport bool
# Patch extracted PyPi package.
patch -p1 < ../fitsne-pypi.patch
# Install package.
python setup.py install
I had the same problem as the original post. For me, this error:
In file included from fitsne/src/nbodyfft.cpp:4:
fitsne/src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
#include <fftw3.h>
^~~~~~~~~
compilation terminated.
turned out to be the main source of the problem. This was solved by installing the libfftw3-dev package (using apt in Debian). After this, I could do
pip install fitsne
without any errors.
Package/system info:
- Debian GNU/Linux 10 (buster)
- Cython 0.29.21
- gcc version 8.3.0 (Debian 8.3.0-6)
