autodiff icon indicating copy to clipboard operation
autodiff copied to clipboard
verified publisher icon autodiff

Unable to build Python bindings with current master

Open rath3t opened this issue 2 years ago • 7 comments

I rebuild my docker container with autodiff and now it fails unfortunately.

The autodiff specific command is:

RUN  git clone https://github.com/autodiff/autodiff && \
  cd autodiff/ && \
  mkdir .build && \
  cd .build/ && \
  cmake .. -DAUTODIFF_BUILD_PYTHON=1 -DAUTODIFF_BUILD_EXAMPLES=0 -DAUTODIFF_BUILD_DOCS=0 -DAUTODIFF_BUILD_TESTS=0 -DCMAKE_CCACHE_IGNORE=1 && \
  cmake --build . --target install --parallel 2

which produces the output

0.337 Cloning into 'autodiff'...
1.836 -- The CXX compiler identification is GNU 12.2.0
1.851 -- Detecting CXX compiler ABI info
1.936 -- Detecting CXX compiler ABI info - done
1.944 -- Check for working CXX compiler: /usr/bin/c++ - skipped
1.945 -- Detecting CXX compile features
1.945 -- Detecting CXX compile features - done
 3.280 -- Found Python: /usr/bin/python3 (found version "3.11.2") found components: Interpreter Development Development.Module Development.Embed 
3.306 -- Performing Test HAS_FLTO
3.407 -- Performing Test HAS_FLTO - Success
3.408 -- Found pybind11: /usr/local/include (found version "2.11.1")
3.412 -- Configuring done
3.420 -- Generating done
3.421 -- Build files have been written to: /autodiff/.build
3.463 [  8%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual1st.py.cxx.o
3.463 [  8%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual0th.py.cxx.o
10.75 [ 12%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual2nd.py.cxx.o
10.80 [ 16%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual3rd.py.cxx.o
17.08 [ 20%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual4th.py.cxx.o
17.16 [ 25%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal0th.py.cxx.o
23.35 [ 29%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal1st.py.cxx.o
23.44 [ 33%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal2nd.py.cxx.o
29.53 [ 37%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal3rd.py.cxx.o
29.58 [ 41%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal4th.py.cxx.o
35.87 [ 45%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual0th.py.cxx.o
35.87 [ 50%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual1st.py.cxx.o
40.34 [ 54%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual2nd.py.cxx.o
40.37 [ 58%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual3rd.py.cxx.o
44.79 [ 62%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual4th.py.cxx.o
44.82 [ 66%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal0th.py.cxx.o
49.29 [ 70%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal1st.py.cxx.o
49.33 [ 75%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal2nd.py.cxx.o
53.76 [ 79%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal3rd.py.cxx.o
53.82 [ 83%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal4th.py.cxx.o
58.36 [ 87%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/autodiff.py.cxx.o
58.42 [ 91%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/dual.py.cxx.o
60.64 [ 95%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/real.py.cxx.o
67.53 [100%] Linking CXX shared module autodiff4py.cpython-311-x86_64-linux-gnu.so
74.04 lto-wrapper: warning: using serial compilation of 12 LTRANS jobs
74.04 lto-wrapper: note: see the '-flto' option documentation for more information
116.2 [100%] Built target autodiff4py
116.3 /bin/sh: 1: setup.py: not found
116.3 gmake[2]: *** [python/package/CMakeFiles/autodiff-setuptools.dir/build.make:72: python/package/CMakeFiles/autodiff-setuptools] Error 127
116.3 gmake[1]: *** [CMakeFiles/Makefile2:175: python/package/CMakeFiles/autodiff-setuptools.dir/all] Error 2
116.3 gmake: *** [Makefile:136: all] Error 2

If I compile with checking out v1.0.3 It still works. Thus, the following just works

RUN  git clone https://github.com/autodiff/autodiff && \
  cd autodiff/ && \
  git checkout v1.0.3 && \
  mkdir .build && \
  cd .build/ && \
  cmake .. -DAUTODIFF_BUILD_PYTHON=1 -DAUTODIFF_BUILD_EXAMPLES=0 -DAUTODIFF_BUILD_DOCS=0 -DAUTODIFF_BUILD_TESTS=0 -DCMAKE_CCACHE_IGNORE=1 && \
  cmake --build . --target install --parallel 2

I'm not sure what further information is needed. Let me know what I can further provide. Thanks!

rath3t avatar Dec 12 '23 07:12 rath3t

Hi @rath3t , thanks for reporting this. autodiff relies on conda as dependency manager and on conda-forge packages as deps. Using the conda environment autodiff produced with the file environment.devenv.yml in the root dir, the compilation happens without any issues (see below).

Using gcc 13.2.0-3.

image

allanleal avatar Dec 12 '23 10:12 allanleal

Sorry for being stupid, but I'm not super sure what your comment has for consequences for my build script. I have several question, again sorry for asking stupid questions.

  1. Are the Python binding not buildable with just my cmake script, if yes what changed from the version I gave before?
  2. What do I have to do to fix my build? Install conda and then execute the build from within this environment as indicated in https://autodiff.github.io/installation/ with conda install conda-forge::autodiff? I don't know if I can do that. My use case is as follows. I want to build the Python binding package. Then later (in another Docker container), I want to install it in a given venv, (which does not use conda, I suppose?), which is out of my control.

Thanks!

rath3t avatar Dec 13 '23 08:12 rath3t

Hi Alex, my point was simply that following the standard build setup adopted for autodiff, (e.g., using conda deps) I could successfully compile the Python bindings. Unfortunately I did not have time to try replicating similar conditions as in your machine. pybind11 has LTO enabled by default:

https://pybind11.readthedocs.io/en/stable/compiling.html#:~:text=As%20stated%20above%2C%20LTO%20is%20enabled%20by%20default.

In the CMakeCache.txt file in the build directory (where I could compile and link without issues the Python bindings), I see:

//Details about finding Python
FIND_PACKAGE_MESSAGE_DETAILS_Python:INTERNAL=[/home/allanleal/miniforge3/envs/autodiff/bin/python3.10][/home/allanleal/miniforge3/envs/autodiff/include/python3.10][/home/allanleal/miniforge3/envs/autodiff/lib/libpython3.10.so][cfound components: Interpreter Development Development.Module Development.Embed ][v3.10.13()]
//Test HAS_FLTO
HAS_FLTO:INTERNAL=1

As to your question on using conda, here are the instructions in the website:

https://autodiff.github.io/installation/#development-environment

Let me know if you have additional questions.

allanleal avatar Dec 13 '23 14:12 allanleal

Thanks Allan, i will let you know, if I find the reason, why this fails. I think somehow it is not an issue of dependencies, is it? Since the error setup.py: not found indicates that somehow this setup.py file is not generated maybe from the cmake config file setup.py.in? I don't know i will have to investigate in detail. Thanks!

rath3t avatar Dec 15 '23 08:12 rath3t

The installation instructions have headers formatted in a way that makes it easy to miss that a conda environment has to be created and activated before starting with CMake.

This is the order of steps I followed to finally be able to compile successfully:

  1. git clone https://github.com/autodiff/autodiff and then (obviously) cd autodiff
  2. conda install -c conda-forge conda-devenv (for me conda-devenv was not available on the defaults channel)
  3. conda devenv: generates environment.yml to create a conda environment out of
  4. Activate environment: conda activate autodiff. Now we're ready to follow the instructions laid out in Installation using CMake. No need to git clone again.
  5. mkdir .build && cd .build
  6. cmake ..
  7. cmake --build . --target install

abhishek47kashyap avatar Jan 19 '24 21:01 abhishek47kashyap

Same error is reported here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=autodiff#n29

==> Making package: autodiff 1.1.0-1 (Sat Mar 23 14:33:36 2024)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found autodiff-1.1.0.tar.gz
==> Validating source files with sha512sums...
    autodiff-1.1.0.tar.gz ... Passed
==> Removing existing $srcdir/ directory...
==> Extracting sources...
  -> Extracting autodiff-1.1.0.tar.gz with bsdtar
==> Sources are ready.
==> Making package: autodiff 1.1.0-1 (Sat Mar 23 14:33:42 2024)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Starting build()...
-- The CXX compiler identification is GNU 13.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/sbin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- No CUDA compiler found
-- Found Python: /usr/sbin/python3.11 (found version "3.11.8") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /usr/include (found version "2.11.1")
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/makepkg/autodiff/src/build
[  4%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal0th.py.cxx.o
[  8%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual4th.py.cxx.o
[ 12%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal2nd.py.cxx.o
[ 16%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal1st.py.cxx.o
[ 20%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal3rd.py.cxx.o
[ 25%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual0th.py.cxx.o
[ 33%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual1st.py.cxx.o
[ 37%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual1st.py.cxx.o
[ 41%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual3rd.py.cxx.o
[ 37%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXdual2nd.py.cxx.o
[ 45%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual2nd.py.cxx.o
[ 54%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal0th.py.cxx.o
[ 54%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal2nd.py.cxx.o
[ 58%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal1st.py.cxx.o
[ 66%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal3rd.py.cxx.o
[ 66%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXreal4th.py.cxx.o
[ 70%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/ArrayXreal4th.py.cxx.o
[ 75%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual0th.py.cxx.o
[ 83%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual3rd.py.cxx.o
[ 83%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/VectorXdual4th.py.cxx.o
[ 87%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/autodiff.py.cxx.o
[ 91%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/dual.py.cxx.o
[ 95%] Building CXX object python/bindings/CMakeFiles/autodiff4py.dir/real.py.cxx.o
[100%] Linking CXX shared module autodiff4py.cpython-311-x86_64-linux-gnu.so
[100%] Built target autodiff4py
/bin/sh: line 1: setup.py: command not found
make[2]: *** [python/package/CMakeFiles/autodiff-setuptools.dir/build.make:72: python/package/CMakeFiles/autodiff-setuptools] Error 127
make[1]: *** [CMakeFiles/Makefile2:175: python/package/CMakeFiles/autodiff-setuptools.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
==> ERROR: A failure occurred in build().
    Aborting...
 -> error making: autodiff-exit status 4
checking dependencies...
:: git optionally requires python: git svn & git p4
:: glib2 optionally requires python: gdbus-codegen, glib-genmarshal, glib-mkenums, gtester-report
:: iproute2 optionally requires python: for routel
:: libevent optionally requires python: event_rpcgen.py
:: libxml2 optionally requires python: Python bindings
:: util-linux-libs optionally requires python: python bindings to libmount

Packages (32) cppdap-1.58.0-1  hicolor-icon-theme-0.17-3  jsoncpp-1.9.5-2  libnsl-2.0.1-1  libuv-1.48.0-1  python-3.11.8-1
              python-annotated-types-0.6.0-1  python-autocommand-2.2.2-4  python-fastjsonschema-2.19.1-1  python-inflect-7.0.0-3
              python-jaraco.context-4.3.0-3  python-jaraco.functools-4.0.0-1  python-jaraco.text-3.12.0-1
              python-more-itertools-10.2.0-1  python-ordered-set-4.1.0-4  python-packaging-23.2-1  python-platformdirs-4.2.0-1
              python-pydantic-2.6.4-1  python-pydantic-core-1:2.16.3-1  python-pyproject-hooks-1.0.0-5  python-tomli-2.0.1-3
              python-trove-classifiers-2024.3.3-1  python-typing_extensions-4.10.0-1  python-validate-pyproject-0.13-1  rhash-1.4.4-1
              cmake-3.29.0-1  eigen-3.4.0-2  pybind11-2.11.1-2  python-build-1.1.1-2  python-installer-0.7.0-4
              python-setuptools-1:69.0.3-4  python-wheel-0.43.0-1

Total Removed Size:  180.05 MiB

:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
removing python-setuptools...
removing python-validate-pyproject...
removing python-trove-classifiers...
removing python-fastjsonschema...
removing python-tomli...
removing python-platformdirs...
removing python-ordered-set...
removing python-jaraco.text...
removing python-jaraco.functools...
removing python-jaraco.context...
removing python-inflect...
removing python-pydantic...
removing python-pydantic-core...
removing python-annotated-types...
removing python-typing_extensions...
removing python-autocommand...
removing python-more-itertools...
removing python-installer...
removing python-build...
removing python-pyproject-hooks...
removing pybind11...
removing eigen...
removing cmake...
removing rhash...
removing libuv...
removing jsoncpp...
removing hicolor-icon-theme...
removing cppdap...
removing python-wheel...
removing python-packaging...
removing python...
removing libnsl...
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
 -> Failed to install the following packages. Manual intervention is required:
autodiff - exit status 4

carlosal1015 avatar Mar 23 '24 19:03 carlosal1015

https://github.com/autodiff/autodiff/blob/6690fed87d2723358ace700cc27bbcade39e3d47/python/package/CMakeLists.txt#L9

I found this line wrong. find_package(Python) set Python_EXECUTABLE instead of PYTHON_EXECUTABLE. (see: https://cmake.org/cmake/help/latest/module/FindPython.html)

This generates

cd /home/bruce/.cache/paru/clone/autodiff/src/build/python/package && setup.py --quiet build --force

at python/package/CMakeFiles/autodiff-setuptools.dir/build.make:72 which case

setup.py: command not find

I also write a simple cmake script for test

cmake_minimum_required(VERSION 3.16)

project(fp VERSION 1.0.0)

find_package(Python COMPONENTS Interpreter Development)

message("PYTHON_EXECUTABLE: ${PYTHON_EXECUTABLE}")
message("Python_EXECUTABLE: ${Python_EXECUTABLE}")

output:

-- The C compiler identification is GNU 13.2.1
-- The CXX compiler identification is GNU 13.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python: /usr/bin/python3.11 (found version "3.11.8") found components: Interpreter Development Development.Module Development.Embed
PYTHON_EXECUTABLE:
Python_EXECUTABLE: /usr/bin/python3.11
-- Configuring done (0.7s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/find-python

BruceKangCN avatar Mar 30 '24 16:03 BruceKangCN