How to build an executable while installing in normal and editable mode

[lionlai1989]

Hello, I'm currently learning how to use CMake and pybind11 with this project. As part of the installation process, I need to build and install a C++ source file. To ensure that everything is clear, I've included the reproduction code below.

Description

• My project structure: It's basically the same except there is my cpp source file helloworld.cpp in src/ folder.

$ tree -L 2
.
├── CMakeLists.txt
├── pybind11
├── pyproject.toml
├── setup.py
└── src
    ├── helloworld.cpp
    └── main.cpp

• src/helloworld.cpp: A trivial helloworld program. $ cat src/helloworld.cpp

#include <iostream>

int main(int argc, char **argv) {
  std::cout << "Hello World!!!\n";
  return 0;
}

• CMakeLists.txt: It's basically the same as CMakeLists.tx except add_executable(helloworld src/helloworld.cpp) is added to build executable.

$ cat CMakeLists.txt 
cmake_minimum_required(VERSION 3.22)
project(cmake_example)

add_executable(helloworld src/helloworld.cpp)

add_subdirectory(pybind11)
pybind11_add_module(cmake_example src/main.cpp)

target_compile_definitions(cmake_example
                           PRIVATE VERSION_INFO=${EXAMPLE_VERSION_INFO})

• pyproject.toml:

$ cat pyproject.toml 
[build-system]
requires = ["setuptools>=62.0.0", "wheel", "cmake>=3.22"]
build-backend = "setuptools.build_meta"

• pybind11 is a git submodule, and setup.py is exactly the same as this setup.py.

Build in normal mode.

• Create virtual env and update pip: python3 -m venv venv && source venv/bin/activate && python3 -m pip install --upgrade pip
• Install in normal mode: python3 -m pip install .
• cmake_example library is installed correctly but my helloworld program is not. There is no helloworld program in venv.

$ ls venv/lib/python3.8/site-packages/
cmake_example-0.0.1.dist-info             
cmake_example.cpython-38-x86_64-linux-gnu.so

Build in editable mode.

• Create virtual env and update pip: python3 -m venv venv && source venv/bin/activate && python3 -m pip install --upgrade pip
• Install in editable mode: python3 -m pip install -e .
• cmake_example library is also installed correctly but my helloworld program is not. There is no helloworld program created in the current folder.

$ ls
cmake_example.cpython-38-x86_64-linux-gnu.so  cmake_example.egg-info  CMakeLists.txt  pybind11  pyproject.toml  setup.py  src  venv

My experiment:

I actually tried to make it work by modifying setup.py with the following way.

• Add code snippet in setup.py in Line:47.

        bindir = extdir / "bin"
        print(f"User's executables resides in {bindir}")
        cmake_args = [
            # My innovation. helloworld can be called in scripts.
            f"-DCMAKE_RUNTIME_OUTPUT_DIRECTORY={bindir}{os.sep}",
            f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={extdir}{os.sep}",
            f"-DPYTHON_EXECUTABLE={sys.executable}",
            f"-DCMAKE_BUILD_TYPE={cfg}",  # not used on MSVC, but no harm
        ]

This approach kind of work. Now I can do pip install . and helloworld program appears in venv/lib/python3.8/site-packages/bin/helloworld. However, this approach doesn't work in editable model. I.e., I do pip install -e ., but helloworld program is no created anywhere in the project.

My question is how to achieve the following goal: I would like the helloworld program to be created automatically when I run pip install . and pip install -e ., and I want this program to be installed in the correct Python root path in both modes. Do you have any suggestions or advice on how I can make this happen? Thank you for your help!

System info: OS: ubuntu20.04 Python: Python3.8

[district10]

@lionlai1989

I did something similar yesterday, by adding set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}) to cmake, executables were added to pip wheel.

Example code: https://github.com/cubao/cubao-tippecanoe/blob/pypi-release/CMakeLists.txt

$ unzip -l dist/cubao_tippecanoe-0.0.1-cp310-cp310-linux_x86_64.whl
Archive:  dist/cubao_tippecanoe-0.0.1-cp310-cp310-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
  1773952  2023-09-17 15:38   __cubao_tippecanoe_decode.exe
  1301248  2023-09-17 15:38   __cubao_tippecanoe_enumerate.exe
    78640  2023-09-17 15:38   __cubao_tippecanoe_jsontool.exe
   537496  2023-09-17 15:38   __cubao_tippecanoe_overzoom.exe
  2123648  2023-09-17 15:38   __cubao_tippecanoe_tile-join.exe
  2657384  2023-09-17 15:38   __cubao_tippecanoe_tippecanoe.exe
   175960  2023-09-17 15:38   _pybind11_tippecanoe.cpython-310-x86_64-linux-gnu.so
       95  2023-09-17 09:11   cubao_tippecanoe/__init__.py
      687  2023-09-17 15:31   cubao_tippecanoe/__main__.py
     1329  2023-09-17 15:38   cubao_tippecanoe-0.0.1.dist-info/LICENSE.md
    63751  2023-09-17 15:38   cubao_tippecanoe-0.0.1.dist-info/METADATA
      105  2023-09-17 15:38   cubao_tippecanoe-0.0.1.dist-info/WHEEL
       17  2023-09-17 15:38   cubao_tippecanoe-0.0.1.dist-info/top_level.txt
     1259  2023-09-17 15:38   cubao_tippecanoe-0.0.1.dist-info/RECORD
---------                     -------
  8715571                     14 files