scikit-hep
Re-building from source during development
If I try to re-build the package from source after making any changes I get the following error: error: [Errno 17] File exists: 'build/lib.linux-x86_64-cpython-39/fastjet/_fastjet_core'.
Is there any recommendation for how to re-build after making changes?
And, relatedly, if the changes are only to the Python code or the extension, is there a way to re-build only those components?
Unfortunately, I don't know a solution to the first question. I usually delete and start from scratch :/
For the second part, yes. You can absolutely only recompile _ext.cpp. The commands should look like:
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall \
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 \
-fPIC -I$(pwd)/src/fastjet/_fastjet_core/include -I$(pwd)/pybind11/include -I$PYTHON_INCLUDE \
-c src/_ext.cpp -o build/temp.linux-x86_64-3.10/src/_ext.o -fvisibility=hidden -std=c++11
x86_64-linux-gnu-g++ -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions \
-fwrapv -O2 -Wl,-Bsymbolic-functions -fstack-protector-strong -Wformat -Werror=format-security \
-Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.10/src/_ext.o \
-L$(pwd)/src/fastjet/_fastjet_core/lib -Wl,--enable-new-dtags,-R\$ORIGIN/_fastjet_core/lib \
-lfastjet -lfastjettools -lsiscone -lsiscone_spherical -lfastjetplugins \
-o build/lib.linux-x86_64-3.10/fastjet/_ext.cpython-310-x86_64-linux-gnu.so
You might want to adjust these according to your setup.
Python builds are not incremental; pip specifically isolates the build directory so that it can't use pre-compiled parts. This error message is saying that's not what's happening here (it's complaining about the build directory already existing), so I'm guessing you didn't pip install ., but maybe python setup.py?
Anyway, you can't expect any continuity between one pip-install and another. The solution to this conflict would be to delete the build directory, possibly via a clean git clone of the repo (I do that a lot, to make sure I'm starting from a clean workbench) and/or
pip uninstall fastjet
multiple times until there are no versions of it installed anywhere (systemwide or in ~/.local).
@chrispap95's answer just appeared; we seem to be in exact disagreement. I'll keep going with my answer because this how I understand it. (It may be that it's possible to rebuild only part of the system by taking advantage of the specific way that setuptools lays out a directory structure, but I think that's not recommended.)
About only changing the Python part: unfortunately, it's all bundled together as a single Python wheel, you can't rebuild just the Python part, taking the already compiled C++ from somewhere else (in a sanctioned way, at least; or maybe @chrispap95 knows more than I do on this point). What you might be able to do, instead, is
pip install -e -vv .
in the fastjet git directory (assuming that your workbench is clean: no build directory, no previous installations of the wheel on your system, etc.). This "editable install" makes a symlink to the source code, so you can change the Python without reinstalling and those changes are immediately felt in the installed package. Naturally, the same can't be true of the C++; it needs a compilation cycle, and therefore the entire build process.
I say "might" because we had trouble with editable installations in Awkward Array, ans to solve that problem, we're breaking it into two packages: https://github.com/scikit-hep/awkward/pull/1778. It makes more sense to do the two-package thing with Awkward Array (which has a little C++ and a lot of Python, with most of the logic and continuing developing in the Python part) than it does for fastjet (which is a thin wrapper of a C++ codebase).
So try an editable installation (with a clean workbench) and see if it works!
Or try @chrispap95 selective recompilation (which might or might not be a hack)!
Yes, definitely not the proper way of compiling, especially when one is not familiar with the process. It can easily produce broken libraries when improperly used. It should only be used within isolated (and easy to discard) environments. But it can speed up the debugging process a lot.
Thank you for the responses! Indeed I was doing a fresh clone each time (since just removing the build directory wasn't enough - this was with pip install '.[test]') which certainly works, but is a bit of a hassle especially if I've only made a minor edit in _ext.cpp or one of the Python files. I will try out the editable installing and selective recompilation suggestions.
I tried the editable installation pip install -e '.[test]' and the build worked, but when I try to import fastjet I receive ModuleNotFoundError: No module named 'fastjet._swig' - do you know what I could be missing?
Can you check whether _swig.py exists? You can find the path by doing fastjet.__path__ in the python prompt.
Edit: that would check whether the link exists which is also useful. The original file should be under build/lib.linux-x86_64-cpython-311/fastjet/_swig.py.
The path is fastjet/src/fastjet and _swig.py doesn't exist either there or in build/lib.linux-x86_64-cpython-39/fastjet/
It does not, and in fact upon doing a fresh clone and install, the build folder itself does not exist.
To be honest, I am not familiar with editable installs, so I am not sure if we should expect a build folder or not. However, a _swig.py file should exist in any case. I suspect that the editable install process is incompatible with the way we handle files in our setup.py. This file specifically is the result of the fastjet-core compilation that is named initially _fastjet.py and needs to be renamed and copied to the correct destination. Apparently, something breaks during this. I can give it a try later.
FYI, editable installs are really hard to support in modern setuptools (64 or something adapted to the new editable mode support from PEP 660 rather than using a custom back-door in pip to support it). I'm quite sure the handwritten setup.py's wouldn't support it, and scikit-build-core doesn't support it yet. I think it would also require CMake 3.15+ (part of why scikit-build core sets that as the minimum). I think there's a setuptools flag to go back to the old system that doesn't use PEP 660 for editable installs, using that likely might work.
Thank you! Just as an update, I've found probably the worst solution for this which is to build it normally and then manually simlink the .py files in my build directory.
Has anyone happened upon a good solution to this rebuilding issue in the last few months? Or is pulling a fresh clone still the only real way to get my changes applied?
