PyElastica
PyElastica copied to clipboard
GazzolaLab
[GSoC 2024] Binding PyElastica with Elastica++
This branch and comment will be kept updated as I work on this proposal
Week 1 (May 27 - June 2)
This week, I'm mostly reading up on and experimenting with build systems.
- In the first commit, I integrate
scikit-build-core(which usescmake), removing the currentsetup.pyin the process. The metadata from the removedsetup.pyhas been ported topyproject.toml - In the second commit, I add a basic
meson-python(which usesmeson) based system that also integrates with a pybind11 example, to serve as a POC. This example compiles and is usable from python code.
The goal this week is to experiment and compare both systems side by side in action, and later pick and polish the more suitable option (based on mentor feedback)
Week 2 (June 3 - June 9)
This week I played around with dependency handling. I was keen to look into the "meson wraps" system, but I quickly hit into a couple of issues with this approach.
For one, to install blaze and blaze_tensor, I had to pass custom arguments to the underlying cmake build, which was not possible via the simple meson wraps interface. I also had other troubles/errors while trying to make wraps I had written work for the other dependencies. The only dependencies that worked via wraps were brigand and cxxopts.
To handle blaze, blaze_tensor and sleef I wrote a custom shell script that can clone, build and install them in a deps directory in the source tree. I got the meson buildconfig to work with these dependencies.
Week 3 (June 10 - June 16)
I experimented with and implemented the batch_matmul linalg function in different ways in C++ for testing and benchmarking purposes. Added the elasticapp._linalg submodule for the same.
Also added a simple benchmarking script in the backend/benchmarks folder.
Week 4 (June 17 - June 23)
I wrapped 6 linalg functions that were in Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp in the python elasticapp._linalg module and also added tests and basic docs for the same.
I spent a while trying to find out more optimal ways to do the wrapping, and I could not figure out a way to improve upon my current naive approach (which involves a lot of copying in input/output arrays).
Week 5 (June 24 - June 30)
[TODO]
Week 6 (July 1 - July 7)
[TODO]
Week 7 (July 8 - July 14)
[TODO]
Week 8 (July 15 - July 21)
[TODO]
Week 9 (July 22 - July 28)
[TODO]
Week 10 (July 29 - August 4)
[TODO]
Week 11 (August 5 - August 11)
[TODO]
Week 12 (August 12 - August 18)
[TODO]
Final week (August 19 - August 26)
[TODO]
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 93.98%. Comparing base (
581672c) to head (70a073b). Report is 3 commits behind head on elasticapp-integration.
Additional details and impacted files
@@ Coverage Diff @@
## elasticapp-integration #391 +/- ##
=======================================================
Coverage 93.98% 93.98%
=======================================================
Files 51 51
Lines 3193 3193
Branches 347 347
=======================================================
Hits 3001 3001
Misses 142 142
Partials 50 50
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Errors I'm getting when trying to compile Utilities/Math/Python
(click to expand) C++ compiler error log
FAILED: src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeMatrix.cpp.o
c++ -Isrc/_PyArrays.cpython-312-x86_64-linux-gnu.so.p -Isrc -I../src -I/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include -I/usr/include/python3.12 -I/usr/include/x86_64-linux-gnu/python3.12 -I/home/ankith/mount/gitstuff/personal/ankith26/PyElastica/deps/blaze -I/home/ankith/mount/gitstuff/personal/ankith26/PyElastica/deps/blaze_tensor -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeMatrix.cpp.o -MF src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeMatrix.cpp.o.d -o src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeMatrix.cpp.o -c ../src/Utilities/Math/Python/BlazeMatrix.cpp
In file included from ../src/Utilities/Math/Python/BlazeMatrix.cpp:11:
../src/Utilities/Math/Python/SliceHelpers.hpp: In function ‘auto py_bindings::check_slice(std::size_t, pybind11::slice)’:
../src/Utilities/Math/Python/SliceHelpers.hpp:43:46: error: invalid operands of types ‘const char [33]’ and ‘long unsigned int’ to binary ‘operator<<’
43 | "step !=1 unsupported along axis " << Axis));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
../src/Utilities/Math/Python/BlazeMatrix.cpp: In lambda function:
../src/Utilities/Math/Python/BlazeMatrix.cpp:95:56: error: ‘MakeString’ was not declared in this scope
95 | +[](const type& self) { return std::string(MakeString{} << self); })
| ^~~~~~~~~~
../src/Utilities/Math/Python/BlazeMatrix.cpp: In function ‘void py_bindings::bind_blaze_matrix(pybind11::module&)’:
../src/Utilities/Math/Python/BlazeMatrix.cpp:95:13: error: no match for ‘operator+’ (operand type is ‘py_bindings::bind_blaze_matrix(pybind11::module&)::<lambda(const type&)>’)
95 | +[](const type& self) { return std::string(MakeString{} << self); })
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[5/9] Compiling C++ object src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeTensor.cpp.o
FAILED: src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeTensor.cpp.o
c++ -Isrc/_PyArrays.cpython-312-x86_64-linux-gnu.so.p -Isrc -I../src -I/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include -I/usr/include/python3.12 -I/usr/include/x86_64-linux-gnu/python3.12 -I/home/ankith/mount/gitstuff/personal/ankith26/PyElastica/deps/blaze -I/home/ankith/mount/gitstuff/personal/ankith26/PyElastica/deps/blaze_tensor -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeTensor.cpp.o -MF src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeTensor.cpp.o.d -o src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeTensor.cpp.o -c ../src/Utilities/Math/Python/BlazeTensor.cpp
In file included from ../src/Utilities/Math/Python/BlazeTensor.cpp:11:
../src/Utilities/Math/Python/SliceHelpers.hpp: In function ‘auto py_bindings::check_slice(std::size_t, pybind11::slice)’:
../src/Utilities/Math/Python/SliceHelpers.hpp:43:46: error: invalid operands of types ‘const char [33]’ and ‘long unsigned int’ to binary ‘operator<<’
43 | "step !=1 unsupported along axis " << Axis));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
../src/Utilities/Math/Python/BlazeTensor.cpp: In lambda function:
../src/Utilities/Math/Python/BlazeTensor.cpp:86:49: error: expected ‘;’ before ‘)’ token
86 | std::get<2>(x));
| ^
| ;
../src/Utilities/Math/Python/BlazeTensor.cpp: In lambda function:
../src/Utilities/Math/Python/BlazeTensor.cpp:100:49: error: expected ‘;’ before ‘)’ token
100 | std::get<2>(x));
| ^
| ;
../src/Utilities/Math/Python/BlazeTensor.cpp: In lambda function:
../src/Utilities/Math/Python/BlazeTensor.cpp:106:34: error: ‘MakeString’ was not declared in this scope
106 | return std::string(MakeString{} << self);
| ^~~~~~~~~~
../src/Utilities/Math/Python/BlazeTensor.cpp: In function ‘void py_bindings::bind_blaze_tensor(pybind11::module&)’:
../src/Utilities/Math/Python/BlazeTensor.cpp:105:24: error: no match for ‘operator+’ (operand type is ‘py_bindings::bind_blaze_tensor(pybind11::module&)::<lambda(const type&)>’)
105 | "__str__", +[](const type& self) {
| ^~~~~~~~~~~~~~~~~~~~~~~
106 | return std::string(MakeString{} << self);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107 | });
| ~
[6/9] Compiling C++ object src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeVector.cpp.o
FAILED: src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeVector.cpp.o
c++ -Isrc/_PyArrays.cpython-312-x86_64-linux-gnu.so.p -Isrc -I../src -I/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include -I/usr/include/python3.12 -I/usr/include/x86_64-linux-gnu/python3.12 -I/home/ankith/mount/gitstuff/personal/ankith26/PyElastica/deps/blaze -I/home/ankith/mount/gitstuff/personal/ankith26/PyElastica/deps/blaze_tensor -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeVector.cpp.o -MF src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeVector.cpp.o.d -o src/_PyArrays.cpython-312-x86_64-linux-gnu.so.p/Utilities_Math_Python_BlazeVector.cpp.o -c ../src/Utilities/Math/Python/BlazeVector.cpp
In file included from ../src/Utilities/Math/Python/BlazeVector.cpp:9:
../src/Utilities/Math/Python/SliceHelpers.hpp: In function ‘auto py_bindings::check_slice(std::size_t, pybind11::slice)’:
../src/Utilities/Math/Python/SliceHelpers.hpp:43:46: error: invalid operands of types ‘const char [33]’ and ‘long unsigned int’ to binary ‘operator<<’
43 | "step !=1 unsupported along axis " << Axis));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
../src/Utilities/Math/Python/BlazeVector.cpp: In function ‘void py_bindings::bind_blaze_vector(pybind11::module&)’:
../src/Utilities/Math/Python/BlazeVector.cpp:115:13: error: no matching function for call to ‘pybind11::class_<blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> > >::def(const char [8])’
114 | py_vector
| ~~~~~~~~~
115 | .def("__str__")
| ~~~~^~~~~~~~~~~
In file included from ../src/Utilities/Math/Python/SliceHelpers.hpp:16:
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1668:13: note: candidate: ‘template<class Func, class ... Extra> pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(const char*, Func&&, const Extra& ...) [with Extra = Func; type_ = blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> >; options = {}]’
1668 | class_ &def(const char *name_, Func &&f, const Extra &...extra) {
| ^~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1668:13: note: template argument deduction/substitution failed:
../src/Utilities/Math/Python/BlazeVector.cpp:115:13: note: candidate expects at least 2 arguments, 1 provided
114 | py_vector
| ~~~~~~~~~
115 | .def("__str__")
| ~~~~^~~~~~~~~~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1693:13: note: candidate: ‘template<class T, class ... Extra, typename std::enable_if<T::op_enable_if_hook, int>::type <anonymous> > pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(const T&, const Extra& ...) [with Extra = T; typename std::enable_if<T::op_enable_if_hook, int>::type <anonymous> = {Extra ...}; type_ = blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> >; options = {}]’
1693 | class_ &def(const T &op, const Extra &...extra) {
| ^~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1693:13: note: template argument deduction/substitution failed:
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1692:95: error: ‘op_enable_if_hook’ is not a member of ‘char [8]’
1692 | template <typename T, typename... Extra, detail::enable_if_t<T::op_enable_if_hook, int> = 0>
| ^
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1705:13: note: candidate: ‘template<class ... Args, class ... Extra> pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(const pybind11::detail::initimpl::constructor<Args ...>&, const Extra& ...) [with Args = {Args ...}; Extra = {Extra ...}; type_ = blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> >; options = {}]’
1705 | class_ &def(const detail::initimpl::constructor<Args...> &init, const Extra &...extra) {
| ^~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1705:13: note: template argument deduction/substitution failed:
../src/Utilities/Math/Python/BlazeVector.cpp:115:13: note: mismatched types ‘const pybind11::detail::initimpl::constructor<Args ...>’ and ‘const char [8]’
114 | py_vector
| ~~~~~~~~~
115 | .def("__str__")
| ~~~~^~~~~~~~~~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1712:13: note: candidate: ‘template<class ... Args, class ... Extra> pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(const pybind11::detail::initimpl::alias_constructor<Args ...>&, const Extra& ...) [with Args = {Args ...}; Extra = {Extra ...}; type_ = blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> >; options = {}]’
1712 | class_ &def(const detail::initimpl::alias_constructor<Args...> &init, const Extra &...extra) {
| ^~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1712:13: note: template argument deduction/substitution failed:
../src/Utilities/Math/Python/BlazeVector.cpp:115:13: note: mismatched types ‘const pybind11::detail::initimpl::alias_constructor<Args ...>’ and ‘const char [8]’
114 | py_vector
| ~~~~~~~~~
115 | .def("__str__")
| ~~~~^~~~~~~~~~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1719:13: note: candidate: ‘template<class ... Args, class ... Extra> pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(pybind11::detail::initimpl::factory<Args ...>&&, const Extra& ...) [with Args = {Args ...}; Extra = {Extra ...}; type_ = blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> >; options = {}]’
1719 | class_ &def(detail::initimpl::factory<Args...> &&init, const Extra &...extra) {
| ^~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1719:13: note: template argument deduction/substitution failed:
../src/Utilities/Math/Python/BlazeVector.cpp:115:13: note: mismatched types ‘pybind11::detail::initimpl::factory<Args ...>’ and ‘const char [8]’
114 | py_vector
| ~~~~~~~~~
115 | .def("__str__")
| ~~~~^~~~~~~~~~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1725:13: note: candidate: ‘template<class ... Args, class ... Extra> pybind11::class_<type_, options>& pybind11::class_<type_, options>::def(pybind11::detail::initimpl::pickle_factory<Args ...>&&, const Extra& ...) [with Args = {Args ...}; Extra = {Extra ...}; type_ = blaze::DynamicVector<double, false, blaze::AlignedAllocator<double> >; options = {}]’
1725 | class_ &def(detail::initimpl::pickle_factory<Args...> &&pf, const Extra &...extra) {
| ^~~
/tmp/pip-build-env-bf59qldc/overlay/lib/python3.12/site-packages/pybind11/include/pybind11/pybind11.h:1725:13: note: template argument deduction/substitution failed:
../src/Utilities/Math/Python/BlazeVector.cpp:115:13: note: mismatched types ‘pybind11::detail::initimpl::pickle_factory<Args ...>’ and ‘const char [8]’
114 | py_vector
| ~~~~~~~~~
115 | .def("__str__")
| ~~~~^~~~~~~~~~~
[7/9] Compiling C++ object src/_linalg.cpython-312-x86_64-linux-gnu.so.p/_linalg.cpp.o
In file included from ../src/_linalg.cpp:5:
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp: In instantiation of ‘auto elastica::batch_dot(const MT1&, const MT2&) [with MT1 = blaze::DynamicMatrix<double>; MT2 = blaze::DynamicMatrix<double>]’:
../src/_linalg.cpp:271:44: required from here
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp:206:27: warning: unused variable ‘dimension’ [-Wunused-variable]
206 | constexpr std::size_t dimension(3UL);
| ^~~~~~~~~
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp:207:23: warning: unused variable ‘n_elems’ [-Wunused-variable]
207 | const std::size_t n_elems = first_vector_batch.columns();
| ^~~~~~~
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp: In instantiation of ‘auto elastica::batch_norm(const MT&) [with MT = blaze::DynamicMatrix<double>]’:
../src/_linalg.cpp:301:45: required from here
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp:238:27: warning: unused variable ‘dimension’ [-Wunused-variable]
238 | constexpr std::size_t dimension(3UL);
| ^~~~~~~~~
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp: In instantiation of ‘void elastica::batch_matmul(TT1&, const TT2&, const TT3&) [with TT1 = blaze::DynamicTensor<double>; TT2 = blaze::DynamicTensor<double>; TT3 = blaze::DynamicTensor<double>]’:
../src/_linalg.cpp:190:27: required from here
../src/Utilities/Math/BlazeDetail/BlazeLinearAlgebra.hpp:113:23: warning: unused variable ‘n_elems’ [-Wunused-variable]
113 | const std::size_t n_elems = first_matrix_batch.columns();
| ^~~~~~~
ninja: build stopped: subcommand failed.
[end of output]
UPDATE: this issue has been fixed now
We'll merge this gsoc -> elasticapp-integration, and continue polishing in the other branch. Thanks @ankith26
