symengine.py Subclassing Basic

@certik hi, hope you are well! Looks like symengine is going well. I am looking at symengine for the sympy.physics.quantum stuff. Some questions:

In sympy.physics.quantum we subclass Basic here (https://github.com/sympy/sympy/blob/master/sympy/physics/quantum/qexpr.py). Is it possible to do that type of thing on top of the python bindings to symengine? Is there an examples like that?
Is it even worth rebuilding the quantum stuff on top of symengine at a deep level?
How would you go about beginning to use symengine for that type of thing where we have lots of custom subclasses of Basic.
How are assumptions handled in symengine?

Sep 28 '17 03:09 ellisonbg

I see this PR that begins to use symengine in physics.quantum, but it isn't obvious how much benefit there will be...

https://github.com/sympy/sympy/pull/13189/files

Sep 28 '17 04:09 ellisonbg

Hi @ellisonbg, I am doing great, I hope you are well too. Thanks for the interest. To answer your questions:

We would need to see. The symengine's Basic class is here: https://github.com/symengine/symengine.py/blob/3fd1e481cf2d3f6b37ec72f8ba89b75ced587283/symengine/lib/symengine_wrapper.pyx#L760, I think you can subclass it in Python. But I don't know what exact things you need for quantum.
That depends if the speed of sympy.quantum is sufficient. If not, where is the bottleneck? If SymPy operations, then it makes sense to port on top of symengine.
As you mentioned, https://github.com/sympy/sympy/pull/13189 is a start. Then we'll go from there.
We haven't implemented assumptions yet, I just made an issue for it: https://github.com/symengine/symengine/issues/1334

In general, I would add, that even though you were initially reserved about using C++ for symengine, it worked really well. The main library (https://github.com/symengine/symengine) is just C++, no Python, and it's pretty clean, and as you can see, lots of people have contributed to it, and students in general have no problems getting up to speed. The modern subset of C++ that we use allows us to be pretty much safe in Debug mode, just like in Python, for all practical purposes. In Release mode there doesn't seem to be any sacrifices in speed that we had to make. It is also completely multiplatform, it works with all major compilers. The main thing however is the speed (in Release mode). I want to implement more systematic benchmarks (https://github.com/symengine/symengine/issues/1318), but from what we ran by hand, it looks like we are one of the fastest, if not the fastest.

So if you need speed, symengine is the way to go. For quantum, do you have some test cases which you want to make faster? One can implement that in C++, then one can concentrate on the speed, data structures, algorithms, and nail it. It's work, but if speed is what you want, I think that's the best way to go. If you only need fast symbolics, then use symengine from Python, and build the rest on top in Python. That's much easier, and delivers most of the time.

Sep 28 '17 16:09 certik

Thanks for the response!

Some of the things we need to do for quantum:

A custom __new__(cls, *args) method where we process args, instantiate using Expr.__new__ and then set some slots.
A custom doit method (is that supported?)
A custom _eval_rewrite method
Other custom _eval methods such as _eval_inverse, _eval_power, _eval_adjoint, _eval_trace. Some of these are for base sympy logic to work, others are custom to quantum.
All the custom print methods for pretty, latex, etc. that return strings (not print to stdout)
Custom __mul__.

Speed might be an issue eventually for certain types of algorithms in quantum computing, but I don't want to start in C++. The speed critical will be very localized and we could drop into C++ at that point if needed. I just want to make sure our custom Basic subclasses in Python will have the right information so that when we do drop into C++, it will all work (without us having to go back and implement everything in C++).

The bigger place we struggled in building the quantum stuff was in the actual overall model and APIs of sympy. I am fine if the overall model is the same as sympy, but that stuff doesn't seem to be documented anywhere. The most subtle stuff was things like:

All the magic in __new__ with @classmethods that process args.
Attempting to have custom multiplication logic - the combination of python's model for __mul__ and sympy made this extremely subtle.
Assumptions
No clear APIs and policies for when things are evaluated (in new versus in methods such as doit).
Documentation about the protocols (such as eval*) that objects need to implement to work with other parts of the code base.

As always, I care much more about those things than speed. But we can't fully ignore speed either. Thanks!

On Thu, Sep 28, 2017 at 9:00 AM, Ondřej Čertík [email protected] wrote:

Hi @ellisonbg https://github.com/ellisonbg, I am doing great, I hope you are well too. Thanks for the interest. To answer your questions:

We would need to see. The symengine's Basic class is here: https://github.com/symengine/symengine.py/blob/ 3fd1e481cf2d3f6b37ec72f8ba89b75ced587283/symengine/lib/ symengine_wrapper.pyx#L760 https://github.com/symengine/symengine.py/blob/3fd1e481cf2d3f6b37ec72f8ba89b75ced587283/symengine/lib/symengine_wrapper.pyx#L760, I think you can subclass it in Python. But I don't know what exact things you need for quantum.

That depends if the speed of sympy.quantum is sufficient. If not, where is the bottleneck? If SymPy operations, then it makes sense to port on top of symengine.

As you mentioned, sympy/sympy#13189 https://github.com/sympy/sympy/pull/13189 is a start. Then we'll go from there.

We haven't implemented assumptions yet, I just made an issue for it: symengine/symengine#1334 https://github.com/symengine/symengine/issues/1334

In general, I would add, that even though you were initially reserved about using C++ for symengine, it worked really well. The main library ( https://github.com/symengine/symengine) is just C++, no Python, and it's pretty clean, and as you can see, lots of people have contributed to it, and students in general have no problems getting up to speed. The modern subset of C++ that we use allows us to be pretty much safe in Debug mode, just like in Python, for all practical purposes. In Release mode there doesn't seem to be any sacrifices in speed that we had to make. It is also completely multiplatform, it works with all major compilers. The main thing however is the speed (in Release mode). I want to implement more systematic benchmarks (symengine/symengine#1318 https://github.com/symengine/symengine/issues/1318), but from what we ran by hand, it looks like we are one of the fastest, if not the fastest.

So if you need speed, symengine is the way to go. For quantum, do you have some test cases which you want to make faster? One can implement that in C++, then one can concentrate on the speed, data structures, algorithms, and nail it. It's work, but if speed is what you want, I think that's the best way to go. If you only need fast symbolics, then use symengine from Python, and build the rest on top in Python. That's much easier, and delivers most of the time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/symengine/symengine.py/issues/194#issuecomment-332883129, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0MCuef3TnZmmG9mOX3RwIEwLV1yAks5sm8KjgaJpZM4Pmsii .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub [email protected] and [email protected]

Sep 28 '17 18:09 ellisonbg

@ellisonbg I see. Your main motivation is to ensure that the SymEngine design allows the incorporation of the quantum stuff, if that was ever needed in the future, so that we don't run into a dead end.

It seems most of the stuff is with relation to the Python wrappers. Few of it requires interaction on the C++ level. The C++ level allows to accept any "unknown" function implemented for example in Python, here is a test which takes SymPy's spherical harmonics (not yet implemented in C++) and passes it to the C++ symbolic expression:

https://github.com/symengine/symengine.py/blob/b03f04434d22ae80667878fd68943200e7a130be/symengine/tests/test_functions.py#L123

On the C++ level, that is tested here:

https://github.com/symengine/symengine/blob/fdf132fcb4425589b69b40d60a90234944870b28/symengine/tests/basic/test_functions.cpp#L3702

where you can see that we defined MySin, implemented in that test file, and pass it to a SymEngine C++ expression.

And we use something similar with Lambdify I believe.

So we should go over the list from your last comment, and write tests for each of these things, to ensure that our API and our design supports these idioms (both in C++ and in the Python wrappers), if they were ever needed in the future.

Sep 28 '17 18:09 certik