Numpy sum imap

[pbrunet]

WIP : I have to add some tests and do the itertools to numpy_expr transformation for others iterators like ifilter

[pbrunet]

There is an issue with this PR as numpy.sum on generator return the same generator. Reason is that numpy expression are not Lazy in numpy as numpy need to know the size of the generator to allocate memory. For this reason, replacing list with generator will never work on numpy functions. Of course, we may still want to replace some numpy call on list to numpy call to generator to avoid memory allocation.

I see 2 solution:

1. The filter-> ifilter transformation is perform after others (so after constant folding) and we accept to have non complient python at this stage. It could be done
2. We may add an attributes to builtin like ReadOnlyNDArray and in this case, the filter can be transform to -> numpy.fromiter(ifilter(...)) but it supports only 1D array (which is most of the cases but not all) and we don't have the size information which may be required to create a numpy_expr.

Any better idea? Or should I just leave this PR as it can't be really applied?

[serge-sans-paille]

On Mon, Jun 27, 2016 at 12:55:06PM -0700, pbrunet wrote:

There is an issue with this PR as numpy.sum on generator return the same generator. Reason is that numpy expression are not Lazy in numpy as numpy need to know the size of the generator to allocate memory. For this reason, replacing list with generator will never work on numpy functions. Of course, we may still want to replace some numpy call on list to numpy call to generator to avoid memory allocation.

I see 2 solution:

1. The filter-> ifilter transformation is perform after others (so after constant folding) and we accept to have non complient python at this stage. It could be done
2. We may add an attributes to builtin like ReadOnlyNDArray and in this case, the filter can be transform to -> numpy.fromiter(ifilter(...)) but it supports only 1D array (which is most of the cases but not all) and we don't have the size information which may be required to create a numpy_expr.

Isn't this similar to saying that if we have a 1D array, we cann turn numpy.sum into sum? Eitherway, we have no way to know at compile time the dimension of an array. This looks dommed to me and I'b better say « do nothing ». Do yiu have a test case that exhibits a ral use for this one?