Up-casting values on the fly to avoid overflow

[sp-1234]

Consider the following simple example:

def calc_squares_sum(ch: bytearray, size: int):
    a = numpy.frombuffer(ch, dtype='uint8', count=size)
    a = a.astype(numpy.int64, copy=False)  # Apparently it does copy anyway, when the type is different
    return numexpr.evaluate("sum(a**2)")

We have an array of uint8 values (by a problem definition) and we want to compute sum of squares. The above code is what works, but it has to make a copy of the array and apparently it's what makes it slower.

If the conversion is omitted, then it's about 1.5 times faster but the answer is wrong (I think it's because numexpr does the squaring without converting to larger type).

Is there a possibility to introduce upcasting right in the expression so that the array copy won't be needed? So that something like this "sum(int64(a)**2)" would work and give correct answer.

[robbmcleod]

Explicit casting is on the timeline for NumExpr3.0. In the meantime, I would consider using NumPy to do the sum instead. I.e.

return numexpr.evaluate( 'a*a' ).sum()

It may be faster and it will upcast.

[sp-1234]

@robbmcleod but it defeats the purpose of using numexpr(avoiding intermediate array copies)? or numexpr.evaluate( 'a*a' ) is not a copy but just a "view" over a?

[robbmcleod]

With NumExpr 2.6 there might be some memory-use advantage to using its sum but I haven't seem a benchmark time advantage to it, because it's single-threaded. I think this is one of those cases where practicality wins over the design goal.

[github-actions[bot]]

Message to comment on stale issues. If none provided, will not mark issues stale