Testing the old benchmark code

[melovv]

Hello,

I'd like to test the old benchmark code (https://github.com/pydata/numexpr/blob/master/bench/vml_timing.py) for evaluating the performance of numexpr3 against numpy. I've removed the vml configuration to use the one of numexpr3. However, it gives the following error:

Traceback (most recent call last):
  File "numexpr3_timing.py", line 146, in <module>
    compare()
  File "numexpr3_timing.py", line 130, in compare
    compare_times(expr, nexpr)
  File "numexpr3_timing.py", line 58, in compare_times
    numexpr_time = round(numexpr_timer.timeit(number=iterations), 4)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/timeit.py", line 178, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 21, in inner
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/site-packages/numexpr3/ne3compiler.py", line 213, in evaluate
    neObj = NumExpr(expr, lib=lib, casting=casting, stackDepth=stackDepth+1)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/site-packages/numexpr3/ne3compiler.py", line 988, in __init__
    self._assemble()
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/site-packages/numexpr3/ne3compiler.py", line 1033, in _assemble
    _ASTAssembler[type(bodyItem),-1](self,bodyItem)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/site-packages/numexpr3/ne3compiler.py", line 268, in _expression_last
    self.assignTarget = _mutate_last(self, targetNode, valueReg)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/site-packages/numexpr3/ne3compiler.py", line 388, in _mutate_last
    raise NotImplementedError('TODO: rewind program')
NotImplementedError: TODO: rewind program

Could you please provide an updated version of that code, please?

Thanks a lot!

[robbmcleod]

Have you tried this one?

https://github.com/pydata/numexpr/blob/numexpr-3.0/bench/ne3vsne2.py

[melovv]

Hi, thanks for the link. I just tried it and the error is below. I'm running on Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] on linux.

Benchmarking for 4 threads
Traceback (most recent call last):
  File "ne3vsne2.py", line 106, in <module>
    bench( expr, arraySizes, dtypes, N_threads=N_threads, tries=5)
  File "ne3vsne2.py", line 68, in bench
    times_np[I,J] = timeit.timeit('out = ' + expr, setup_np, number=tries)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/timeit.py", line 233, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/timeit.py", line 178, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 13, in inner
  File "mtrand.pyx", line 1312, in mtrand.RandomState.uniform
  File "mtrand.pyx", line 242, in mtrand.cont2_array_sc
TypeError: 'float' object cannot be interpreted as an integer

[melovv]

Hi, thanks for the link. I just tried it and the error is below. I'm running on Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] on linux.

Benchmarking for 4 threads
Traceback (most recent call last):
  File "ne3vsne2.py", line 106, in <module>
    bench( expr, arraySizes, dtypes, N_threads=N_threads, tries=5)
  File "ne3vsne2.py", line 68, in bench
    times_np[I,J] = timeit.timeit('out = ' + expr, setup_np, number=tries)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/timeit.py", line 233, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/home/vvmelo/anaconda3/envs/py36/lib/python3.6/timeit.py", line 178, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 13, in inner
  File "mtrand.pyx", line 1312, in mtrand.RandomState.uniform
  File "mtrand.pyx", line 242, in mtrand.cont2_array_sc
TypeError: 'float' object cannot be interpreted as an integer

Hello.

Just fixed the error. Must add dtype=int in the line below.

arraySizes = np.logspace(14,22,25,base=2,dtype=int)

I was wondering why the code does not work with float16 or float32.

Cheers.

[melovv]

Hi, again.

I was testing with a longer random expression and the results show numexpr3 is slower than numexpr2:

Benchmarking for 27 threads ===RESULTS for sqrt(AB + 2C)-A/BC+(ABCBCA)+(A+B+C+A+A+A+B)-(A-B-sqrt(C))/sqrt(A/B/C)+sqrt(AB + 2C)-A/BC+(ABCBCA)+(A+B+C+A+A+A+B)-(A-B-sqrt(C))/sqrt(A/B/C)sqrt(AB + 2C)-A/BC+(ABCBC*A)+(A+B+C+A+A+A+B)-(A-B-sqrt(C))/sqrt(A/B/C) for dtype float===

Mean time for NE2: 0.022592734638601543
Mean time for NE3: 0.05026317464187741
Mean speedup for NE3 versus NumPy: 1833.0 %
Mean speedup for NE3 versus NE2: 46.8 %

model name : Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] on linux

[robbmcleod]

Difficult to say without seeing source code, but it does look like you have common terms in that expression. NE3 permits multiple lines and assignment to temporary variables. However it does not analyze the expression for common terms like NE2 did. It was computationally expensive and one of the things that limited NE2's ability to compete with NumPy for smaller array sizes (<64k elements generally). E.g.

ne3.NumExpr('AB = A*B'
                       'C2 = C*2'
                       'D = sqrt(A/B/C)'
                       'E = sqrt(AB + C2)'
                       [etc.])()

These intermediate assignment targets in NE3 will never exist in Python unless they existed before the function call. They also won't be full-sized arrays, rather just single blocks, so it also reduces memory consumption. So one can hand optimize rather easily, and the calculation will all be done within the virtual machine, but it does require the programmer doing said optimization, whereas in NE2 the programmer could be lazier.

Someone could write some code that (optionally) does refactor code, but it's probably a problem better suited for a project like sympy. NumExpr's niche is generally to be best for quick-and-dirty speedup of NumPy-based scripts, which is why there was so much emphasis in NE3 in making it better adhere to the NumPy standards.

[github-actions[bot]]

Message to comment on stale issues. If none provided, will not mark issues stale