Invalid result for HO function with bound instancemethod argument

[quasilyte]

Numba does partially support higher-order functions and local closures, but the boundaries are not clearly defined as well as the inlining effect on it.

For HPAT, code below actually results in invalid results and no JIT compilation time error. We should fix this bug and properly document what kinds of callable values HPAT supports. After that, it would be simpler to give proper error messages to the user. Right now it's unclear what are should be compiled and what should result in a compilation error.

Reproducer:

import hpat
import pandas as pd

# Higher-order function
@hpat.jit
def indirect(call, fn):
    if call:
        return fn()
    return 0

@hpat.jit
def weird():
    df = pd.DataFrame({'A': [1]})
    return indirect(False, df.head)


# This one works fine
@hpat.jit
def indirect_ok(fn):
    return fn()

@hpat.jit
def works_as_expected():
    df = pd.DataFrame({'A': [1]})
    return indirect_ok(df.head)

print(weird())
print(works_as_expected())

Results for code without @hpat.hit decorator:

0
   A
0  1

Results for code with JIT:

Empty DataFrame
Columns: [A]
Index: []
   A
0  1

[quasilyte]

Both HPAT and Numba fail to compile this code:

import numpy as np
from numba import jit

@jit(nopython=True)
def indirect_ok(fn):
    return fn()

@jit(nopython=True)
def works_as_expected():
    a = np.array([1])
    return indirect_ok(a.sum)

print(works_as_expected())

For HPAT:

import numpy as np
import hpat

@hpat.jit
def indirect_ok(fn):
    return fn()

@hpat.jit
def works_as_expected():
    a = np.array([1])
    return indirect_ok(a.sum)

print(works_as_expected())

[ehsantn]

HPAT has an inline pass that inlines other jit functions. With inlining in place, I don't think these example will have higher order function call.

However, this pass was a quick hack for a use case written long time ago and definitely needs to be revisited. For example, it might not set the IR definitions data structures properly.

[quasilyte]

Ah, I see. With inlining disabled, HPAT gives same typing error for bound instancemethod passed as a function argument. Thanks.

[ehsantn]

I looked at these examples some more. Looks like inlining is fine. For the top example, the weird function has a type unification issue, since either a dataframe (from df.head) or 0 is assigned to a variable based on a conditional. This will be caught when we make the hiframes pass a typed pass (the current pass has no way of knowing).

I think the a.sum example triggers a bug in Numba's parfor preprocessing stage where a.sum should be converted to np.sum(a) but looks like the call doesn't have the array argument.