rbc icon indicating copy to clipboard operation
rbc copied to clipboard

Published 20 hours ago • verified publisher icon heavyai

Reimplement numpy functions as python functions

Open guilhermeleobas opened this issue 5 years ago • 5 comments

Why this is needed

Currently, there is no way to use a NumPy function with an omnisci array. Reimplement those functions would allow one to just call np.sum(an_omnisci_array) for instance. Furthermore, some functions/methods might also require the usage of Numba Runtime (NRT) extension which is probably not available in the server-side.

An example

def foo(omnisci_array):
    return np.sum(omnisci_arrray)

List of functions that we must implement

NumPy functions CPU CUDA
np.abs :white_check_mark: :white_check_mark:
np.absolute :white_check_mark: :white_check_mark:
np.add :white_check_mark: :white_check_mark:
np.alen :x: :x:
np.all ! :x: :x:
np.allclose ! :x: :x:
np.alltrue ! :x: :x:
np.amax ! :x: :x:
np.amin ! :x: :x:
np.angle :x: :x:
np.any ! :x: :x:
np.append ! :x: :x:
np.apply_along_axis ? :x: :x:
np.apply_over_axes ? :x: :x:
np.arange ! :x: :x:
np.arccos :white_check_mark: :white_check_mark:
np.arccosh :white_check_mark: :white_check_mark:
np.arcsin :white_check_mark: :white_check_mark:
np.arcsinh :white_check_mark: :white_check_mark:
np.arctan :white_check_mark: :white_check_mark:
np.arctan2 :white_check_mark: :white_check_mark:
np.arctanh :white_check_mark: :white_check_mark:
np.argmax ! :x: :x:
np.argmin ! :x: :x:
np.argpartition :x: :x:
np.argsort :x: :x:
np.argwhere :x: :x:
np.around ! :white_check_mark: :white_check_mark:
np.array ? :x: :x:
np.array_equal ! :x: :x:
np.array_equiv ! :x: :x:
np.array_split ? :x: :x:
np.asanyarray ? :x: :x:
np.asarray ? :x: :x:
np.asarray_chkfinite ? :x: :x:
np.asfarray ? :x: :x:
np.asmatrix ? :x: :x:
np.asscalar ? :x: :x:
np.atleast_1d ? :x: :x:
np.atleast_2d ? :x: :x:
np.atleast_3d ? :x: :x:
np.average ! :x: :x:
np.bartlett !? :x: :x:
np.bincount ! :x: :x:
np.bitwise_and :white_check_mark: :white_check_mark:
np.bitwise_not ! :white_check_mark: :white_check_mark:
np.bitwise_or :white_check_mark: :white_check_mark:
np.bitwise_xor :white_check_mark: :white_check_mark:
np.blackman !? :x: :x:
np.block ? :x: :x:
np.bmat ? :x: :x:
np.broadcast ? :x: :x:
np.broadcast_arrays ? :x: :x:
np.broadcast_to ? :x: :x:
np.busday_count ? :x: :x:
np.busday_offset ? :x: :x:
np.busdaycalendar ? :x: :x:
np.byte_bounds ? :x: :x:
np.c_ ? :x: :x:
np.can_cast ? :x: :x:
np.cbrt ! :x: :x:
np.ceil :white_check_mark: :white_check_mark:
np.chararray ? :x: :x:
np.choose ? :x: :x:
np.clip :x: :x:
np.column_stack ? :x: :x:
np.common_type ? :x: :x:
np.compare_chararrays ? :x: :x:
np.compress ! :x: :x:
np.concatenate ? :x: :x:
np.conj ! :white_check_mark: :white_check_mark:
np.conjugate ! :white_check_mark: :white_check_mark:
np.convolve ! :x: :x:
np.copy ! :x: :x:
np.copysign :white_check_mark: :white_check_mark:
np.copyto ! :x: :x:
np.corrcoef ! :x: :x:
np.correlate ! :x: :x:
np.cos :white_check_mark: :white_check_mark:
np.cosh :white_check_mark: :white_check_mark:
np.count_nonzero ! :x: :x:
np.cov ! :x: :x:
np.cross ! :x: :x:
np.cumprod ! :x: :x:
np.cumproduct ! :x: :x:
np.cumsum - PR #74 :white_check_mark: :white_check_mark:
np.datetime64 ? :x: :x:
np.datetime_data ? :x: :x:
np.deg2rad :white_check_mark: :white_check_mark:
np.degrees :white_check_mark: :white_check_mark:
np.delete ! :x: :x:
np.diag ? :x: :x:
np.diag_indices ? :x: :x:
np.diag_indices_from ? :x: :x:
np.diagflat ? :x: :x:
np.diagonal ? :x: :x:
np.diff ! :x: :x:
np.digitize :x: :x:
np.divide :white_check_mark: :white_check_mark:
np.divmod ? :x: :x:
np.dot ! :x: :x:
np.dsplit ? :x: :x:
np.dstack ? :x: :x:
np.e ! :x: :x:
np.ediff1d ! :x: :x:
np.einsum ? :x: :x:
np.einsum_path ? :x: :x:
np.empty :white_check_mark: :white_check_mark:
np.empty_like :white_check_mark: :white_check_mark:
np.equal :white_check_mark: :white_check_mark:
np.exp :white_check_mark: :white_check_mark:
np.exp2 :white_check_mark: :white_check_mark:
np.expand_dims ? :x: :x:
np.expm1 ! :white_check_mark: :white_check_mark:
np.extract ! :x: :x:
np.eye ? :x: :x:
np.fabs :white_check_mark: :white_check_mark:
np.fastCopyAndTranspose ? :x: :x:
np.fft ! :x: :x:
np.fill_diagonal ? :x: :x:
np.fix ! :x: :x:
np.flip ! :x: :x:
np.fliplr ! :x: :x:
np.flipud ! :x: :x:
np.float_power :x: :x:
np.floor :white_check_mark: :white_check_mark:
np.floor_divide :white_check_mark: :white_check_mark:
np.fmax :white_check_mark: :white_check_mark:
np.fmin :white_check_mark: :white_check_mark:
np.fmod :white_check_mark: :white_check_mark:
np.frexp :x: :x:
np.full :white_check_mark: :white_check_mark:
np.full_like :white_check_mark: :white_check_mark:
np.gcd :white_check_mark: :white_check_mark:
np.geomspace ! :x: :x:
np.gradient ? :x: :x:
np.greater :white_check_mark: :white_check_mark:
np.greater_equal :white_check_mark: :white_check_mark:
np.hamming !? :x: :x:
np.hanning !? :x: :x:
np.heaviside :white_check_mark: :white_check_mark:
np.histogram ! :x: :x:
np.histogram2d !? :x: :x:
np.histogram_bin_edges ? :x: :x:
np.histogramdd ? :x: :x:
np.hsplit ? :x: :x:
np.hstack ? :x: :x:
np.hypot :white_check_mark: :white_check_mark:
np.i0 ! :x: :x:
np.identity ? :x: :x:
np.imag ? :x: :x:
np.in1d !? :x: :x:
np.indices ? :x: :x:
np.inf ! :x: :x:
np.infty !? :x: :x:
np.inner !? :x: :x:
np.insert ! :x: :x:
np.interp ! :x: :x:
np.intersect1d !? :x: :x:
np.invert ! :x: :x:
np.is_busday ? :x: :x:
np.isclose ! :x: :x:
np.isfinite :white_check_mark: :white_check_mark:
np.isfortran ? :x: :x:
np.isin ! :x: :x:
np.isinf :white_check_mark: :white_check_mark:
np.isnan :white_check_mark: :white_check_mark:
np.isnat ? :white_check_mark: :white_check_mark:
np.isneginf ! :x: :x:
np.isposinf ! :x: :x:
np.isreal ? :x: :x:
np.isscalar ? :x: :x:
np.issctype ? :x: :x:
np.ix_ ? :x: :x:
np.kaiser !? :x: :x:
np.kron ? :x: :x:
np.lcm :white_check_mark: :white_check_mark:
np.ldexp ! :white_check_mark: :white_check_mark:
np.left_shift :white_check_mark: :white_check_mark:
np.less :white_check_mark: :white_check_mark:
np.less_equal :white_check_mark: :white_check_mark:
np.lexsort ? :x: :x:
np.linspace ! :x: :x:
np.log :white_check_mark: :white_check_mark:
np.log10 :white_check_mark: :white_check_mark:
np.log1p :white_check_mark: :white_check_mark:
np.log2 :white_check_mark: :white_check_mark:
np.logaddexp :white_check_mark: :white_check_mark:
np.logaddexp2 :white_check_mark: :white_check_mark:
np.logical_and :white_check_mark: :white_check_mark:
np.logical_not :white_check_mark: :white_check_mark:
np.logical_or :white_check_mark: :white_check_mark:
np.logical_xor :white_check_mark: :white_check_mark:
np.logspace ! :x: :x:
np.mask_indices :x: :x:
np.matmul ! :x: :x:
np.max ! :x: :x:
np.maximum :white_check_mark: :white_check_mark:
np.mean ! :x: :x:
np.median ! :x: :x:
np.meshgrid ? :x: :x:
np.mgrid ? :x: :x:
np.min ! :x: :x:
np.minimum :white_check_mark: :white_check_mark:
np.mod :white_check_mark: :white_check_mark:
np.modf :white_check_mark: :white_check_mark:
np.moveaxis ? :x: :x:
np.msort ? :x: :x:
np.multiply :white_check_mark: :white_check_mark:
np.nan ! :x: :x:
np.nan_to_num ? :x: :x:
np.nanargmax ? :x: :x:
np.nanargmin ? :x: :x:
np.nancumprod ? :x: :x:
np.nancumsum ? :x: :x:
np.nanmax ? :x: :x:
np.nanmean ? :x: :x:
np.nanmedian ? :x: :x:
np.nanmin ? :x: :x:
np.nanpercentile ? :x: :x:
np.nanprod ? :x: :x:
np.nanquantile ? :x: :x:
np.nanstd ? :x: :x:
np.nansum ? :x: :x:
np.nanvar ? :x: :x:
np.ndim ! :x: :x:
np.negative :white_check_mark: :white_check_mark:
np.newaxis ? :x: :x:
np.nextafter :white_check_mark: :white_check_mark:
np.nonzero! :x: :x:
np.not_equal :white_check_mark: :white_check_mark:
np.ones :white_check_mark: :white_check_mark:
np.ones_like :white_check_mark: :white_check_mark:
np.outer ! :x: :x:
np.packbits ? :x: :x:
np.pad ! :x: :x:
np.partition ! :x: :x:
np.percentile ! :x: :x:
np.pi ! :x: :x:
np.piecewise ? :x: :x:
np.place ! :x: :x:
np.poly ? :x: :x:
np.positive :white_check_mark: :white_check_mark:
np.power :white_check_mark: :white_check_mark:
np.prod(t) :white_check_mark: :white_check_mark:
np.product :x: :x:
np.ptp ! :x: :x:
np.put ! :x: :x:
np.put_along_axis ! :x: :x:
np.putmask ! :x: :x:
np.quantile ! :x: :x:
np.rad2deg ! :white_check_mark: :white_check_mark:
np.radians :white_check_mark: :white_check_mark:
np.ravel ! :x: :x:
np.ravel_multi_index ! :x: :x:
np.real ? :x: :x:
np.real_if_close ? :x: :x:
np.reciprocal :white_check_mark: :white_check_mark:
np.remainder :white_check_mark: :white_check_mark:
np.repeat ! :x: :x:
np.require ! :x: :x:
np.reshape ! :x: :x:
np.resize ! :x: :x:
np.right_shift :white_check_mark: :white_check_mark:
np.rint :white_check_mark: :white_check_mark:
np.roll ! :x: :x:
np.rollaxis ! :x: :x:
np.roots ! :x: :x:
np.rot90 ! :x: :x:
np.round ! :x: :x:
np.round_ ! :white_check_mark: :white_check_mark:
np.row_stack ! :x: :x:
np.searchsorted ! :x: :x:
np.select ! :x: :x:
np.setdiff1d ! :x: :x:
np.setxor1d ! :x: :x:
np.shape ! :x: :x:
~np.short~ :x: :x:
np.sign :white_check_mark: :white_check_mark:
np.signbit :white_check_mark: :white_check_mark:
np.sin :white_check_mark: :white_check_mark:
np.sinc :x: :x:
np.sinh :white_check_mark: :white_check_mark:
np.size ! :x: :x:
np.sometrue ! :x: :x:
np.sort ! :x: :x:
np.spacing ! :x: :x:
np.split ! :x: :x:
np.sqrt :white_check_mark: :white_check_mark:
np.square :white_check_mark: :white_check_mark:
np.squeeze ! :x: :x:
np.stack ! :x: :x:
np.std ! :x: :x:
np.subtract :white_check_mark: :white_check_mark:
np.sum :white_check_mark: :white_check_mark:
np.swapaxes ! :x: :x:
np.take ! :x: :x:
np.take_along_axis ! :x: :x:
np.tan :white_check_mark: :white_check_mark:
np.tanh :white_check_mark: :white_check_mark:
np.tensordot ! :x: :x:
np.tile ! :x: :x:
np.trace ! :x: :x:
np.transpose ! :x: :x:
np.trapz ! :x: :x:
np.tri ! :x: :x:
np.tril ! :x: :x:
np.tril_indices ! :x: :x:
np.tril_indices_from ! :x: :x:
np.trim_zeros ! :x: :x:
np.triu ! :x: :x:
np.triu_indices ! :x: :x:
np.triu_indices_from ! :x: :x:
np.true_divide :white_check_mark: :white_check_mark:
np.trunc :white_check_mark: :white_check_mark:
np.union1d ! :x: :x:
np.unique ! :x: :x:
np.unpackbits ! :x: :x:
np.unravel_index ! :x: :x:
np.unwrap ! :x: :x:
np.vander ! :x: :x:
np.var ! :x: :x:
np.vdot ! :x: :x:
np.vsplit ! :x: :x:
np.vstack ! :x: :x:
np.where ! :x: :x:
np.zeros :white_check_mark: :white_check_mark:
np.zeros_like :white_check_mark: :white_check_mark:

List of functions that we are not implementing

Please raise an issue if some of the following functions should be implemented.

  • [/] ~np.absolute_import~
  • [/] ~np.add_docstring~
  • [/] ~np.add_newdoc~
  • [/] ~np.add_newdoc_ufunc~
  • [/] ~np.array2string~
  • [/] ~np.array_repr~
  • [/] ~np.array_str~
  • [/] ~np.ascontiguousarray~
  • [/] ~np.asfortranarray~
  • [/] ~np.cfloat~
  • [/] ~np.char~
  • [/] ~np.character~
  • [/] ~np.base_repr~
  • [/] ~np.binary_repr~
  • [/] ~np.bool~
  • [/] ~np.bool8~
  • [/] ~np.bool_~
  • [/] ~np.byte~
  • [/] ~np.bytes0~
  • [/] ~np.bytes_~
  • [/] ~np.cdouble~
  • [/] ~np.clongdouble~
  • [/] ~np.clongfloat~
  • [/] ~np.complex~
  • [/] ~np.complex128~
  • [/] ~np.complex256~
  • [/] ~np.complex64~
  • [/] ~np.complex_~
  • [/] ~np.complexfloating~
  • [/] ~np.core~
  • [/] ~np.datetime_as_string~
  • [/] ~np.deprecate~
  • [/] ~np.deprecate_with_doc~
  • [/] ~np.double~
  • [/] ~np.float~
  • [/] ~np.float128~
  • [/] ~np.float16~
  • [/] ~np.float32~
  • [/] ~np.float64~
  • [/] ~np.float_~
  • [/] ~np.frombuffer~
  • [/] ~np.fromfile~
  • [/] ~np.genfromtxt~
  • [/] ~np.get_include~
  • [/] ~np.get_printoptions~
  • [/] ~np.info~
  • [/] ~np.int~
  • [/] ~np.int0~
  • [/] ~np.int16~
  • [/] ~np.int32~
  • [/] ~np.int64~
  • [/] ~np.int8~
  • [/] ~np.int_~
  • [/] ~np.int_asbuffer~
  • [/] ~np.intc~
  • [/] ~np.integer~
  • [/] ~np.intp~
  • [/] ~np.little_endian~
  • [/] ~np.load~
  • [/] ~np.loads~
  • [/] ~np.loadtxt~
  • [/] ~np.long~
  • [/] ~np.longcomplex~
  • [/] ~np.longdouble~
  • [/] ~np.longfloat~
  • [/] ~np.longlong~
  • [/] ~np.mafromtxt~
  • [/] ~np.matrixlib~
  • [/] ~np.may_share_memory~
  • [/] ~np.memmap~
  • [/] ~np.min_scalar_type~
  • [/] ~np.mintypecode~
  • [/] ~np.ndfromtxt~
  • [/] ~np.print_function~
  • [/] ~np.printoptions~
  • [/] ~np.safe_eval~
  • [/] ~np.save~
  • [/] ~np.savetxt~
  • [/] ~np.sctype2char~
  • [/] ~np.sctypeDict~
  • [/] ~np.sctypeNA~
  • [/] ~np.sctypes~
  • [/] ~np.set_numeric_ops~
  • [/] ~np.set_printoptions~
  • [/] ~np.set_string_function~
  • [/] ~np.setbufsize~
  • [/] ~np.number~
  • [/] ~np.obj2sctype~
  • [/] ~np.object~
  • [/] ~np.object0~
  • [/] ~np.object_~
  • [/] ~np.recfromcsv~
  • [/] ~np.recfromtxt~
  • [/] ~np.seterrcall~
  • [/] ~np.seterrobj~
  • [/] ~np.str~
  • [/] ~np.str0~
  • [/] ~np.str_~
  • [/] ~np.string_~
  • [/] ~np.timedelta64~
  • [/] ~np.typeDict~
  • [/] ~np.typeNA~
  • [/] ~np.typename~
  • [/] ~np.typecodes~
  • [/] ~np.ubyte~
  • [/] ~np.ufunc~
  • [/] ~np.uint~
  • [/] ~np.uint0~
  • [/] ~np.uint16~
  • [/] ~np.uint32~
  • [/] ~np.uint64~
  • [/] ~np.uint8~
  • [/] ~np.uintc~
  • [/] ~np.uintp~
  • [/] ~np.ulonglong~
  • [/] ~np.unicode~
  • [/] ~np.unicode_~
  • [/] ~np.ushort~
  • [/] ~np.void~
  • [/] ~np.void0~
  • [/] ~np.division~
  • [/] ~np.dtype~
  • [/] ~np.emath~
  • [/] ~np.errstate~
  • [/] ~np.euler_gamma~
  • [/] ~np.find_common_type~
  • [/] ~np.finfo~
  • [/] ~np.flatiter~
  • [/] ~np.flatnonzero~
  • [/] ~np.flexible~
  • [/] ~np.floating~
  • [/] ~np.format_float_positional~
  • [/] ~np.format_float_scientific~
  • [/] ~np.format_parser~
  • [/] ~np.fromfunction~
  • [/] ~np.fromiter~
  • [/] ~np.frompyfunc~
  • [/] ~np.fromregex~
  • [/] ~np.fromstring~
  • [/] ~np.generic~
  • [/] ~np.fv~ deprected
  • [/] ~np.get_array_wrap~
  • [/] ~np.getbufsize~
  • [/] ~np.geterr~
  • [/] ~np.geterrcall~
  • [/] ~np.geterrobj~
  • [/] ~np.cast~
  • [/] ~np.compat~
  • [/] ~np.csingle~
  • [/] ~np.ctypeslib~
  • [/] ~np.disp~
  • [/] ~np.half~
  • [/] ~np.index_exp~
  • [/] ~np.iinfo~
  • [/] ~np.inexact~
  • [/] ~np.ipmt~ deprecated
  • [/] ~np.irr~ deprecated
  • [/] ~np.iscomplex~
  • [/] ~np.iscomplexobj~
  • [/] ~np.isrealobj~
  • [/] ~np.issubclass_~
  • [/] ~np.issubdtype~
  • [/] ~np.issubsctype~
  • [/] ~np.iterable~
  • [/] ~np.lib~
  • [/] ~np.linalg~
  • [/] ~np.lookfor~
  • [/] ~np.ma~
  • [/] ~np.mat~
  • [/] ~np.math~
  • [/] ~np.matrix~
  • [/] ~np.maximum_sctype~
  • [/] ~np.mirr~ deprecated
  • [/] ~np.mkl~
  • [/] ~np.nbytes~
  • [/] ~np.ndarray~
  • [/] ~np.ndenumerate~
  • [/] ~np.ndindex~
  • [/] ~np.nditer~
  • [/] ~np.nested_iters~
  • [/] ~np.nper~ deprecated
  • [/] ~np.npv~ deprecated
  • [/] ~np.numarray~
  • [/] ~np.ogrid~
  • [/] ~np.oldnumeric~
  • [/] ~np.pmt~ deprecated
  • [/] ~np.poly1d~
  • [/] ~np.polyadd~
  • [/] ~np.polyder~
  • [/] ~np.polydiv~
  • [/] ~np.polyfit~
  • [/] ~np.polyint~
  • [/] ~np.polymul~
  • [/] ~np.polynomial~
  • [/] ~np.polysub~
  • [/] ~np.polyval~
  • [/] ~np.ppmt~ deprecated
  • [/] ~np.promote_types~
  • [/] ~np.pv~ deprecated
  • [/] ~np.r_~
  • [/] ~np.random~
  • [/] ~np.rate~ deprecated
  • [/] ~np.rec~
  • [/] ~np.recarray~
  • [/] ~np.record~
  • [/] ~np.result_type~
  • [/] ~np.s_~
  • [/] ~np.savez~
  • [/] ~np.savez_compressed~
  • [/] ~np.seterr~
  • [/] ~np.shares_memory~
  • [/] ~np.show_config~
  • [/] ~np.signedinteger~
  • [/] ~np.single~
  • [/] ~np.singlecomplex~
  • [/] ~np.sort_complex~
  • [/] ~np.source~
  • [/] ~np.sys~
  • [/] ~np.test~
  • [/] ~np.testing~
  • [/] ~np.tracemalloc_domain~
  • [/] ~np.unsignedinteger~
  • [/] ~np.vectorize~
  • [/] ~np.version~
  • [/] ~np.warnings~
  • [/] ~np.who~

guilhermeleobas avatar Feb 13 '20 17:02 guilhermeleobas

I'm probably missing some context here, so not sure if this makes sense, but: have you considered adding __array_function__ and __array_ufunc__ to the omnisci array object? Or is that what you mean to do here, so np.sum dispatches to your new sum function that does support omnisci arrays?

rgommers avatar Feb 13 '20 18:02 rgommers

Or is that what you mean to do here, so np.sum dispatches to your new sum function that does support omnisci arrays?

That's what I mean. Provide an implementation for sum/np.sum for omnisci array. For instance, using the sum example:

@numba.extending.overload(np.sum)
@numba.extending.overload(sum)
def omnisci_sum(a):
    def impl(a):
        s = 0 
        n = len(a)
        for i in range(n):
            s += a[i]
        return s
    return impl

@omni('i32(i32[])')
def bar(t):
  return np.sum(t)

omni.register()

_, result = omni.sql_execute('select t, bar(t) from math') # works

guilhermeleobas avatar Feb 13 '20 20:02 guilhermeleobas

@rgommers , yes, the context here is a bit different from the context where __array_function__ could be used.

All functions that rbc decorates, are never executed. Instead, the rbc decorated functions are used as templates for generating LLVM IR module strings that are sent to the omniscidb server. Omniscidb server does not know anything about Python nor numpy arrays. This means that the LLVM IR module must be self-contained in the sense that the functions in the module can use only LLVM intrinsic functions or functions and data structures that the omniscidb server defines. One of the omniscidb data structures is Array that represents a 1-d array that basically holds only the data pointer value and information about the size and type of the array.

When omniscidb Array is used inside the rbc decorated function as x, currently, only the following operations are implemented:

len(x)        # returns the size of Array as integer
v = x[i]      # set `v` to the value of the `i`-th Array item
x[i] = v      # set the value of the `i`-th Array item to `v`

Using these basic operations, one can easily implement all other operators such as sum, etc.

Now, when using np.sum inside a rbc decorated functions, it expects numba view of the numpy array as the argument to np.sum. To use omniscidb Array as the argument to np.sum, there seems to be two options:

  1. Create a temporary numpy array object, copy the values of Array to the numpy object, and use the temporary array object as argument to np.sum:
@omnisci(..)
def foo(x):
    # x is omnisci Array
    # compute np.sum(x) as follows
    y = np.empty(len(x), dtype=...)
    for i in range(len(x)):
        y[i] = x[i]
    s = np.sum(y)
    ...
  1. Provide Python implementations of numpy methods:
@omnisci(..)
def foo(x):
    # compute np.sum(x) as follows
    s = 0
    for i in range(len(x)):
        s += x[i]

My cut feeling is that option 1. is suboptimal because of it requires an unnecessary copy of the data and one needs to implement a new data type that is a omniscidb view of the numpy array. Option 2. would involve replacing the real numpy namespace with an rbc namespace that implements numpy API functions as pure python functions that can be rbc decorated and therefore used from UDF definitions.

pearu avatar Feb 13 '20 20:02 pearu

Thanks both, I get the picture now. Interesting stuff!

rgommers avatar Feb 13 '20 22:02 rgommers

~@pearu and @rgommers, with respect to the first option. Wouldn't it be possible to convert the inner structure of an OmniSci array to a NumPy array? I do not know how a NumPy array is implemented under the hood but I think we can achieve something similar to what PyTorch does with torch.from_numpy() and torch.tensor.numpy().~

~This way, if we convert the underlying data structure to a NumPy array, OmniSci will benefit from functions already implemented by the Numba folks.~

guilhermeleobas avatar Feb 14 '20 01:02 guilhermeleobas