stdlib-js
[RFC]: achieve ndarray API parity with built-in JavaScript arrays
Description
This RFC proposes achieving ndarray API parity with built-in JavaScript arrays. Built-in JavaScript Array and TypedArray objects have a number of methods for searching, manipulating, sorting, and transforming array data.
The goal of this RFC is to add functional APIs providing equivalent functionality for ndarrays. By providing these APIs, stdlib can offer a powerful toolset using a similar vocabulary and interface design as existing art for working with ndarrays. This should help reduce the barrier to ndarray adoption and encourage their more widespread use.
Note, however, that ndarrays have considerable additional complexity due to their multi-dimensional nature, and, in particular, element-wise iteration requires specialized kernels for handling non-contiguous underlying data.
There does exist precedent in stdlib for such kernels (e.g., ndarray/base/assign, ndarray/base/nullary, and ndarray/base/unary). Those packages also provide C APIs which may or may not be relevant to the functional APIs proposed in this RFC.
What follows is an initial list of Array.prototype.* methods and notes regarding whether an equivalent already exists or what constraints we need to consider when designing ndarray equivalent packages.
Top-level: ndarray/*
-
[x]
Array.prototype.atndarray/at
-
[x]
Array.prototype.entriesndarray/iter/entries
-
[x]
Array.prototype.everyndarray/every
-
[x]
Array.prototype.forEachndarray/for-each
-
[x]
Array.prototype.mapndarray/map
-
[x]
Array.prototype.filterndarray/filter
-
[x]
Array.prototype.slicendarray/slice
-
[x]
Array.prototype.keysndarray/iter/indices
-
[x]
Array.prototype.valuesndarray/iter/values
-
[x]
Array.prototype.includesndarray/includes
-
[x]
Array.prototype.indexOfblas/ext/index-of
-
[x]
Array.prototype.findndarray/base/findndarray/find
-
[x]
Array.prototype.findIndexblas/ext/find-index
-
[x]
Array.prototype.lastIndexOfblas/ext/last-index-of
-
[x]
Array.prototype.somendarray/any
-
[ ]
Array.prototype.findLast(WIP)- reduction; specify one or more axes
-
[x]
Array.prototype.findLastIndexblas/ext/find-last-index
-
[x]
Array.prototype.reversendarray/reverse
-
[x]
Array.prototype.toReversedndarray/base/to-reversedndarray/reversed
-
[x]
Array.prototype.sort- multiple functions for different sorting algorithms.
blas/ext/sorthp.
-
[x]
Array.prototype.toSorted- multiple functions for different sorting algorithms.
blas/ext/to-sortedhp
-
[x]
Array.prototype.fillndarray/fillandndarray/fill-byndarray/fill-slice
-
[x]
Array.prototype.flatndarray/flatten
-
[ ]
Array.prototype.reduce- reduction; specify one or more axes
-
[x]
Array.prototype.concatndarray/concat
-
[ ]
Array.prototype.copyWithin- specify axis
- how would this work for strided arrays?
- for ideal case, would prefer delegating to array prototype method, as likely faster; for non-ideal case, would need to take special care to avoid overwriting accessed values.
- Slice + assign?
- Temporary buffer for when there is overlap.
-
[ ]
Array.prototype.reduceRight- reduction; specify one or more axes
- iterate in reverse direction
-
[x]
Array.prototype.flatMapndarray/flatten-by
-
[ ]
Array.prototype.join- needs R&D
- how would dimension separators work? a different separator for each dimension? configurable?
-
[x]
Array.prototype.popndarray/pop
-
[x]
Array.prototype.pushndarray/concat1d
-
[x]
Array.prototype.shiftndarray/shift
-
[ ]
Array.prototype.splice- needs R&D
- would involving slicing, concatenation, and data copy
-
[ ]
Array.prototype.toLocaleString- needs R&D
-
[ ]
Array.prototype.toSpliced- needs R&D
-
[ ]
Array.prototype.toString- just call
ndarray.toString()?
- just call
-
[ ]
Array.prototype.unshift- this would essentially be concat along an axis
- support for broadcasting?
- would require a data copy, no mutation
-
[x]
Array.prototype.withndarray/with
Base: ndarray/base/*
-
[x]
Array.prototype.fillndarray/base/fill
-
[x]
Array.prototype.forEachndarray/base/for-each
-
[x]
Array.prototype.mapndarray/base/map
-
[x]
Array.prototype.reversendarray/base/reverse
-
[x]
Array.prototype.slicendarray/base/slice
-
[x]
Array.prototype.toReversedndarray/base/to-reversed
-
[x]
Array.prototype.everyndarray/base/every-by
-
[x]
Array.prototype.somendarray/base/any
-
[x]
Array.prototype.findndarray/base/find
-
[x]
Array.prototype.popndarray/base/pop
-
[x]
Array.prototype.shiftndarray/base/shift
Related Issues
None.
Questions
No.
Other
No.
Checklist
- [X] I have read and understood the Code of Conduct.
- [X] Searched for existing issues and pull requests.
- [X] The issue name begins with
RFC:.
I want to work on this issue @kgryte and could you elaborate on the specific goals of aligning ndarray APIs with JavaScript arrays? How will this benefit users and developers?
How will this benefit users and developers?
This is already addressed in the OP. Second paragraph.
@SarthakPaandey Before you begin, it would be best to communicate what you're planning to work on. Some of the above routines are easier than others and should be addressed first. Reductions require R&D and should only be worked on once we've determined the best approach.
"Understood, @kgryte. I'll prioritize the routines that are comparatively easier and communicate my plan before starting. I'll defer work on reductions until we've established the best approach. Thanks for the guidance!
@kgryte I have started to work on the map method. I plan to submit PR for it when I've implemented the kernels for it up to 3d. This way, it would be possible to get feedback from you earlier and it would be easier for me to work on the feedback. Does that seem reasonable?
@headlessNode That makes sense. I am currently working on for-each, which is related.
@headlessNode I suggest working a "base" implementation of map first. For @stdlib/ndarray/base/map, we can assume that an output array is provided, similar to ndarray/base/unary. In fact, unary is a good reference for map. The main difference being the support of a thisArg and no C implementation.
@kgryte The thisArg in our case would be similar to how it is handled in the Array.prototype.map right? e.g.
//Array.prototype.map
let numbers = [1, 2, 3];
let obj = { multiplier: 2 };
numbers.map(function(num) {
return num * this.multiplier;
}, obj);```
@headlessNode Yes. As an example, see https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/array/base/any-by.
@headlessNode Pushed up a POC implementation of for-each: https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/ndarray/base/for-each
Hey it looks like i joined the conversation late but is there some pending work which i can contribute on?
@headlessNode Do you currently have anything in progress?
@DhruvArvindSingh If this is your first time contributing to stdlib, you may want to first knock out a few "Good First Issues" in order to get acquainted with the project. The ndarray packages are typically pretty involved.
@kgryte yeah, working towards adding the top-level implementations. Should be ready for initial review in a few days.
@kgryte, Are there any tasks or functionalities that still need to be implemented? Do you have any suggestions on how I can get started?
@gururaj1512 If you are interested in working on this, I suggest the two lowest hanging fruit are fill and with, as they can wrap base/assign, which is already implemented. We can skip adding the native add-on for now.
@gururaj1512 And I suggest prioritizing with first, as fill will require optional slice support (e.g., fill( x, value[, MultiSlice])).
I see that base/fill has been added (ref: https://github.com/stdlib-js/stdlib/blob/develop/lib/node_modules/%40stdlib/ndarray/base/fill/lib/main.js). However, we may want to revisit that package in order to add slice support.
IIRC, we may want separate packages for filling and assigning only selected axis of the input ndarray.
@headlessNode Yeah, I don't recall the context or outcome of those prior discussions. 😔
@kgryte We'd have slice-fill(value, start, end) where start = index to start filling from and end = index to end filling at. Achieving which would, I think, require setting-up the kernels.
So, I think, a base implementation of slice-fill and then in top-level fill implementation we could offer more control to the end-user?
@headlessNode Yeah, I suppose we could add fill-slice; however, start and end don't make sense, IMO. Instead, we'd just require a user provide a MultiSlice slice argument, which would define the range of values to update. In terms of kernels, we already have base/slice-assign which should fulfill this purpose, as it wraps base/assign.
Hey @kgryte, I am really interested on working on this issue, I have read the whole conversation here and I see that this issue is broken into chunks and tackled and assigned to the interested ones. May I know where can I get started related to this.
@ShabiShett07 This is already being worked on: https://github.com/stdlib-js/stdlib/pull/4356. And also, see the discussion at https://github.com/stdlib-js/stdlib/pull/4398#issuecomment-2565875736. Namely, before we work on reductions, we need to determine how to write the reduction kernels supporting an arbitrary subset of axes.