DiskArrays.jl
DiskArrays.jl copied to clipboard
meggart
Use TiledIteration.jl?
For GridChunks, we could use TiledIteration.jl instead of implementing everything here. In general, I wonder if TiledIteration.jl couldn't become the interface module for all tiled array iteration (if it isn't already).
I have an implementation of iteration using TiledIteration.jl in HDF5Arrays.jl, so I could make a PR, if there's interest.
If we limit ourselves to tiles (GridChunks in DiskArrays.jl), we can also use tiled indexing to make a lot of the code implementing broadcasting, reduce operations, etc. cleaner. Further, the code could be moved to TiledIteration.jl, so non-DiskArrays can benefit from the tools here, too. See this proposal for tiled indexing
Yes, I would be open for that. Does the TileIterator support offsets? This would be an important functionality I would not want to lose to be able to efficiently loop over views of tiled arrays. If yes, feel free to replace the GridChunks object with a TileIterator, I am happy about every piece of code that doesn't need to be double-maintained.
I will read a bit more on your proposal for Tiled indexing and maybe comment later, but the idea sounds reasonable.
Another field of tension would be the question if we support non-regular chunks. Currently the DiskArray interface is written in a way that it would support this in theory. However, in practice this is only tested with regular chunks and I know one or 2 places where code breaks when eachchunk does not return a GridChunks object.
TileIterator splits a tuple of unitranges into even(-ish) tiles. If you have a global offset for the whole array, that would just be represented in the axes. If, instead, you want the tiling to start at some offset away from (1, 1, 1, ...), so that the upper left corner of the first full tile starts at something like (3, 4, 5, ...), another tiling 'strategy' would have to be implemented. Currently, only RelaxLastTile and RelaxStride exist, but it shouldn't be too difficult to implement something like OffsetTiles as a strategy.
As far as non-regular tiling goes: theoretically, any tiling could be mapped to some N-dimensional indices indexing the tile and some M-dimensional indices indexing into each tile. We could have something like the following:
struct TileIndex{N, TN}
tile::NTuple{N, Int}
index::NTuple{TN, Int}
end
abstract type TileStyle end # basically the equivalent of strategy in TiledIteration.jl
struct OffsetGridTiles <: TileStyle end # grid with offset
struct GridTiles <: TileStyle end # grid with no offset
struct RelaxedGridTiles <: TileStyle end # grid with relaxed grid size, so tiles will fit
# This trait can be used for arrays with IndexStyle = IndexTiled to determine the kind of tiling.
struct IndexTiled <: IndexStyle end # trait to dispatch on for all manner of operations
We would have to reroute a few functions in base julia to dispatch on index style for all of this to work on any array type. In the meantime, it's possible to subtype AbstractArray with some AbstractTiledArray, and only reroute base julia functions for that array type. though. This is an issue I raise in the TiledIteration.jl issue, too.
DiskArrays.jl now actually handles way more of the edge cases than TiledIteration.jl does - like now Zip and Generator collect in the right order. Closing this.
