DiskArrays.jl
DiskArrays.jl copied to clipboard
meggart
Implement PermutedDimsArray for DiskArrays
Hi, I noticed bad performance when using PermutedDimsArray with ArchGDAL and the issue probably is located here. See https://github.com/yeesian/ArchGDAL.jl/issues/225, which explains the problem. However I don't see an easy fix like proposed over there, as it runs into a stack overflow for AbstractDiskArrays.
As explained in the issue you linked, it would be preferable to call permutedims on a DiskArray, which returns a PermuteDiskArray which has the desired behavior.
In case you insist on working with PermutedDimsArray from Base directly, you could frist define const AltPermuteDiskArray = PermuteDimsArray{<:Any, <:Any, <:Any, <:Any, <:AbstractDiskArray} and then you define readblock!, writeblock! and eachchunk as in permute_reshape.jl and implement the remaining DiskArray interface through implement_diskarray AltPermuteDiskArray.
The main drawback of this approach (using Base types for permutations, views, etc instead of making DiskArray analogies) is that the resulting arrays still are not AbstractDiskArrays, so dispatch will not work on them when you recursively use them as wrappers, for example when you create a view into the Permuted DiskArray.
I have a use-case for this too. We need a dimension permutation method that is always lazy - permutedims is only lazy for disk arrays now, but PermuteDimsArray would be lazy for both.
This causes problems generalising some methods in Rasters.jl because we really want lazy permutation of large in-memory rasters, but cant without special-casing everything.
