DimensionalData.jl
DimensionalData.jl copied to clipboard
Published
20 hours ago •
rafaqz
rafaqz
error showing DimArray
i'm getting a "ERROR: ArgumentError: Chunk sizes must be strictly positive" error, yet the chunk sizes seem to be positive.
julia> using DimensionalData, YAXArrays, YAXArrayBase, Zarr, DiskArrays
julia> A = rand(X(6), Y(5), Z(4), metadata = Dict{String, Any}())
╭───────────────────────────╮
│ 6×5×4 DimArray{Float64,3} │
├───────────────────── dims ┤
↓ X, → Y, ↗ Z
├───────────────── metadata ┤
Dict{String, Any}()
└───────────────────────────┘
[:, :, 1]
0.784275 0.125566 0.397801 0.807615 0.393778
0.255679 0.32185 0.919381 0.932538 0.974425
0.997114 0.577818 0.808486 0.0384444 0.56319
0.589664 0.661931 0.38077 0.037413 0.0355789
0.440041 0.0455827 0.601652 0.165645 0.619262
0.810021 0.187474 0.585673 0.509805 0.624613
julia> B = yaxconvert(YAXArray, A)
╭───────────────────────────╮
│ 6×5×4 YAXArray{Float64,3} │
├───────────────────── dims ┤
↓ X, → Y, ↗ Z
├───────────────── metadata ┤
Dict{String, Any}()
├──────────────── file size ┤
file size: 960.0 bytes
└───────────────────────────┘
julia> C = setchunks(B, (2,5,4))
╭───────────────────────────╮
│ 6×5×4 YAXArray{Float64,3} │
├───────────────────── dims ┤
↓ X, → Y, ↗ Z
├───────────────── metadata ┤
Dict{String, Any}()
├──────────────── file size ┤
file size: 960.0 bytes
└───────────────────────────┘
julia> savecube(C, "foo.zarr", driver=:zarr)
╭───────────────────────────╮
│ 6×5×4 YAXArray{Float64,3} │
├───────────────────── dims ┤
↓ X, → Y, ↗ Z
├───────────────── metadata ┤
Dict{String, Any}()
├──────────────── file size ┤
file size: 960.0 bytes
└───────────────────────────┘
julia> D = Cube("foo.zarr")
╭───────────────────────────╮
│ 6×5×4 YAXArray{Float64,3} │
├───────────────────────────┴─────────────────────── dims ┐
↓ X Sampled{Int64} 1:1:6 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:1:5 ForwardOrdered Regular Points,
↗ Z Sampled{Int64} 1:1:4 ForwardOrdered Regular Points
├─────────────────────────────────────────────── metadata ┤
Dict{String, Any} with 2 entries:
"name" => "layer"
"_FillValue" => 1.0e32
├────────────────────────────────────────────── file size ┤
file size: 960.0 bytes
└─────────────────────────────────────────────────────────┘
julia> E = DimArray(D.data, dims(D))
╭───────────────────────────╮
│ 6×5×4 DimArray{Float64,3} │
├───────────────────────────┴─────────────────────── dims ┐
↓ X Sampled{Int64} 1:1:6 ForwardOrdered Regular Points,
→ Y Sampled{Int64} 1:1:5 ForwardOrdered Regular Points,
↗ Z Sampled{Int64} 1:1:4 ForwardOrdered Regular Points
└─────────────────────────────────────────────────────────┘
[:, :, 1]
Error showing value of type DimArray{Float64, 3, Tuple{X{DimensionalData.Dimensions.Lookups.Sampled{Int64, StepRange{Int64, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Int64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, Y{DimensionalData.Dimensions.Lookups.Sampled{Int64, StepRange{Int64, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Int64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}, Z{DimensionalData.Dimensions.Lookups.Sampled{Int64, StepRange{Int64, Int64}, DimensionalData.Dimensions.Lookups.ForwardOrdered, DimensionalData.Dimensions.Lookups.Regular{Int64}, DimensionalData.Dimensions.Lookups.Points, DimensionalData.Dimensions.Lookups.NoMetadata}}}, Tuple{}, ZArray{Float64, 3, Zarr.BloscCompressor, DirectoryStore}, DimensionalData.NoName, DimensionalData.Dimensions.Lookups.NoMetadata}:
ERROR: ArgumentError: Chunk sizes must be strictly positive
Stacktrace:
[1] RegularChunks
@ ~/.julia/packages/DiskArrays/bZBJE/src/chunks.jl:26 [inlined]
[2] subsetchunks(r::DiskArrays.RegularChunks, subs::UnitRange{Int64})
@ DiskArrays ~/.julia/packages/DiskArrays/bZBJE/src/chunks.jl:50
[3] #87
@ ./none:0 [inlined]
[4] iterate
@ ./generator.jl:47 [inlined]
[5] grow_to!(dest::Vector{DiskArrays.ChunkType}, itr::Base.Generator{Base.Iterators.Filter{…}, DiskArrays.var"#87#89"{…}})
@ Base ./array.jl:907
[6] collect
@ ./array.jl:831 [inlined]
[7] eachchunk_view
@ ~/.julia/packages/DiskArrays/bZBJE/src/subarray.jl:32 [inlined]
[8] eachchunk
@ ~/.julia/packages/DiskArrays/bZBJE/src/subarray.jl:27 [inlined]
[9] (::DiskArrays.var"#66#71")(ar::DiskArrays.SubDiskArray{Float64, 2, ZArray{…}, Tuple{…}, false})
@ DiskArrays ~/.julia/packages/DiskArrays/bZBJE/src/broadcast.jl:56
[10] _all(f::DiskArrays.var"#66#71", itr::Vector{AbstractMatrix{Float64}}, ::Colon)
@ Base ./reduce.jl:1288
[11] all(f::Function, a::Vector{AbstractMatrix{Float64}}; dims::Function)
@ Base ./reducedim.jl:1023
[12] common_chunks(::Tuple{Int64, Int64}, ::SubArray{Float64, 2, Matrix{…}, Tuple{…}, false}, ::Vararg{Any})
@ DiskArrays ~/.julia/packages/DiskArrays/bZBJE/src/broadcast.jl:56
[13] copyto!(dest::SubArray{…}, bc::Base.Broadcast.Broadcasted{…})
@ DiskArrays ~/.julia/packages/DiskArrays/bZBJE/src/broadcast.jl:37
[14] materialize!
@ ./broadcast.jl:914 [inlined]
[15] materialize!
@ ./broadcast.jl:911 [inlined]
[16] _copyto!(dest::Matrix{…}, Rdest::CartesianIndices{…}, src::DiskArrays.SubDiskArray{…}, Rsrc::CartesianIndices{…})
@ DiskArrays ~/.julia/packages/DiskArrays/bZBJE/src/array.jl:64
[17] copyto!
@ ~/.julia/packages/DiskArrays/bZBJE/src/array.jl:17 [inlined]
[18] _print_matrix(io::IOContext{…}, A::DiskArrays.SubDiskArray{…}, lookups::Tuple{…})
@ DimensionalData ~/.julia/packages/DimensionalData/vXseP/src/array/show.jl:289
[19] print_matrix
@ ~/.julia/packages/DimensionalData/vXseP/src/array/show.jl:240 [inlined]
[20] print_array(io::IOContext{…}, mime::MIME{…}, A::DimArray{…})
@ DimensionalData ~/.julia/packages/DimensionalData/vXseP/src/array/show.jl:208
[21] show_after(io::IOContext{…}, mime::MIME{…}, A::DimArray{…})
@ DimensionalData ~/.julia/packages/DimensionalData/vXseP/src/array/show.jl:76
[22] show(io::IOContext{…}, mime::MIME{…}, A::DimArray{…})
@ DimensionalData ~/.julia/packages/DimensionalData/vXseP/src/array/show.jl:17
[23] (::REPL.var"#55#56"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:273
[24] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:569
[25] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:259
[26] display
@ ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:278 [inlined]
[27] display(x::Any)
@ Base.Multimedia ./multimedia.jl:340
[28] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{…})
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:0
[29] (::REPL.var"#57#58"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:284
[30] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:569
[31] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:282
[32] (::REPL.var"#do_respond#80"{…})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:911
[33] (::REPL.var"#98#108"{…})(::REPL.LineEdit.MIState, ::Any, ::Vararg{…})
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1248
[34] #invokelatest#2
@ ./essentials.jl:892 [inlined]
[35] invokelatest
@ ./essentials.jl:889 [inlined]
[36] (::VimBindings.var"#8#10"{REPL.var"#98#108"{…}, String})(s::REPL.LineEdit.MIState, p::REPL.LineEditREPL)
@ VimBindings ~/.julia/dev/VimBindings/src/lineeditalt.jl:69
[37] prompt!(term::REPL.Terminals.TTYTerminal, prompt::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ VimBindings ~/.julia/dev/VimBindings/src/lineeditalt.jl:34
[38] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ REPL.LineEdit ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/LineEdit.jl:2651
[39] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1312
[40] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
@ REPL ~/.julia/juliaup/julia-1.10.2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/REPL/src/REPL.jl:386
Some type information was truncated. Use `show(err)` to see complete types.
julia> DiskArrays.eachchunk(E.data)
3×1×1 DiskArrays.GridChunks{3, Tuple{DiskArrays.RegularChunks, DiskArrays.RegularChunks, DiskArrays.RegularChunks}}:
[:, :, 1] =
(1:2, 1:5, 1:4)
(3:4, 1:5, 1:4)
(5:6, 1:5, 1:4)
julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin22.4.0)
CPU: 12 × Apple M2 Max
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
Environment:
JULIA_PROJECT = @.
JULIA_EDITOR = vi
(jl_a3hv6U) pkg> st
Status `/private/var/folders/s5/8d629n5d7nsf37f60_91wzr40000gq/T/jl_a3hv6U/Project.toml`
[0703355e] DimensionalData v0.26.3
⌅ [3c3547ce] DiskArrays v0.3.23
[90b8fcef] YAXArrayBase v0.6.1
[c21b50f5] YAXArrays v0.5.4
[0a941bbe] Zarr v0.9.2
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`
(jl_a3hv6U) pkg> st --outdated
Status `/private/var/folders/s5/8d629n5d7nsf37f60_91wzr40000gq/T/jl_a3hv6U/Project.toml`
⌅ [3c3547ce] DiskArrays v0.3.23 (<v0.4.1): DiskArrayTools, Zarr
DD doesnt know anything about chunks...
If you cant reproduce a bug with a basic DimArray or other objects defined here, take the issue to the extending package first, in this case YAXArrays.jl.
The problem is in the internal object, not DD
I guess the only way will be to bring the data into memory first:
E = DimArray(D.data[:,:,:], dims(D))
This is not a YAXArray issue, because YAXArrays is only used to construct the data and to open the underlying Zarr data. The object that is failing is a normal DimArray with a DiskArray as data.
This seems to rather be an issue with calling copyto! on a DiskArray but I am wondering why this is not a problem on NetCDF or GDAL files. This call happens in the print_matrix function of DD.
It might be that YAXArrays is constructing a slightly broken zarr file here.
Do you also run into this problem on actual data that you get from somewhere else or is it only a problem when you save the data with YAXArrays?
Either DiskArrays.jl not working with some base method that works on normal arrays, or YAX constructing something broken. But I assumed it was YAX making a broken disk array.
I looked into it a bit more and it seems to be an issue DiskArrays not handling the copyto! case when one of the indices is length zero. This happens when the size of the terminal is too small to show something but it still tries to retrieve some data. I will open a DiskArrays.jl issue.