YAXArrays.jl yaxconvert not defined

yaxconvert is mentioned in the docs, and three issues (#169, #194, #229), but i don't see a definition anywhere in the code in the entirety of the git history (pkg.dev, git log -p, then search). what am i missing?

i have a DimensionalData array, and would like to convert it to a YAXArray so that i can save it to a chunked file and then memory map just the parts i need. that's what yaxconvert is documented to do IIUC. is there another way?

thanks!

Jul 31 '23 22:07 bjarthur

You need to load YAXArrayBase and we should clarify this in the docs. The long term plan is to make the conversion unnecessary at least for DimensioalArrays. You could also try to use the savecube function on the DimArray directly without conversion. Since version 0.5 YAXArrays uses DimensionalData internally.

Jul 31 '23 22:07 felixcremer

doh. in fact, needing YAXArrayBase is documented on the very page i linked. i just missed it. sorry, and thanks.

okay, so following along those docs from the top of the page, i get an error when showing the output of the conversion:

julia> dim = yaxconvert(DimArray, ds)
30×10×15×2 DimArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
Error showing value of type DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}}:
ERROR: Lookups overlap or are not in order at 2022-01-15 and 2022-01-15

Stacktrace

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] _lookup_index_cat_error(lookup::DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}, xl::Date)
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:365
  [3] (::DimensionalData.var"#127#129"{DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}})(lookup::DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:348
  [4] #59
    @ ./tuple.jl:602 [inlined]
  [5] BottomRF
    @ ./reduce.jl:81 [inlined]
  [6] afoldl
    @ ./operators.jl:535 [inlined]
  [7] _foldl_impl
    @ ./tuple.jl:329 [inlined]
  [8] foldl_impl
    @ ./reduce.jl:48 [inlined]
  [9] mapfoldl_impl
    @ ./reduce.jl:44 [inlined]
 [10] #mapfoldl#288
    @ ./reduce.jl:170 [inlined]
 [11] mapfoldl
    @ ./reduce.jl:170 [inlined]
 [12] #foldl#289
    @ ./reduce.jl:193 [inlined]
 [13] foldl
    @ ./reduce.jl:193 [inlined]
 [14] foreach
    @ ./tuple.jl:602 [inlined]
 [15] _vcat_index
    @ /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:344 [inlined]
 [16] _vcat_lookups
    @ /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:299 [inlined]
 [17] vcat
    @ /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:289 [inlined]
 [18] _horvcat(f::Base.Splat{typeof(vcat)}, As::Tuple{DimArray{Float64, 2, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Matrix{Float64}, DimensionalData.NoName, Dict{String, Any}}, DimArray{Float64, 2, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Matrix{Float64}, DimensionalData.NoName, Dict{String, Any}}}, #unused#::Val{1})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:281
 [19] vcat
    @ /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/methods.jl:272 [inlined]
 [20] _print_matrix(io::IOContext{Base.TTY}, A::DimArray{Float64, 2, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, SubArray{Float64, 2, Array{Float64, 4}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64, Int64}, true}, DimensionalData.NoName, Dict{String, Any}}, lookups::Tuple{DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/show.jl:111
 [21] print_array(io::IOContext{Base.TTY}, mime::MIME{Symbol("text/plain")}, A::DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/show.jl:61
 [22] show_after
    @ /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/show.jl:35 [inlined]
 [23] show(io::IOContext{Base.TTY}, mime::MIME{Symbol("text/plain")}, A::DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/4TpBG/src/array/show.jl:27
 [24] (::REPL.var"#55#56"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:276
 [25] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [26] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:262
 [27] display
    @ /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:281 [inlined]
 [28] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:340
 [29] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [30] invokelatest
    @ ./essentials.jl:813 [inlined]
 [31] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:305
 [32] (::REPL.var"#57#58"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:287
 [33] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [34] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:285
 [35] (::REPL.var"#do_respond#80"{Bool, Bool, REPL.var"#93#103"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:899
 [36] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [37] invokelatest
    @ ./essentials.jl:813 [inlined]
 [38] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2647
 [39] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
 [40] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514

Aug 01 '23 11:08 bjarthur

and though yaxconverting a subset of a DimArray back to YAXArray as demonstrated in the YAXArray docs works, converting a DimArray that was created as such, as in the docs for DimensionalData, does not:

julia> using DimensionalData

julia> A = rand(X(50), Y(10.0:40.0))
50×31 DimArray{Float64,2} with dimensions: 
  X,
  Y Sampled{Float64} 10.0:1.0:40.0 ForwardOrdered Regular Points
 10.0       11.0        12.0        13.0         …  38.0         39.0        40.0
  0.551876   0.0671004   0.961211    0.329316        0.00824898   0.638931    0.408412
  0.555526   0.542853    0.9268      0.853312        0.761852     0.081645    0.154222
  0.6243     0.50352     0.0356045   0.00962141      0.0674333    0.500321    0.655994
  0.719308   0.0484075   0.851234    0.0311157       0.957866     0.881494    0.452655
  0.171072   0.345615    0.236108    0.502687    …   0.611146     0.783291    0.281872
  0.809951   0.387474    0.394327    0.772617        0.274723     0.792827    0.421075
  0.518043   0.684303    0.0131663   0.752819        0.725358     0.566616    0.243544
  0.580212   0.898939    0.0667302   0.47279         0.533915     0.54374     0.467786
  0.578125   0.78626     0.982852    0.942122        0.955552     0.661442    0.961473
  0.864014   0.881012    0.144593    0.734155    …   0.59873      0.225211    0.505407
  0.507311   0.297512    0.486179    0.664663        0.0782181    0.305352    0.00574907
  0.122751   0.398858    0.856911    0.869954        0.340969     0.663311    0.236891
  0.614106   0.429924    0.385895    0.615486        0.729074     0.363152    0.679494
  0.827123   0.81563     0.908682    0.373007        0.449758     0.155345    0.144505
  ⋮                                              ⋱                            ⋮
  0.317299   0.425185    0.216379    0.887328        0.925633     0.413373    0.109993
  0.771927   0.890942    0.273482    0.398071        0.788144     0.189702    0.782103
  0.194516   0.0876607   0.37452     0.0966418       0.720949     0.305293    0.423926
  0.404393   0.980967    0.87512     0.960275    …   0.789058     0.686704    0.265909
  0.677634   0.707456    0.544522    0.820837        0.246203     0.828991    0.302738
  0.187551   0.672093    0.695891    0.0722544       0.94657      0.433121    0.956936
  0.45732    0.228346    0.347599    0.566268        0.343101     0.602927    0.345002
  0.241566   0.443836    0.909689    0.54991         0.422589     0.0469856   0.269347
  0.938177   0.879758    0.86941     0.956951    …   0.556482     0.221584    0.625994
  0.644881   0.971069    0.0750275   0.720524        0.0288233    0.738464    0.2819
  0.359579   0.989729    0.577934    0.0628705       0.731525     0.366281    0.389653
  0.201807   0.18214     0.563899    0.689889        0.910816     0.334008    0.350888
  0.756964   0.736997    0.383178    0.504977        0.486544     0.460765    0.660528
  0.722019   0.851155    0.177384    0.333402    …   0.59969      0.850551    0.776403

julia> using YAXArrays, YAXArrayBase

julia> yaxconvert(YAXArray, A)
ERROR: MethodError: Cannot `convert` an object of type 
  DimensionalData.Dimensions.LookupArrays.NoMetadata to an object of type 
  Dict{String}

Closest candidates are:
  convert(::Type{T}, ::T) where T<:AbstractDict
   @ Base abstractdict.jl:565
  convert(::Type{T}, ::AbstractDict) where T<:AbstractDict
   @ Base abstractdict.jl:567
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:64

Stacktrace

Stacktrace:
 [1] YAXArray(axes::Tuple{Dim{:X, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:Y, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, data::Matrix{Float64}, properties::DimensionalData.Dimensions.LookupArrays.NoMetadata, chunks::DiskArrays.GridChunks{2}, cleaner::Vector{YAXArrays.Cubes.CleanMe})
   @ YAXArrays.Cubes /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/Cubes/Cubes.jl:123
 [2] YAXArray(axes::Tuple{Dim{:X, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:Y, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, data::Matrix{Float64}, properties::DimensionalData.Dimensions.LookupArrays.NoMetadata; cleaner::Vector{YAXArrays.Cubes.CleanMe}, chunks::DiskArrays.GridChunks{2})
   @ YAXArrays.Cubes /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/Cubes/Cubes.jl:136
 [3] YAXArray(axes::Tuple{Dim{:X, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:Y, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, data::Matrix{Float64}, properties::DimensionalData.Dimensions.LookupArrays.NoMetadata)
   @ YAXArrays.Cubes /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/Cubes/Cubes.jl:136
 [4] yaxcreate(#unused#::Type{YAXArray}, data::Matrix{Float64}, dimnames::Tuple{Symbol, Symbol}, dimvals::Vector{DimensionalData.Dimensions.LookupArrays.Aligned}, atts::DimensionalData.Dimensions.LookupArrays.NoMetadata)
   @ YAXArrays.Cubes /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/Cubes/Cubes.jl:371
 [5] yaxconvert(T::Type, x::DimArray{Float64, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{Float64}, DimensionalData.NoName, DimensionalData.Dimensions.LookupArrays.NoMetadata})
   @ YAXArrayBase /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/axisarrays/axisinterface.jl:81
 [6] top-level scope
   @ REPL[4]:1

this is with julia 1.9.2 and the master branches of YAXArrays, YAXArrayBase, and DimensionalData. on a mac.

Aug 01 '23 12:08 bjarthur

i also tried savecube on a DimArray as suggested and it does not work:

julia> using DimensionalData

julia> A = rand(X(50), Y(10.0:40.0))
50×31 DimArray{Float64,2} with dimensions: 
  X,
  Y Sampled{Float64} 10.0:1.0:40.0 ForwardOrdered Regular Points
 10.0       11.0         12.0        13.0         …  38.0        39.0         40.0
  0.881287   0.629509     0.322425    0.192821        0.735262    0.452089     0.929576
  0.71166    0.730386     0.195872    0.545969        0.837671    0.559783     0.746776
  0.308719   0.821133     0.23253     0.00672459      0.310229    0.339925     0.981813
  0.132109   0.159522     0.835997    0.13891         0.342058    0.658513     0.218166
  0.617631   0.555759     0.630413    0.101652    …   0.644106    0.00982902   0.581067
  0.222884   0.00914366   0.770757    0.704762        0.0601903   0.667565     0.236102
  0.330236   0.6275       0.961671    0.343005        0.0853408   0.660941     0.695706
  0.393381   0.461041     0.455061    0.983439        0.347273    0.934276     0.656817
  0.321412   0.623917     0.816849    0.679605        0.177762    0.207565     0.806719
  0.748949   0.952613     0.951004    0.977449    …   0.680954    0.843969     0.517942
  0.470153   0.894079     0.114897    0.672928        0.13251     0.911878     0.938442
  0.878412   0.359869     0.543454    0.933525        0.170847    0.262136     0.610139
  0.624271   0.409461     0.395193    0.42783         0.733056    0.259033     0.520604
  0.889335   0.703164     0.341896    0.867595        0.73462     0.489358     0.249999
  ⋮                                               ⋱                            ⋮
  0.548636   0.674991     0.618104    0.0306931       0.576078    0.921138     0.444401
  0.559885   0.979166     0.617086    0.664534        0.0955654   0.655609     0.158219
  0.804452   0.0814024    0.560151    0.425425        0.148086    0.999522     0.413779
  0.266441   0.382251     0.405739    0.857045    …   0.110575    0.169862     0.669286
  0.669948   0.0458019    0.275357    0.305383        0.446119    0.847235     0.307079
  0.768782   0.652949     0.179838    0.789262        0.75623     0.00367788   0.662351
  0.810297   0.254601     0.330307    0.918308        0.783549    0.985174     0.802206
  0.447934   0.850845     0.0587327   0.332171        0.161717    0.149704     0.125113
  0.444282   0.771794     0.217707    0.761087    …   0.50669     0.457434     0.777414
  0.482268   0.399347     0.463964    0.886698        0.408144    0.966391     0.265343
  0.301592   0.357324     0.296951    0.902823        0.84001     0.0614831    0.901588
  0.414188   0.699354     0.913981    0.690822        0.271221    0.071632     0.381269
  0.548247   0.713637     0.0305305   0.144523        0.58663     0.814366     0.268899
  0.161776   0.247457     0.984563    0.424615    …   0.955446    0.639996     0.109052

julia> using YAXArrays, YAXArrayBase, Zarr

julia> savecube(A, "foo.zarr", driver=:zarr)
ERROR: type DimArray has no field properties
Stacktrace:
 [1] getproperty
   @ ./Base.jl:37 [inlined]
 [2] top-level scope
   @ REPL[5]:1

works fine for me on a YAXArray though:

julia> a = YAXArray(rand(10,20))
10×20 YAXArray{Float64,2} with dimensions: 
  Dim{:Dim_1} Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
  Dim{:Dim_2} Sampled{Int64} Base.OneTo(20) ForwardOrdered Regular Points
Total size: 1.56 KB

julia> savecube(a, "foo.zarr", driver=:zarr)
10×20 YAXArray{Float64,2} with dimensions: 
  Dim{:Dim_1} Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
  Dim{:Dim_2} Sampled{Int64} Base.OneTo(20) ForwardOrdered Regular Points
Total size: 1.56 KB

Aug 01 '23 12:08 bjarthur

I cannot reproduce the show error, this seems to be rather a DimensionalData error. Could you please check, whether also constructing the DimArray directly from the data gives you the error?

julia> ddim = DimArray(data, axlist)

I can reproduce the yaxconvert(YAXArray, dim) error. This happens, because the yaxconvert function does not properly handle the NoMetadata indicator from DimensionalData. You can circumvent the problem by providing an empty dictionary as metadata with the correct type, but this is something that we would need to fix in the future.

julia> A = rand(X(50), Y(10.0:40.0), metadata=Dict{String, Any}())

Aug 01 '23 15:08 felixcremer

I misremembered that with savecube of a DimArray. I thought this would work now after we changed the internals to use DimensionalData but apparently this is not the case. I fixed it in #172 but we closed it in favour of switching to DimensionalData internally. Do you think it would be helpful to be able to use savecube on a DimArray?

Aug 01 '23 15:08 felixcremer

re. the show error, here is the full transcript of my julia session which demonstrates it, again copied straight from the docs:

julia> using YAXArrays

julia> using DimensionalData: DimensionalData as DD

julia> using DimensionalData

julia> a = YAXArray(rand(10, 20, 5))
10×20×5 YAXArray{Float64,3} with dimensions: 
  Dim{:Dim_1} Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
  Dim{:Dim_2} Sampled{Int64} Base.OneTo(20) ForwardOrdered Regular Points,
  Dim{:Dim_3} Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points
Total size: 7.81 KB

julia> using Dates

julia> axlist = (
           Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
           Dim{:lon}(range(1, 10, length=10)),
           Dim{:lat}(range(1, 5, length=15)),
           Dim{:Variable}(["var1", "var2"])
           )
Dim{:time} Date("2022-01-01"):Day(1):Date("2022-01-30"),
Dim{:lon} 1.0:1.0:10.0,
Dim{:lat} 1.0:0.2857142857142857:5.0,
Dim{:Variable} String["var1", "var2"]

julia> data = rand(30, 10, 15, 2);

julia> ds = YAXArray(axlist, data)
30×10×15×2 YAXArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
Total size: 70.31 KB

julia> using DimensionalData, YAXArrayBase

julia> dim = yaxconvert(DimArray, ds)
30×10×15×2 DimArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
Error showing value of type DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}}:
ERROR: Lookups overlap or are not in order at 2022-01-15 and 2022-01-15

Stacktrace

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] _lookup_index_cat_error(lookup::DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}, xl::Date)
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:365
  [3] (::DimensionalData.var"#127#129"{DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}})(lookup::DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:348
  [4] #59
    @ ./tuple.jl:602 [inlined]
  [5] BottomRF
    @ ./reduce.jl:81 [inlined]
  [6] afoldl
    @ ./operators.jl:535 [inlined]
  [7] _foldl_impl
    @ ./tuple.jl:329 [inlined]
  [8] foldl_impl
    @ ./reduce.jl:48 [inlined]
  [9] mapfoldl_impl
    @ ./reduce.jl:44 [inlined]
 [10] #mapfoldl#288
    @ ./reduce.jl:170 [inlined]
 [11] mapfoldl
    @ ./reduce.jl:170 [inlined]
 [12] #foldl#289
    @ ./reduce.jl:193 [inlined]
 [13] foldl
    @ ./reduce.jl:193 [inlined]
 [14] foreach
    @ ./tuple.jl:602 [inlined]
 [15] _vcat_index
    @ /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:344 [inlined]
 [16] _vcat_lookups
    @ /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:299 [inlined]
 [17] vcat
    @ /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:289 [inlined]
 [18] _horvcat(f::Base.Splat{typeof(vcat)}, As::Tuple{DimArray{Float64, 2, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Matrix{Float64}, DimensionalData.NoName, Dict{String, Any}}, DimArray{Float64, 2, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Matrix{Float64}, DimensionalData.NoName, Dict{String, Any}}}, #unused#::Val{1})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:281
 [19] vcat
    @ /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/methods.jl:272 [inlined]
 [20] _print_matrix(io::IOContext{Base.TTY}, A::DimArray{Float64, 2, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, SubArray{Float64, 2, Array{Float64, 4}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64, Int64}, true}, DimensionalData.NoName, Dict{String, Any}}, lookups::Tuple{DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/show.jl:111
 [21] print_array(io::IOContext{Base.TTY}, mime::MIME{Symbol("text/plain")}, A::DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/show.jl:61
 [22] show_after
    @ /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/show.jl:35 [inlined]
 [23] show(io::IOContext{Base.TTY}, mime::MIME{Symbol("text/plain")}, A::DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}})
    @ DimensionalData /groups/scicompsoft/home/arthurb/.julia/dev/DimensionalData/src/array/show.jl:27
 [24] (::REPL.var"#55#56"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:276
 [25] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [26] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:262
 [27] display
    @ /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:281 [inlined]
 [28] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:340
 [29] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [30] invokelatest
    @ ./essentials.jl:813 [inlined]
 [31] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:305
 [32] (::REPL.var"#57#58"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:287
 [33] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [34] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:285
 [35] (::REPL.var"#do_respond#80"{Bool, Bool, REPL.var"#93#103"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:899
 [36] (::REPL.var"#98#108"{Regex, Regex, Int64, Int64, REPL.LineEdit.Prompt, REPL.LineEdit.Prompt, REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Any, ::Vararg{Any})
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1236
 [37] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [38] invokelatest
    @ ./essentials.jl:813 [inlined]
 [39] (::VimBindings.var"#1#3"{REPL.var"#98#108"{Regex, Regex, Int64, Int64, REPL.LineEdit.Prompt, REPL.LineEdit.Prompt, REPL.LineEdit.Prompt}, String})(s::REPL.LineEdit.MIState, p::REPL.LineEditREPL)
    @ VimBindings /groups/scicompsoft/home/arthurb/.julia/packages/VimBindings/hvwCq/src/lineeditalt.jl:69
 [40] prompt!(term::REPL.Terminals.TTYTerminal, prompt::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ VimBindings /groups/scicompsoft/home/arthurb/.julia/packages/VimBindings/hvwCq/src/lineeditalt.jl:34
 [41] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2642
 [42] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
 [43] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514

Aug 01 '23 15:08 bjarthur

Do you think it would be helpful to be able to use savecube on a DimArray?

not if i can convert to a YAXArray and savecube on that. but i still can't get that to work. thanks for the metadata tip above. that helps. but now the problem seems to be that the elements in one of my Dims are symbols:

julia> using DimensionalData

julia> DimensionalData.@dim MyDim1

julia> A = rand(MyDim1([:x, :y, :z]), metadata = Dict{String, Any}())
3-element DimArray{Float64,1} with dimensions: 
  MyDim1 Categorical{Symbol} Symbol[:x, :y, :z] ForwardOrdered
 :x  0.17538
 :y  0.148049
 :z  0.11512

julia> using YAXArrays, YAXArrayBase

julia> AY = yaxconvert(YAXArray, A)
3-element YAXArray{Float64,1} with dimensions: 
  Dim{:MyDim1} Categorical{Symbol} Symbol[:x, :y, :z] ForwardOrdered
Total size: 24.0 bytes

julia> A[1]
0.17538010986958785

julia> AY[1]
0.17538010986958785

julia> using NetCDF

julia> savecube(AY, "AY.netcdf", driver=:netcdf)
ERROR: ArgumentError: array must be non-empty

Stacktrace

Stacktrace:
  [1] popfirst!
    @ ./array.jl:1463 [inlined]
  [2] jl2nc(t::DataType)
    @ NetCDF /groups/scicompsoft/home/arthurb/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:92
  [3] getNCType
    @ /groups/scicompsoft/home/arthurb/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:102 [inlined]
  [4] NcVar(name::String, dimin::Vector{NcDim}; atts::Dict{String, Any}, t::DataType, compress::Int64, chunksize::Tuple{Int64})
    @ NetCDF /groups/scicompsoft/home/arthurb/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:257
  [5] nccreate(::String, ::String, ::String, ::Vararg{Any}; atts::Dict{String, Any}, gatts::Dict{Any, Any}, compress::Int64, t::DataType, mode::UInt16, chunksize::Tuple{Int64})
    @ NetCDF /groups/scicompsoft/home/arthurb/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:1211
  [6] nccreate
    @ /groups/scicompsoft/home/arthurb/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:1195 [inlined]
  [7] #add_var#90
    @ /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/datasets/netcdf.jl:63 [inlined]
  [8] add_var
    @ /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/datasets/netcdf.jl:60 [inlined]
  [9] #add_var#1
    @ /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:48 [inlined]
 [10] add_var
    @ /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:47 [inlined]
 [11] create_dataset(T::Type, path::String, gatts::Dict{String, Any}, dimnames::Vector{String}, dimvals::Vector{DimensionalData.Dimensions.LookupArrays.Categorical{Symbol, Vector{Symbol}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, dimattrs::Vector{Dict{String, Any}}, vartypes::Vector{DataType}, varnames::Vector{String}, vardims::Vector{Tuple{String}}, varattrs::Vector{Dict{String, Any}}, varchunks::Vector{Tuple{Int64}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ YAXArrayBase /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:57
 [12] create_dataset(T::Type, path::String, gatts::Dict{String, Any}, dimnames::Vector{String}, dimvals::Vector{DimensionalData.Dimensions.LookupArrays.Categorical{Symbol, Vector{Symbol}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, dimattrs::Vector{Dict{String, Any}}, vartypes::Vector{DataType}, varnames::Vector{String}, vardims::Vector{Tuple{String}}, varattrs::Vector{Dict{String, Any}}, varchunks::Vector{Tuple{Int64}})
    @ YAXArrayBase /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:53
 [13] savedataset(ds::Dataset; path::String, persist::Nothing, overwrite::Bool, append::Bool, skeleton::Bool, backend::Symbol, driver::Symbol, max_cache::Float64, writefac::Float64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ YAXArrays.Datasets /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/DatasetAPI/Datasets.jl:571
 [14] savedataset
    @ /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/DatasetAPI/Datasets.jl:521 [inlined]
 [15] savecube(c::YAXArray{Float64, 1, Vector{Float64}, Tuple{Dim{:MyDim1, DimensionalData.Dimensions.LookupArrays.Categorical{Symbol, Vector{Symbol}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}}, path::String; layername::String, datasetaxis::String, max_cache::Float64, backend::Symbol, driver::Symbol, chunks::Nothing, overwrite::Bool, append::Bool, skeleton::Bool, writefac::Float64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ YAXArrays.Datasets /groups/scicompsoft/home/arthurb/.julia/dev/YAXArrays/src/DatasetAPI/Datasets.jl:637
 [16] top-level scope
    @ REPL[7]:1

no problem if the elements are numbers:

julia> B = rand(MyDim1([1, 2, 3]), metadata = Dict{String, Any}())
3-element DimArray{Float64,1} with dimensions: 
  MyDim1 Sampled{Int64} Int64[1, 2, 3] ForwardOrdered Irregular Points
 1  0.340417
 2  0.0840058
 3  0.646981

julia> BY = yaxconvert(YAXArray, B)
3-element YAXArray{Float64,1} with dimensions: 
  Dim{:MyDim1} Sampled{Int64} Int64[1, 2, 3] ForwardOrdered Irregular Points
Total size: 24.0 bytes


julia> savecube(BY, "BY.netcdf", driver=:netcdf)
3-element YAXArray{Float64,1} with dimensions: 
  Dim{:MyDim1} Sampled{Int64} Int64[1, 2, 3] ForwardOrdered Irregular Points
Total size: 24.0 bytes

Aug 01 '23 17:08 bjarthur

I copied your show example into the REPL in a new environment with the latest YAXArrays, YAXArrayBase, DimensionalData versions and it works for me and it looks like this:

Details of shown array

julia> dim = yaxconvert(DimArray, ds)
30×10×15×2 DimArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
                      1.0        2.0        …  7.0        8.0         9.0        10.0
  2022-01-01  0.25224    0.57937       0.506676   0.00119183  0.455278    0.207376
  2022-01-02  0.0557954  0.956016      0.351715   0.839776    0.863212    0.00934017
  2022-01-03  0.730516   0.111143      0.217946   0.683535    0.662452    0.702881
  2022-01-04  0.831242   0.555042      0.0806665  0.55273     0.759573    0.802987
  2022-01-05  0.106859   0.402748   …  0.478955   0.148713    0.939527    0.254214
  2022-01-06  0.611891   0.435965      0.408148   0.163618    0.462386    0.266909
  2022-01-07  0.778541   0.124514      0.660918   0.82015     0.0510813   0.133885
  2022-01-08  0.0864876  0.590058      0.676214   0.593179    0.829276    0.504266
  2022-01-09  0.611419   0.708731      0.085667   0.544805    0.512177    0.533554
  2022-01-10  0.268984   0.641603   …  0.263332   0.387504    0.0745554   0.0355572
  2022-01-11  0.135782   0.808536      0.159677   0.652452    0.427886    0.685817
  2022-01-12  0.213845   0.115216      0.158461   0.8521      0.719588    0.762179
  2022-01-13  0.308991   0.809956      0.127138   0.742123    0.438436    0.170422
  2022-01-14  0.729415   0.16056       0.708445   0.728159    0.21577     0.738906
  2022-01-15  0.769786   0.321205   …  0.100973   0.278482    0.23814     0.14518
  2022-01-16  0.0493651  0.0557759     0.0976277  0.653383    0.0107009   0.848799
  2022-01-17  0.73619    0.364057      0.758723   0.797984    0.651297    0.0627265
  2022-01-18  0.77829    0.342004      0.236923   0.803619    0.323583    0.569309
  2022-01-19  0.896427   0.734217      0.96433    0.146138    0.733856    0.295015
  2022-01-20  0.147742   0.617593   …  0.235058   0.19522     0.720362    0.444467
  2022-01-21  0.508874   0.168179      0.118952   0.504789    0.679678    0.50959
  2022-01-22  0.315509   0.524227      0.527727   0.294725    0.75189     0.204255
  2022-01-23  0.277914   0.533151      0.215969   0.278824    0.0990664   0.192069
  2022-01-24  0.935064   0.595783      0.46333    0.0741327   0.131488    0.265558
  2022-01-25  0.326445   0.910082   …  0.309823   0.946742    0.102515    0.0188348
  2022-01-26  0.0198457  0.434458      0.866245   0.671358    0.954846    0.706557
  2022-01-27  0.0086935  0.437067      0.23929    0.915465    0.872483    0.252025
  2022-01-28  0.646609   0.0948915     0.157426   0.172049    0.229253    0.165325
  2022-01-29  0.783618   0.0249393     0.667314   0.6306      0.100676    0.842124
  2022-01-30  0.27857    0.240208   …  0.816217   0.987482    0.772402    0.190181
[and 29 more slices...]

The main difference in the show method between DimensionalData and YAXArrays is, that YAXArrays doesn't try to show the elements of the array because we mainly deal with larger than memory data and it is quite costly to even show a few values at the edges.

Could you please check whether using only DimensionalData gives you the same error? This might be a minimal example for that:

julia> using DimensionalData

julia> axlist = (
           Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
           Dim{:lon}(range(1, 10, length=10)),
           Dim{:lat}(range(1, 5, length=15)),
           Dim{:Variable}(["var1", "var2"])
           )
Dim{:time} Date("2022-01-01"):Day(1):Date("2022-01-30"),
Dim{:lon} 1.0:1.0:10.0,
Dim{:lat} 1.0:0.2857142857142857:5.0,
Dim{:Variable} String["var1", "var2"]

julia> data = rand(30, 10, 15, 2);
julia> ddim = DimArray(data, axlist)

Aug 01 '23 19:08 felixcremer

Regarding the saving of a YAXArray with Symbol entries in the Dimension. It seems to be not possible to save the Symbols directly into netcdf, because also Rasters runs into similar issues. As a quick fix you can convert the symbols into strings and then save it.

Aug 01 '23 19:08 felixcremer

We could convert the symbols to strings in the background to hide this step from the user, but then we might have to save somewhere in the metadata that this dimension is supposed to be symbols because otherwise we would get a different YAXArray by saving to disk and reopening the dataset and this might be confusing.

Aug 01 '23 19:08 felixcremer

mysteriously, i can not repeat the show error now. even in same environment. perhaps i upgraded it and don't remember. anyway, thanks!

Aug 02 '23 13:08 bjarthur

YAXArrays.jl YAXArrays.jl copied to clipboard

yaxconvert not defined

YAXArrays.jl
YAXArrays.jl copied to clipboard