Update to Diskarrays 0.4

[meggart]

This is a placeholder PR to update the DiskArray interface implementation once DiskArrays 0.4 is released.

[Alexander-Barth]

Thanks a lot Fabian for your help!

[rafaqz]

Can we re-run CI and see if this passes now?

[Alexander-Barth]

There was a failure when accessing a variable out-of-bounds. Previously test was just checking for an NetCDFError now it is also checking for a BoundsError. The test code is updated here:

https://github.com/Alexander-Barth/NCDatasets.jl/pull/247/commits/332c8461961764236b1ed9f6596d3f9c27a2a394

Currently, we have a failing test for an array of strings and strided writes

using Test
using NCDatasets

sz = (4,5)
filename = tempname()

ds = NCDataset(filename,"c")
# define the dimension "lon" and "lat"
ds.dim["lon"] = sz[1]
ds.dim["lat"] = sz[2]

T = String
data = [Char(i+64) * Char(j+64) for i = 1:sz[1], j = 1:sz[2]]
v = defVar(ds,"var-$T",T,("lon","lat"))
v[:,:] = data[:,:] # ok

# stridded write and read
v[1:2:end,1:2:end] = data[1:2:end,1:2:end]
#    @test all(v[1:2:end,1:2:end] .== data[1:2:end,1:2:end])
close(ds)

We do not have a failure for non-strided writes. The data given to NCDatasets.writeblock! seems to contain #undef

("writeblock!", data) = ("writeblock!", ["AA" #undef "AC" #undef "AE"; #undef #undef #undef #undef #undef; "CA" #undef "CC" #undef "CE"])

Do you have any ideas?

[rafaqz]

@meggart ?

[asinghvi17]

Bump, this would be pretty useful to have nowadays since I'm looking at Zarr + raster ecosystem improvements

[Alexander-Barth]

I am currently hitting this error with DiskArrays 0.4.4 and NCDatasets 2d0e7cb:

  Test threw exception
  Expression: isempty(ncv[Int[]])
  MethodError: reducing over an empty collection is not allowed; consider supplying `init` to the reducer
  Stacktrace:
    [1] reduce_empty(op::Base.MappingRF{Base.ExtremaMap{typeof(identity)}, typeof(Base._extrema_rf)}, ::Type{Int64})           
      @ Base ./reduce.jl:361
    [2] reduce_empty_iter
      @ ./reduce.jl:384 [inlined]
    [3] mapreduce_empty_iter(f::Function, op::Function, itr::Vector{Int64}, ItrEltype::Base.HasEltype)                                                                         
      @ Base ./reduce.jl:380
    [4] _mapreduce
      @ ./reduce.jl:432 [inlined]
    [5] _mapreduce_dim
      @ ./reducedim.jl:365 [inlined]
    [6] mapreduce
      @ ./reducedim.jl:357 [inlined]
    [7] _extrema
      @ ./reducedim.jl:1015 [inlined]
    [8] _extrema
      @ ./reducedim.jl:1014 [inlined]
    [9] extrema
      @ ./reducedim.jl:1010 [inlined]
   [10] span(v::Vector{Int64})
      @ DiskArrays ~/.julia/packages/DiskArrays/6JA8Z/src/batchgetindex.jl:70
   [11] #is_sparse_index#13
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/batchgetindex.jl:84 [inlined]
   [12] is_sparse_index
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/batchgetindex.jl:83 [inlined]
   [13] need_batch_index
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/diskarray.jl:72 [inlined]
   [14] _need_batch
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/diskarray.jl:57 [inlined]
   [15] need_batch
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/diskarray.jl:54 [inlined]
   [16] getindex_disk!(out::Nothing, a::NCDatasets.Variable{Int64, 1, NCDataset{Nothing, Missing}}, i::Vector{Int64})                              
      @ DiskArrays ~/.julia/packages/DiskArrays/6JA8Z/src/diskarray.jl:249
   [17] getindex_disk
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/diskarray.jl:210 [inlined]
   [18] getindex
      @ ~/.julia/packages/DiskArrays/6JA8Z/src/diskarray.jl:348 [inlined]
   [19] getindex(v::CommonDataModel.CFVariable{Int64, 1, NCDatasets.Variable{Int64, 1, NCDataset{Nothing, Missing}}, CommonDataModel.Attributes{NCDatasets.Variable{Int64, 1, NCDataset{Nothing, Missing}}}, @NamedTuple{fillvalue::Nothing, missing_values::Tuple{}, scale_factor::Nothing, add_offset::Nothing, calendar::Nothing, time_origin::Nothing, time_factor::Nothing, maskingvalue::Missing}}, indexes::Vector{Int64})
      @ CommonDataModel ~/.julia/packages/CommonDataModel/G3moc/src/cfvariable.jl:445
   [20] macro expansion
      @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/
Test.jl:669 [inlined]                                                                  
   [21] top-level scope
      @ ~/.julia/dev/NCDatasets/test/test_variable.jl:508
ERROR: LoadError: There was an error during testing
in expression starting at /home/abarth/.julia/dev/NCDatasets/test/test_variable.jl:275

Reported also here: https://github.com/meggart/DiskArrays.jl/issues/186