YAXArrays.jl

YAXArrays.jl copied to clipboard

I would like to exclude one variable from a Dataset

e.g.

using Pkg
cd("/Net/Groups/BGI/work_3/OEMC/oemc_towers/")
Pkg.activate("/Net/Groups/BGI/work_3/OEMC/oemc_towers/")

using YAXArrays, CairoMakie, YAXArraysToolbox, Zarr, Statistics,  DiskArrays, DiskArrayTools, Revise, DataFrames, CSV, Plots, ProgressMeter, DimensionalData


YAXArrays.YAXdir("/Net/Groups/BGI/scratch/dpabon/YAXA_tmp")

cubeloc = "/Net/Groups/BGI/scratch/zmhamdi/sen2cube/"

sen2cube = open_dataset(cubeloc)

site_name = "ES-LMa"


#site_cube = sen2cube[site = At(site_name), Ti = Date("2019-01-01") .. Date("2019-01-30")]


site_cube = sen2cube[site = At(site_name)]


site_cube[[ !in "Sentinel2__L2A__uuid"]]

How can I achieve this?

I think, that in the Future we should be using the Not Selector which is going to be implemented in DimensionalData in this PR: https://github.com/rafaqz/DimensionalData.jl/pull/521

So this should be

site_cube[Not(At("Sentinel2__L2A__uuid"))]

Until we get this through DImensionalData you could use something like this:

using EarthDataLab
ds = esdd()
Dataset(;(k=>ds.cubes[k] for k in keys(ds.cubes) if k!=:gross_primary_productivity)...)

If anyone wants to review it I can merge it ASAP, otherwise in a few days https://github.com/rafaqz/DimensionalData.jl/pull/521

Another trick for now is to use the Where selector.

site_cube[site=Where(!=("Sentinel2__L2A__uuid"))]

Another trick for now is to use the Where selector.

site_cube[site=Where(!=("Sentinel2__L2A__uuid"))]

Thanks, I tried this one, when I printed the results it appears correct but when I do

site_2 = site_cube[site=Where(!=("Sentinel2__L2A__uuid"))]

Cube(site_2)

I got

ERROR: ArgumentError: Could not promote element types of cubes in dataset to a common concrete type, because of Variable Sentinel2__L2A__uuid
Stacktrace:

Cube(ds::Dataset; joinname::String, target_type::Nothing) at [/Net/Groups/BGI/people/dpabon/bin/julia_packages/packages/YAXArrays/lvvMa/src/DatasetAPI/Datasets.jl](vscode-file://vscode-app/usr/share/code/resources/app/out/vs/code/electron-sandbox/workbench/workbench.html)

Cube(ds::Dataset) at [/Net/Groups/BGI/people/dpabon/bin/julia_packages/packages/YAXArrays/lvvMa/src/DatasetAPI/Datasets.jl](vscode-file://vscode-app/usr/share/code/resources/app/out/vs/code/electron-sandbox/workbench/workbench.html)

top-level scope at [/Net/Groups/BGI/work_3/OEMC/oemc_towers/bin/ES-LMa_developing_all_bands.jl](vscode-file://vscode-app/usr/share/code/resources/app/out/vs/code/electron-sandbox/workbench/workbench.html)

Until we get this through DImensionalData you could use something like this:

using EarthDataLab
ds = esdd()
Dataset(;(k=>ds.cubes[k] for k in keys(ds.cubes) if k!=:gross_primary_productivity)...)

Thanks, Fabian, this work around works

test = Dataset(;(k=>site_cube.cubes[k] for k in keys(site_cube.cubes) if k!=:Sentinel2__L2A__uuid)...)

Cube(test)

@dpabon

site_2 = site_cube[site=Where(!=("Sentinel2__L2A__uuid"))]
Cube(site_2)

if there is an error here, it will probably be on the YAXArrays side, since the type is a little bit different when using the selectors. Maybe we will need to extend to those.

Underneath Where is returning a Vector{Bool} index. If YAXArrays doesnt suppprt that then no, it wont work ;)

The Not selector should work, but would need https://github.com/meggart/DiskArrays.jl/pull/181 to work for Diskbased arrays.