Failed to load some recent asdf files of TiledDataset

[yjzhu-solar]

Description

dkist.load_dataset failed to load some recent TiledDataset.

Steps to Reproduce

WMIXVA is a recently created (2025-04-01) dataset. However, I cannot load the dataset . I suspect this is possibly related to some changes in asdf files because I can still load the dataset AWGMV in the same observation program, but created earlier (2024-12-20).

dataset_WMIXVA = dkist.load_dataset("/cluster/home/zhuyin/work/pid_2_126/WMIXVA/VBI_L1_20241015T192625_WMIXVA_metadata.asdf")

error message:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[34], line 1
----> 1 dataset_WMIXVA = dkist.load_dataset("/cluster/home/zhuyin/work/pid_2_126/WMIXVA/VBI_L1_20241015T192625_WMIXVA_metadata.asdf")

File /cluster/software/stacks/2024-05/python-cuda/3.11.6/lib/python3.11/functools.py:909, in singledispatch..wrapper(*args, **kw)
    905 if not args:
    906     raise TypeError(f'{funcname} requires at least '
    907                     '1 positional argument')
--> 909 return dispatch(args[0].__class__)(*args, **kw)

File [~/sunpy_env/lib/python3.11/site-packages/dkist/dataset/loader.py:125](https://vscode-remote+jupyter-002eeuler-002ehpc-002eethz-002ech.vscode-resource.vscode-cdn.net/cluster/home/zhuyin/Solar/DKIST_pid_2_126/ipynb/vbi_mosaic/~/sunpy_env/lib/python3.11/site-packages/dkist/dataset/loader.py:125), in _load_from_string(path)
    121 """
    122 A string representing a directory or an ASDF file.
    123 """
    124 # TODO Adjust this to accept URLs as well
--> 125 return _load_from_path(Path(path))

File [~/sunpy_env/lib/python3.11/site-packages/dkist/dataset/loader.py:137](https://vscode-remote+jupyter-002eeuler-002ehpc-002eethz-002ech.vscode-resource.vscode-cdn.net/cluster/home/zhuyin/Solar/DKIST_pid_2_126/ipynb/vbi_mosaic/~/sunpy_env/lib/python3.11/site-packages/dkist/dataset/loader.py:137), in _load_from_path(path)
    135     if not path.exists():
    136         raise ValueError(f"{path} does not exist.")
--> 137     return _load_from_asdf(path)
    139 return _load_from_directory(path)

File [~/sunpy_env/lib/python3.11/site-packages/dkist/dataset/loader.py:246](https://vscode-remote+jupyter-002eeuler-002ehpc-002eethz-002ech.vscode-resource.vscode-cdn.net/cluster/home/zhuyin/Solar/DKIST_pid_2_126/ipynb/vbi_mosaic/~/sunpy_env/lib/python3.11/site-packages/dkist/dataset/loader.py:246), in _load_from_asdf(filepath)
    244 if isinstance(ds, TiledDataset):
    245     for sub in ds.flat:
--> 246         sub.files.basepath = base_path
    247 else:
    248     ds.files.basepath = base_path

AttributeError: 'NoneType' object has no attribute 'files'

System Details

• DKIST Version: 1.13.0
• SunPy Version: 6.1.1
• Astropy Version: 7.1.0
• Python Version: 3.11.6
• OS information: Ubuntu 22.04.5 LTS

[Cadair]

Hi @yjzhu-solar thanks for the bug report. This VBI dataset looks like one of the newer VBI ones where it's not a 3x3 grid of tiles, but some of the tiles are missing (it's really a cross of tiles). It looks like there's a bug in the way the file has been created, the mask attribute of the TiledDataset is all False even though there is missing tiles.

[yjzhu-solar]

Hi @Cadair thanks for the quick response! I’m not sure if this is directly related to missing tiles. Since the observation was taken with VBI Red, which has a slightly larger center FOV (69 arcsec compared to 45 arcsec for VBI Blue), a 2×2 tile should be sufficient to cover the 120 arcsec full field FOV. Besides, I can reconstruct the mosaic using the first four files by loading them as FITS files with astropy.wcs.

[Cadair]

My understanding of this dataset is that there's 5 tile positions in the mosaic with the 5th one being at the center of the FOV overlapping wholly the other 4. We made the decision to represent that as a 3x3 grid with 4 missing tiles, as in the TiledDataset / ASDF representation we don't have any way of representing the overlap.

[Cadair]

I've tracked this bug down and made a patch for it: https://bitbucket.org/dkistdc/dkist-inventory/pull-requests/72 it will take a few days to get the tests written and it deployed and the ASDFs regenerated. At some point soon though, new ASDFs will exist with the bug fixed.

[yjzhu-solar]

Perfect. Thank you so much!

[Cadair]

Hi both, sorry for the delay but the VBI ASDFs have bow been rebuilt. We also included a large performance increase in the file layout which means you need dkist 1.14 to read them.

(For extra performance on asdf load install asdf 4.3.0 at the same time)