specutils Issue with reading JWST MRS cubes

Hi all, I ran into an issue with reading JWST MRS cubes. Is this a known issue or have others run into this problem before?

spec1d = Spectrum1D.read('file_s3d.fits') ImportError: cannot import name 'convert_fitsrec_to_array_in_tree' from 'stdatamodels.util'

specutils version: 1.15.1.dev6+g28b0cdb stdatamodels version: 1.10.2.dev5+g0326fd4 python version: 3.12.2

May 21 '24 19:05 kykyelric

Hmm, I haven't run into this, and I was unable to reproduce locally. What happens if you just try from stdatamodels.util import convert_fitsrec_to_array_in_tree in python? It seems like it might be a problem with your stdatamodels installation.

May 21 '24 20:05 rosteen

Hmm, I haven't run into this, and I was unable to reproduce locally. What happens if you just try from stdatamodels.util import convert_fitsrec_to_array_in_tree in python? It seems like it might be a problem with your stdatamodels installation.

This imported just fine when I run python in terminal. Strange.

May 21 '24 20:05 kykyelric

That is odd. I would probably try reinstalling specutils and see if that fixes the issue - I don't know if there's some way it could have ended up looking for stdatamodels in the wrong place, but you never know.

May 21 '24 20:05 rosteen

Unfortunately that didn't work. I found a way to accomplish what I needed to via a different package than specutils, but I hope this issue can get resolved later.

May 21 '24 20:05 kykyelric

What is the full traceback leading to that error? That would help track this down if it's something on our end. As-is, without being able to reproduce the error with the package versions you posted, I can't fix it.

May 21 '24 20:05 rosteen

ImportError                               Traceback (most recent call last)
Cell In[77], line 14
     10 q3di = q3din(name, 'test', datext=1, varext=2, dqext=3, 
     11              outdir='F20551-4250/s3d_files/test/', 
     12              logfile='F20551-4250/s3d_files/test/ch'+str(i+1)+'-fitlog.txt')
     13 wav = q3di.load_cube().wave
---> 14 spec1d = Spectrum1D.read(name)
     15 #wav = np.array(spec1d.spectral_axis.value)
     16 corr_interp = interp1d(corrx, corry)(wav)

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/astropy/nddata/mixins/ndio.py:59, in NDDataRead.__call__(self, *args, **kwargs)
     58 def __call__(self, *args, **kwargs):
---> 59     return self.registry.read(self._cls, *args, **kwargs)

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/astropy/io/registry/core.py:218, in UnifiedInputRegistry.read(self, cls, format, cache, *args, **kwargs)
    213     format = self._get_valid_format(
    214         "read", cls, path, fileobj, args, kwargs
    215     )
    217 reader = self.get_reader(format, cls)
--> 218 data = reader(*args, **kwargs)
    220 if not isinstance(data, cls):
    221     # User has read with a subclass where only the parent class is
    222     # registered.  This returns the parent class, so try coercing
    223     # to desired subclass.
    224     try:

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/specutils/io/default_loaders/jwst_reader.py:539, in jwst_s3d_single_loader(filename, **kwargs)
    521 @data_loader(
    522     "JWST s3d", identifier=identify_jwst_s3d_fits, dtype=Spectrum1D,
    523     extensions=['fits'], priority=10,
    524 )
    525 def jwst_s3d_single_loader(filename, **kwargs):
    526     """
    527     Loader for JWST s3d 3D rectified spectral data in FITS format.
    528 
   (...)
    537         The spectrum contained in the file.
    538     """
--> 539     spectrum_list = _jwst_s3d_loader(filename, **kwargs)
    540     if len(spectrum_list) == 1:
    541         return spectrum_list[0]

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/specutils/io/default_loaders/jwst_reader.py:563, in _jwst_s3d_loader(filename, **kwargs)
    549 def _jwst_s3d_loader(filename, **kwargs):
    550     """
    551     Loader for JWST s3d 3D rectified spectral data in FITS format.
    552 
   (...)
    561         The spectra contained in the file.
    562     """
--> 563     from stdatamodels import asdf_in_fits
    565     spectra = []
    567     # Get a list of GWCS objects from the slits

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/stdatamodels/__init__.py:1
----> 1 from .model_base import DataModel
      2 from . import _version
      5 __all__ = ['DataModel', '__version__']

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/stdatamodels/model_base.py:28
     26 from . import properties
     27 from . import schema as mschema
---> 28 from . import validate
     29 from .util import get_envar_as_boolean, remove_none_from_tree
     30 from . import s3_utils

ImportError: cannot import name 'convert_fitsrec_to_array_in_tree' from 'stdatamodels.util' (/home/kydan/anaconda3/envs/correct_cube/lib/python3.12/site-packages/stdatamodels/util.py)

May 22 '24 17:05 kykyelric

Hmm, looks like this could be ASDF-related. Could you tell me what the output of pip list | grep asdf is for you?

May 28 '24 18:05 rosteen

asdf 3.2.0 asdf-astropy 0.5.0 asdf-coordinates-schemas 0.2.0 asdf_standard 1.1.1 asdf_transform_schemas 0.5.0 asdf-unit-schemas 0.1.0 asdf-wcs-schemas 0.3.0

May 28 '24 18:05 kykyelric

Hm, some of those are out of date compared to what I have. Have you tried installing specutils in a fresh environment to ensure you have up-to-date/compatible package versions? I downgraded my asdf packages (and gwcs for compatibility) to what you have there and still couldn't reproduce, so I think new environment is probably the best way to solve this if you haven't tried it already. It seems to be an upstream package thing, not specutils, but I remain stumped on what package or why.

May 28 '24 18:05 rosteen

I am already using an isolated conda environment. I can export an environment file if you'd like to try it?

May 28 '24 18:05 kykyelric

I am already using an isolated conda environment. I can export an environment file if you'd like to try it?

Sure. I guess my question was more "when was the environment created", in case it's older and something didn't get updated that was needed for compatibility. But yes, the env export or pip list would help.

May 28 '24 18:05 rosteen

name: correct_cube channels:

anaconda
conda-forge
defaults dependencies:
_libgcc_mutex=0.1=conda_forge
_openmp_mutex=4.5=2_gnu
affine=2.4.0=pyhd8ed1ab_0
aom=3.8.1=h59595ed_0
arrow=1.2.3=py312h06a4308_1
asdf-astropy=0.5.0=pyhd8ed1ab_0
asdf-coordinates-schemas=0.2.0=pyhd8ed1ab_1
asdf-unit-schemas=0.1.0=pyhd8ed1ab_0
asdf-wcs-schemas=0.3.0=pyhd8ed1ab_0
astropy=5.3.4=py312ha883a20_0
asttokens=2.0.5=pyhd3eb1b0_0
attrs=23.2.0=pyh71513ae_0
azure-core-cpp=1.10.3=h91d86a7_1
azure-storage-blobs-cpp=12.10.0=h00ab1b0_0
azure-storage-common-cpp=12.5.0=hb858b4b_2
backcall=0.2.0=pyhd3eb1b0_0
blas=1.0=mkl
blosc=1.21.5=h0f2a231_0
bottleneck=1.3.7=py312hc7c0aa3_1
brotli=1.1.0=hd590300_1
brotli-bin=1.1.0=hd590300_1
brunsli=0.1=h9c3ff4c_0
bzip2=1.0.8=h7b6447c_0
c-ares=1.27.0=hd590300_0
c-blosc2=2.13.2=hb4ffafa_0
ca-certificates=2024.2.2=hbcca054_0
cairo=1.18.0=h3faef2a_0
certifi=2024.2.2=pyhd8ed1ab_0
cfitsio=4.3.1=hbdc6101_0
charls=2.4.2=h59595ed_0
click=8.1.7=unix_pyh707e725_0
click-plugins=1.1.1=py_0
cligj=0.7.2=pyhd8ed1ab_1
colorama=0.4.6=pyhd8ed1ab_0
comm=0.1.2=py312h06a4308_0
contourpy=1.2.0=py312h8572e83_0
cycler=0.12.1=pyhd8ed1ab_0
dav1d=1.2.1=hd590300_0
debugpy=1.6.7=py312h6a678d5_0
decorator=5.1.1=pyhd3eb1b0_0
entrypoints=0.4=py312h06a4308_0
executing=0.8.3=pyhd3eb1b0_0
expat=2.5.0=hcb278e6_1
font-ttf-dejavu-sans-mono=2.37=hab24e00_0
font-ttf-inconsolata=3.000=h77eed37_0
font-ttf-source-code-pro=2.038=h77eed37_0
font-ttf-ubuntu=0.83=h77eed37_1
fontconfig=2.14.2=h14ed4e7_0
fonts-conda-ecosystem=1=0
fonts-conda-forge=1=0
fonttools=4.49.0=py312h98912ed_0
freetype=2.12.1=h267a509_2
freexl=2.0.0=h743c826_0
geos=3.12.1=h59595ed_0
geotiff=1.7.1=h6b2125f_15
gettext=0.21.1=h27087fc_0
giflib=5.2.1=h0b41bf4_3
gwcs=0.20.0=pyhd8ed1ab_0
hdf4=4.2.15=h2a13503_7
hdf5=1.14.3=nompi_h4f84152_100
icu=73.2=h59595ed_0
idna=3.4=py312h06a4308_0
imagecodecs=2024.1.1=py312ha6884dc_0
imageio=2.34.0=pyh4b66e23_0
importlib-metadata=7.0.1=pyha770c72_0
importlib-resources=6.1.1=pyhd8ed1ab_0
importlib_resources=6.1.1=pyhd8ed1ab_0
intel-openmp=2023.1.0=hdb19cb5_46306
ipykernel=6.28.0=py312h06a4308_0
ipython=8.15.0=py312h06a4308_0
isoduration=20.11.0=pyhd8ed1ab_0
jedi=0.18.1=py312h06a4308_1
jmespath=1.0.1=pyhd8ed1ab_0
joblib=1.3.2=pyhd8ed1ab_0
json-c=0.17=h7ab15ed_0
jupyter_client=7.1.2=pyhd3eb1b0_0
jupyter_core=5.5.0=py312h06a4308_0
jxrlib=1.1=hd590300_3
kealib=1.5.3=h2f55d51_0
keyutils=1.6.1=h166bdaf_0
kiwisolver=1.4.5=py312h8572e83_1
krb5=1.21.2=h659d440_0
lazy_loader=0.3=pyhd8ed1ab_0
lcms2=2.16=hb7c19ff_0
ld_impl_linux-64=2.38=h1181459_1
lerc=4.0.0=h27087fc_0
libabseil=20230802.1=cxx17_h59595ed_0
libaec=1.1.2=h59595ed_1
libarchive=3.7.2=h2aa1ff5_1
libavif16=1.0.4=h1dcd450_0
libblas=3.9.0=1_h86c2bf4_netlib
libboost-headers=1.84.0=ha770c72_1
libbrotlicommon=1.1.0=hd590300_1
libbrotlidec=1.1.0=hd590300_1
libbrotlienc=1.1.0=hd590300_1
libcblas=3.9.0=5_h92ddd45_netlib
libcrc32c=1.1.2=h9c3ff4c_0
libcurl=8.5.0=hca28451_0
libdeflate=1.19=hd590300_0
libedit=3.1.20191231=he28a2e2_2
libev=4.33=hd590300_2
libexpat=2.5.0=hcb278e6_1
libffi=3.4.4=h6a678d5_0
libgcc-ng=13.2.0=h807b86a_5
libgdal=3.8.4=h9323651_0
libgfortran-ng=13.2.0=h69a702a_5
libgfortran5=13.2.0=ha4646dd_5
libglib=2.78.4=h783c2da_0
libgomp=13.2.0=h807b86a_5
libgoogle-cloud=2.17.0=h31df0ca_2
libgoogle-cloud-storage=2.17.0=hc7a4891_2
libgrpc=1.60.1=h74775cd_0
libiconv=1.17=hd590300_2
libjpeg-turbo=3.0.0=hd590300_1
libkml=1.3.0=h01aab08_1018
liblapack=3.9.0=5_h92ddd45_netlib
libnetcdf=4.9.2=nompi_h9612171_113
libnghttp2=1.58.0=h47da74e_1
libnsl=2.0.1=hd590300_0
libpng=1.6.42=h2797004_0
libpq=16.2=h33b98f1_0
libprotobuf=4.25.1=hf27288f_2
libre2-11=2023.06.02=h7a70373_0
librttopo=1.1.0=h8917695_15
libsodium=1.0.18=h7b6447c_0
libspatialite=5.1.0=h7bd4643_4
libsqlite=3.45.1=h2797004_0
libssh2=1.11.0=h0841786_0
libstdcxx-ng=13.2.0=h7e041cc_5
libtiff=4.6.0=ha9c0a0a_2
libuuid=2.38.1=h0b41bf4_0
libwebp-base=1.3.2=hd590300_0
libxcb=1.15=h0b41bf4_0
libxcrypt=4.4.36=hd590300_1
libxml2=2.12.5=h232c23b_0
libzip=1.10.1=h2629f0a_3
libzlib=1.2.13=hd590300_5
libzopfli=1.0.3=h9c3ff4c_0
lz4-c=1.9.4=hcb278e6_0
lzo=2.10=h516909a_1000
matplotlib-base=3.8.2=py312he5832f3_0
matplotlib-inline=0.1.6=py312h06a4308_0
minizip=4.0.4=h0ab5242_0
mkl=2023.1.0=h213fc3f_46344
mkl-service=2.4.0=py312h5eee18b_1
mkl_fft=1.3.8=py312h5eee18b_0
mkl_random=1.2.4=py312hdb19cb5_0
mpl_animators=1.1.1=pyhd8ed1ab_0
munkres=1.1.4=pyh9f0ad1d_0
ncurses=6.4=h6a678d5_0
ndcube=2.2.0=pyhd8ed1ab_0
nest-asyncio=1.5.6=py312h06a4308_0
networkx=3.2.1=pyhd8ed1ab_0
nspr=4.35=h27087fc_0
nss=3.98=h1d7d5a4_0
numpy=1.26.3=py312hc5e2394_0
numpy-base=1.26.3=py312h0da6c21_0
openjpeg=2.5.0=h488ebb8_3
openssl=3.3.0=h4ab18f5_2
packaging=23.1=py312h06a4308_0
pandas=2.2.2=py312h1d6d2e6_1
parso=0.8.3=pyhd3eb1b0_0
patsy=0.5.6=pyhd8ed1ab_0
pcre2=10.42=hcad00b1_0
pexpect=4.8.0=pyhd3eb1b0_3
photutils=1.10.0=py312hc7c0aa3_1
pickleshare=0.7.5=pyhd3eb1b0_1003
pillow=10.2.0=py312hf3581a9_0
pip=23.3.1=py312h06a4308_0
pixman=0.43.2=h59595ed_0
platformdirs=3.10.0=py312h06a4308_0
poppler=24.02.0=h590f24d_0
poppler-data=0.4.12=hd8ed1ab_0
postgresql=16.2=h7387d8b_0
proj=9.3.1=h1d62c97_0
prompt-toolkit=3.0.36=py312h06a4308_0
psutil=5.9.0=py312h5eee18b_0
pthread-stubs=0.4=h36c2ea0_1001
ptyprocess=0.7.0=pyhd3eb1b0_2
pure_eval=0.2.2=pyhd3eb1b0_0
pyerfa=2.0.0=py312h5eee18b_0
pygments=2.15.1=py312h06a4308_1
pyparsing=3.1.1=pyhd8ed1ab_0
python=3.12.2=hab00c5b_0_cpython
python-dateutil=2.8.2=pyhd8ed1ab_0
python-tzdata=2024.1=pyhd8ed1ab_0
python_abi=3.12=4_cp312
pytz=2024.1=pyhd8ed1ab_0
pywavelets=1.4.1=py312hc7c0aa3_1
pyyaml=6.0.1=py312h5eee18b_0
pyzmq=25.1.0=py312h6a678d5_0
rasterio=1.3.9=py312h26ef92c_2
rav1e=0.6.6=he8a937b_2
re2=2023.06.02=h2873b5e_0
readline=8.2=h5eee18b_0
scikit-image=0.22.0=py312hfb8ada1_2
scikit-learn=1.4.0=py312h394d371_0
scipy=1.12.0=py312heda63a1_2
semantic_version=2.10.0=pyhd8ed1ab_0
setuptools=68.2.2=py312h06a4308_0
shapely=2.0.2=py312h9e6bd2c_1
six=1.16.0=pyh6c4a22f_0
snappy=1.1.10=h9fff704_0
snuggs=1.4.7=py_0
sqlite=3.41.2=h5eee18b_0
stack_data=0.2.0=pyhd3eb1b0_0
svt-av1=1.8.0=h59595ed_0
tbb=2021.8.0=hdb19cb5_0
threadpoolctl=3.3.0=pyhc1e730c_0
tifffile=2024.2.12=pyhd8ed1ab_0
tiledb=2.20.0=hd75ad12_0
tk=8.6.13=noxft_h4845f30_101
tornado=6.3.3=py312h5eee18b_0
tqdm=4.66.2=pyhd8ed1ab_0
traitlets=5.7.1=py312h06a4308_0
tzcode=2024a=h3f72095_0
tzdata=2023d=h04d1e81_0
uriparser=0.9.7=hcb278e6_1
wcwidth=0.2.5=pyhd3eb1b0_0
wheel=0.41.2=py312h06a4308_0
xerces-c=3.2.5=hac6953d_0
xorg-kbproto=1.0.7=h7f98852_1002
xorg-libice=1.1.1=hd590300_0
xorg-libsm=1.2.4=h7391055_0
xorg-libx11=1.8.7=h8ee46fc_0
xorg-libxau=1.0.11=hd590300_0
xorg-libxdmcp=1.1.3=h7f98852_0
xorg-libxext=1.3.4=h0b41bf4_2
xorg-libxrender=0.9.11=hd590300_0
xorg-renderproto=0.11.1=h7f98852_1002
xorg-xextproto=7.3.0=h0b41bf4_1003
xorg-xproto=7.0.31=h7f98852_1007
xz=5.4.5=h5eee18b_0
yaml=0.2.5=h7b6447c_0
zeromq=4.3.4=h2531618_0
zfp=1.0.1=h59595ed_0
zipp=3.17.0=pyhd8ed1ab_0
zlib=1.2.13=hd590300_5
zlib-ng=2.0.7=h0b41bf4_0
zstd=1.5.5=hfc55251_0
pip:
- asdf==3.2.0
- asdf-standard==1.1.1
- asdf-transform-schemas==0.5.0
- asteval==0.9.32
- dill==0.3.8
- future==1.0.0
- jsonschema==4.22.0
- jsonschema-specifications==2023.12.1
- lmfit==1.3.1
- plotbin==3.1.5
- ppxf==9.2.2
- q3dfit==1.1.4
- referencing==0.35.1
- rpds-py==0.18.1
- specutils==1.15.1.dev6+g28b0cdb
- statsmodels==0.15.0.dev280+gbc1899510
- stdatamodels==1.10.2.dev5+g0326fd4
- uncertainties==3.1.7 prefix: /home/kydan/anaconda3/envs/correct_cube

May 28 '24 18:05 kykyelric