astropy
JWST data triggers 'datfix' and 'obsfix' FITSFixedWarning
Summary
When opening the WCS of a JWST/NIRCam image, I encountering two FITSFixedWarning warnings (see full example further below):
WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATE-BEG to '2022-06-07T02:22:37.094' from MJD-BEG.
Set DATE-AVG to '2022-06-07T02:29:41.197' from MJD-AVG.
Set DATE-END to '2022-06-07T02:36:45.299' from MJD-END'. [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'obsfix' made the change 'Set OBSGEO-L to -80.828196 from OBSGEO-[XYZ].
Set OBSGEO-B to -32.874352 from OBSGEO-[XYZ].
Set OBSGEO-H to 1674994226.369 from OBSGEO-[XYZ]'. [astropy.wcs.wcs]
As a user it seems surprising that brand new JWST images need fixes applied.
Is the 'datfix' warning desired or expected here? I note that an issue is already open related to the 'obsfix' warning (see https://github.com/astropy/astropy/issues/10365).
Full example
Python 3.9.7 (default, Sep 7 2021, 13:48:18)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy
>>> astropy.__version__
'5.2.dev271+gfe3486171'
>>> from astropy.io import fits
>>> from astropy.wcs import WCS
>>>
>>> url = "https://stpubdata-jwst.stsci.edu/ero/jw02736/jw02736001001/jw02736001001_02101_00008_nrcblong_i2d.fits"
>>> with fits.open(url) as hdul:
... wcs = WCS(hdul["SCI"].header)
WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATE-BEG to '2022-06-07T02:22:37.094' from MJD-BEG.
Set DATE-AVG to '2022-06-07T02:29:41.197' from MJD-AVG.
Set DATE-END to '2022-06-07T02:36:45.299' from MJD-END'. [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'obsfix' made the change 'Set OBSGEO-L to -80.828196 from OBSGEO-[XYZ].
Set OBSGEO-B to -32.874352 from OBSGEO-[XYZ].
Set OBSGEO-H to 1674994226.369 from OBSGEO-[XYZ]'. [astropy.wcs.wcs]
This should be asked over at https://github.com/spacetelescope/jwst/issues ? astropy does not control how JWST makes their data. JWST uses a unique format called ASDF-in-FITS. The canonical WCS is actually the GWCS. The FITS portion is created after-the-fact and might not be complete.
I think there is actually potentially a small astropy issue here.
From the look of those warnings, what JWST have produced is fully standard compliant, but it looks like wcslib is throwing warnings saying it's populating alternative representations in it's struct with the data it already has. These seem like redundant warnings?
But still, you should grab GWCS instead if it is JWST data. Also, it might take more time for the telescope to fine tune the calibration as initial calibration was done with ground data. See https://jwst-docs.stsci.edu/jwst-calibration-pipeline-caveats
As an aside, @Cadair's comment triggered me to run fitscheck --compliance on JWST data just now and it has the following to say:
$ fitscheck --compliance https://stpubdata-jwst.stsci.edu/ero/jw02736/jw02736001001/jw02736001001_02101_00008_nrcblong_i2d.fits
MISSING 'https://stpubdata-jwst.stsci.edu/ero/jw02736/jw02736001001/jw02736001001_02101_00008_nrcblong_i2d.fits' .. Checksum not found in HDU #0
1 errors
Why is this an error?
I suppose it ought to be a warning. It looks like the FITS standard strongly recommends a CHECKSUM in every extension, but does not require it:
Is this worth a separate AstroPy issue, or are we too far down the weeds?
But still, you should grab GWCS instead if it is JWST data.
Should we introduce a warning when users attempt to use the traditional WCS class with JWST data?
So should we have a warning for every thing that is recommended but not required? That is, if an observatory chooses not to write checksums, should users of those files have to read warnings? about what? that the observatory does not implement an optional feature or that astropy.fits cannot verify authenticity of the file because [the optional, not required] checksum is missing? Is verification of authenticity always required of astropy.fits?
I think there is actually potentially a small astropy issue here.
From the look of those warnings, what JWST have produced is fully standard compliant, but it looks like wcslib is throwing warnings saying it's populating alternative representations in it's struct with the data it already has. These seem like redundant warnings?
I disagree with
a small astropy issue here
but I do agree with the rest of the assessment. In particular that it is a WCSLIb issue. This a warning/info message from WCSLIB. I do not know how important this info is for the users: should users know that LBH values were derived from XYZ or that XYZ values were overwritten by values derived from LBH (current WCSLIB logic/preference). I do not know why it is even necessary to have both LBH and XYZ in WCSLIB. Shouldn't it be able to keep track of what was provided and call appropriate function for internal computations [and just leave user-provided info in the headers alone]?
Same thing is happening with DATE/MJD/... From the paper (VII, I think):
As in the case of the time reference value (see Sect. 4.1.2), the JD values supersede DATE values, and MJD values supersede both, in cases where conflicting values are present.
Why WCSLIB does not simply work with what was provided in the header and has to derive DATE from MJD is beyond my expertise. Also, the significance of these messages to the user and whether they should be interpreted a INFO or WARNING is again beyond my expertise.
I would suggest maybe @Cadair contact directly Mark Calabreta about this issue if you have a better understanding of the standards. Alternatively, on our end, if no one cares about these messages, we could probably filter them out without changes to WCSLIB. I do not feel in a position to decide the best approach here.
Is this worth a separate AstroPy issue or are we too far down the weeds?
Sure, feel free to open issue (or better yet, PR) about the checksum error/warning.
And yes, we are always in the weeds when it comes to WCS and FITS.
Hi humans :wave: - this issue was labeled as Close? approximately 13 hours ago. If you think this issue should not be closed, a maintainer should remove the Close? label - otherwise, I will close this issue in 7 days.
If you believe I commented on this issue incorrectly, please report this here
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
In particular that it is a WCSLIb issue.
I was conflating these two things. It's a usability issue for WCS even if it's not actually our code which is causing it :wink:
if you have a better understanding of the standards
I would not be so bold to claim this, I might go eat the book again and see what it says.
I would not be so bold to claim this
... ~if you have a better understanding of the standards~ -> ... if you are courageous enough 😄
I think we need to understand whether these messages are useful, helpful, and/or informative. For example, if someone has a header with DATE and WCSLIB in to_header() replaces DATE with MJD (or even if it just adds MJD to the header), should the user know how MJD was derived? Could this info be used to diagnose issues with files? Is that important? Or should WCSLIB just stick with what user has provided and not try to "upgrade" DATE or OBSGEO [requires WCSLIB changes]? Or can we just always ignore these messages [we can do that in astropy]?
I simply do not know enough on this topic to say anything but I feel we need to understand this before we decide on how to proceed.
Back to the original JWST WCS... @barentsen , are you able to grab the GWCS from ASDF extension?
So, @barentsen , did you hear back from JWST help desk? I am not sure if there is anything astropy can do here.
I do not think this should be labelled as "close". There is work for someone to do here, it might be a fix in wcslib or it might be on us but either way these warnings are very often superfluous and we should work on that.