jwst
jwst copied to clipboard
spacetelescope
NIRSpec MOS level 3 master background oversubtraction
Issue JP-2247 was created on JIRA by James Muzerolle:
I tried creating a master background for a MOS simulation using the S1600A1 slit. The 1D master background spectrum looks fine, however, when I apply it in spec3 to one of the MOS slits (point source), it massively oversubtracts (see first attachment). I think perhaps the units are not being converted properly? I get better results if I change srctype for the source slit to 'extended' (second attachment) - the source trace is actually visible, although the off-source flux is negative instead of ~ zero.
Comment by Howard Bushouse on JIRA:
James Muzerolle what exactly were the steps you used to try to do the master bkg subtraction (using your supplied 1D bkg spectrum)? Normally the master bkg subtraction for MOS mode is done during calwebb_spec2, using the data from dedicated bkg slits to act as the master bkg, which then gets subtracted from all the other slitlets in the exposure. Did you just run the master_background step by itself, supplying the cal product and your bkg spectrum product as inputs?
Comment by James Muzerolle on JIRA:
Howard Bushouse This was part of testing the capability of MBS with a user-supplied background spectrum for all modes. That may not be a common use case for MOS given the ability to include background slits in the same exposure, but there are scenarios where it might be desirable.
I first created a level 3 association file that includes just one level 2b science exposure and the FS 1d spectrum as the associated background (a copy is in the central store directory). The science cal file itself was generated by running spec2 without any background subtraction. Then I ran the full spec3 pipeline (skipping outlier detection) with that asn file as input.
Comment by Howard Bushouse on JIRA:
James Muzerolle what exactly were the steps you used to try to do the master bkg subtraction (using your supplied 1D bkg spectrum)? Normally the master bkg subtraction for MOS mode is done during calwebb_spec2, using the data from dedicated bkg slits to act as the master bkg, which then gets subtracted from all the other slitlets in the exposure. Did you just run the master_background step by itself, supplying the cal product and your bkg spectrum product as inputs?
Comment by James Muzerolle on JIRA:
Howard Bushouse This was part of testing the capability of MBS with a user-supplied background spectrum for all modes. That may not be a common use case for MOS given the ability to include background slits in the same exposure, but there are scenarios where it might be desirable.
I first created a level 3 association file that includes just one level 2b science exposure and the FS 1d spectrum as the associated background (a copy is in the central store directory). The science cal file itself was generated by running spec2 without any background subtraction. Then I ran the full spec3 pipeline (skipping outlier detection) with that asn file as input.
Comment by Jane Morrison on JIRA:
James Muzerolle I could not find your directory in ████████████████████████████████
I might not have mounted a directory correctly. Does the data for this ticket still exist in the directory you point to at the top of this ticket ?
Comment by Howard Bushouse on JIRA:
██████████████ is one of the central store directories that needs to be mounted using st-QuickMount on Mac Monterey. Ah, but I see what you mean: after mounting it the "Muzerolle" directory still doesn't show up anymore under wit4/nirspec.
Comment by James Muzerolle on JIRA:
Jane Morrison shoot, sorry about that! I changed the directory name to ███████████████████████████ (recently changed my last name, but haven't propagated it through everything yet!).
Comment by Jane Morrison on JIRA:
James Muzerolle I am getting caught up on MOS processing. As I understand it background subtraction for MOS is done in calspec2 in the master background step. There is an option to use a user supplied background in the master background step. I ran it and it run to completion. But need a little help on which user background to use. in your directory: bkgd_from_fs
2027_MOS_NRSCEERS-DC1-F170LP-G235M_1shutter_fsbkgd_s02140_x1d.fits
and in the ticket you mentioned an extended source background is that:
2027_MOS_NRSCEERS-DC1-F170LP-G235M_1shutter_fsbkgd_s02140_extended_x1d.fits
I plotted and looked at the x1d files. The extended one looks reasonable to me but the other one has flux units of -1e14 that seems wrong or I do not understand the units
So my first question is what is the name of the user background you want to use ?
Comment by Jane Morrison on JIRA:
James Muzerolle in the images that you originally attached are those from the cal file or the s2d ?
Not that it really matters just trying to understand your plots and see if what I am going is an improvement
Comment by James Muzerolle on JIRA:
Jane Morrison the user background should be
2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_FS_cts_STready_x1d.fits. The files you listed are the MOS background-subtracted spectra (one where the source was given srctype=point, the other srctype=extended). Note the units will be funny as these were generated over a year ago before we had any kind of flux calibration.
The attached images I think show the cal files for the two different source types (2027s02140_cal.fits and 2027s02140_extended_cal.fits), demonstrating that the subtraction seems to work ok in the latter case, but not the former.
Comment by Jane Morrison on JIRA:
James Muzerolle Howard Bushouse I am somewhat new to working with MOS spectra so might have done this all wrong. Let me step through what I have done.
Masterbackground subtraction for MOS data is done in calspec2. There is an option in calspec2 to use a user provided masterbackground. So I tried
- in calspec2 use your user provide background as a master background
- In calspec2 skip the masterbackground step.
- Just run calspec2 with no options - this will run master background using the background slits and combining them
Here is syntax for the first thing I tried with the user provide background:
strun calwebb_spec2 nirspec_mossim_d1_prism_nrs1_1slitlet_MBS_rate.fits --steps.master_background_mos.user_background='2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_FS_cts_STready_x1d.fits'
When I ran it skipping master background I doubled checked the log file and it does SKIP it.
The results are not what I expected. I plotted the user provided background. Then I plotted the x1d output from the three runs of calspec2 - I selected the 13th extension in the fits file because that is for the point source data (I hope).
Where the user provide background is defined there is no difference in the two x1d files when using the user background and skipping masterbackground. To double check this I subtracted the two results and got zero there the user provided x1d file is defined.
So if I have done this correct what I need is to fix is how the masterbackground step is done in calspec2 when a user provided background is used. Sounds about right ?
Comments ?
Comment by Howard Bushouse on JIRA:
It sounds like you've done everything correctly. Did you by any chance compare the outputs for the case of no mbkg subtraction and the "normal" mbkg subtraction (i.e. using the dedicated bkg slitlets)? Do those show a difference? If not, then mbkg isn't working at all. If it does show the expected change, then we've got a problem with just the user-supplied mbkg subtraction and we should try to debug the flow of all the data (both a science slitlet and the mbkg spectrum) through the various phases. It could be that the user-supplied spectrum isn't being inserted into the overall flow properly.
Comment by Jane Morrison on JIRA:
After further testing. I think the default masterbackground step using the background slits is working. It is just when the user provided background is provided- no background is subtracted. I updated the x1d plots of the results from different runs with the user provided background and the COMBINE X1d determined from default running.
Also subtracted the point source CAL images for NO SUBTRACTION RESULT - USER PROVIDED RESULT (~ 0) and NO SUBTRACTION RESULT - DEFAULT method
Comment by Jane Morrison on JIRA:
James Muzerolle Howard Bushouse I believe the problem is not with the master background step but with the User provided spectrum. The Surface brightness seems too small and the surface brightness is what is used in the master background step.
As a test I ran the pipeline with all the defaults which will create a master background using the background slits. I saved that master background and used it in a test run of the pipeline setting the master_background.mos.user_background to this saved one. The results of using this "user" provided background and the default of internally creating one - gave the same results. I then looked closer at the user provide background from James (2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_FS_cts_STready_x1d.fits) and looked at the results of using it. On further inspection the background is being subtracted but the background is so small it was not obvious on plots (until I zoomed in a lot).I compared for background values, the science values before subtraction and the science after subtraction and the background is being subtracted. I have a plot of the background created using combine1d and the background slits and the user provided background from James which hopefully shows the problem . I made two plots with different scales.
Maybe the real problem is in how the User background James created was made. I am assuming you used to the pipeline process of a FS to create this background ?
Comment by Howard Bushouse on JIRA:
Could this be due to the fact that the spectrum supplied by James Muzerolle is old enough that it was created before there was anything close to a flux calibration included in the reference files for NIRSpec, whereas now there is? Hence when you rerun spec2 on the science exposure, it's getting flux calibration applied, but the bkg spectrum doesn't have flux calibration (or at least not the proper one). That could be throwing off the scale between the two.
Comment by Jane Morrison on JIRA:
James Muzerolle Have you had a chance to remake the user provided sprectrum made from FS data ?
Comment by Jane Morrison on JIRA:
James Muzerolle how did you create the user background. Can I pull it out from this data and finish the testing ?
Comment by James Muzerolle on JIRA:
Apologies Jane Morrison for the delay... What I did was to reprocess the simulation as FS data by replacing the appropriate keywords. I've copied the converted rate file and original processed cal file to the directory on central store (2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_FS_cts_STready.fits is the rate file). These are still the original files, I haven't reprocessed them with the latest reference files. They probably should be, and using the dev version of the pipeline to make sure you have the pixel area bug fix (relevant for the background extraction with SRCTYPE=EXTENDED).
I am curious as to why you're using a PRISM exposure as the science data, when the background was created from a grating? I guess in principle that should work if the calibrations are all correct, but it's not what a user would do.
Comment by Jane Morrison on JIRA:
James Muzerolle I think we might need to back up. I used the PRISM as the science data because I thought that is what you were testing. What did you want to use as the science data ?
I think I will need to reprocess the data that was used to make the user background. I can take the rate file: 027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_FS_cts_STready.fits and re-run calspec2 on this data produce a new cal file. Then I take the x1d file that results and use it as the user background ?
Comment by James Muzerolle on JIRA:
Jane Morrison I originally used the same exposure as the one from which I extracted the background (just a different slit). I just realized, though, that I neglected to provide the original rate file that you need to reprocess using the supplied background. It's there now: 2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_cts_STready_1shutter.fits.
Comment by Jane Morrison on JIRA:
James Muzerolle I am just getting back to this and reading over the thread to see what I need to do. You put 2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_cts_STready_1shutter.fits on the directory for me to use. But that is 2048 X 2048 rate file. So what do I do with that ? Obviously I am a little confused on the test.
This ticket is about MOS data processing. I am testing running a mos science image through calspec2 using a user provided background.
Could you clearly state what the MOS science data is ?
Is it still true that the user provided background is 2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_FS_cts_STready_x1d.fits.
Comment by James Muzerolle on JIRA:
Jane Morrison That rate file is the MOS data, which you need to run through calspec2 along with the x1d file containing the background spectrum.
Comment by Jane Morrison on JIRA:
James Muzerolle the rate file
2027_MOS_NRSCEERS-DC1-F170LP-G235M-00_1_491_SE_2021-05-26T23h43m50_cts_STready_1shutter.fits has a msa file in the header as
newmetafile_1shutter_msa.fits
But I do not have that msa file and I cannot find it in your directory.
I have one that I must have gotten from you called
newmetafile_1slitlet+background_msa.fits Should I use that one (just copy it to newmetafile_1shutter_msa.fits ?)
Comment by Jane Morrison on JIRA:
James Muzerolle Thanks for creating the fixed slit user background x1d file. That was very helpful.
I ran the three tests on the data. For each run I changed the MSA file to be
newmetafile_1slitlet+background_msa.fits. This has the information for background slits and the 1 point source slit.
Default method: run the calspec2 pipeline using master background step to determine the background from the 12 background slits. Save the master x1d background that was calculated.
User the user background in the 'master_background_user' option in calspec2
Turn off subtracting a background.
I plotted the calculated master background and the user background to confirm they are ~similar. Units are 1e-06.
!image-2023-02-07-10-05-19-450.png|width=659,height=236! Then I plotted the point source 1xd values (13th extension in the x1d final output). I plotted the results from the 3 different runs. They look very reasonable to me and expected given the calculated and user provided backgrounds !image-2023-02-07-10-06-25-583.png|width=643,height=224! James Muzerolle do they look ok to you? If so I will close this ticket.
Comment by James Muzerolle on JIRA:
Jane Morrison yes, that looks about right, thanks! Just for completeness, I presume the two background spectra in the first plot have roughly the same flux level when plotted in surface brightness units?
I think my problem all along was trying to process with the user-supplied background at level 3, instead of level 2.
Comment by Howard Bushouse on JIRA:
No code work or updates needed. Just a misunderstanding in the correct approach to doing the processing.