Background subtraction in spectral extraction (cubeviz)

[kecnry]

Description

This pull request implements support for background subtraction in cubeviz's spectral extraction.

Since this has now gone far beyond "collapsing", this PR also takes this and the 4.0 major version bump as an excuse to rename collapse_to_spectrum(...) to extract(...) and then also adds a new extract_bg_spectrum(...) (using the same name as in specviz2d in case we ever do want to support extracting the background as well... we could consider just naming this to extract_background(...) if we don't want to account for that possibility down the road).

https://github.com/spacetelescope/jdaviz/assets/877591/cdc5eebe-2405-4dda-9aab-df3e659334d0

TODO:

• [x] live preview of background spectrum
• [x] ability to export background spectrum independently (in same way as is done in specviz2d)
• [x] determine what to do for sum case with different sized subsets (possibly by rescaling the background spectrum by the relative areas - *= aperture_area / background_area?)
• [x] test coverage
• [ ] investigate glitchy disappearing of preview markers - this seems to have been happening before as well so might be better as a follow-up effort

Change log entry

• [ ] Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts, list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• [x] Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• [ ] Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• [ ] Do the proposed changes follow the STScI Style Guides?
• [ ] Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• [ ] Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• [ ] Did the CI pass? If not, are the failures related?
• [x] Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• [ ] After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 94.44444% with 8 lines in your changes missing coverage. Please review.

Project coverage is 88.69%. Comparing base (1c5589e) to head (ba4d660). Report is 154 commits behind head on main.

Files	Patch %	Lines
...plugins/spectral_extraction/spectral_extraction.py	92.38%	8 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2859   +/-   ##
=======================================
  Coverage   88.68%   88.69%           
=======================================
  Files         109      109           
  Lines       16996    17044   +48     
=======================================
+ Hits        15073    15117   +44     
- Misses       1923     1927    +4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[camipacifici]

The workflow is nice and the orange highlights in the viewers are nice too, but it looks like it is inverting flux and background when subtracting. If I set 'none' for the background, the extracted spectrum is not flipped.

[camipacifici]

Another thing I noticed:

• create subset 1 for source and subset 2 for background
• use spectral extraction plugin with subset 1 (not using subset 2 for background now...see above). i see the orange preview in the spectrum viewer.
• hit extract -> spectrum turns red and preview disappears
• modify subset 1 -> preview does not reappear following the newly defined subset 1 I would have intuitively expect the preview to come back, but please let me know if this is intentional and we can discuss. Thank you!

[kecnry]

but it looks like it is inverting flux and background when subtracting.

Is it possible that there are negative surface brightnesses in either of those apertures?

I would have intuitively expect the preview to come back, but please let me know if this is intentional and we can discuss.

There are some known bugs with the preview disappearing that need to be addressed yet - but I'm quite sure that they are not introduced here and so probably don't need to block this PR (but maybe should block the next release).

[kecnry]

If I have a spectral subset, the spectrum created matches. If I update the color of the spatial subset in plot options, the spectrum doesn't update. I think we talked about this with Patrick that this was the intended behavior, so leaving this here just to make sure!

Yes, this was decided to be the desired behavior.

[camipacifici]

@bmorris3 naively I thought that the background spectrum was subtracted per spaxel before extraction. If that is not the case, then yes, that normalization is necessary.

[kecnry]

@camipacifici - I think I found and fixed the bug resulting in the unexpected negative spectrum (it was incorrectly still using the aperture mask but weighing by the relative area ratios). Please let me know if it seems to act as expected on your end now.

~EDIT: looking into why this seems to have caused test failures in model fitting~ 🤔