JP-2807: Update ATOCA algorithm

[tapastro]

Resolves JP-2807

Closes #

This PR updates the ATOCA algorithm from the initial merge to the current development branch, increasing stability and reducing runtime. The modeling is now off by default, and some documentation is required to cover the new step arguments.

Checklist

• [x] added entry in CHANGES.rst within the relevant release section
• [ ] updated or added relevant tests
• [ ] updated relevant documentation
• [x] added relevant milestone
• [x] added relevant label(s)

[codecov[bot]]

Codecov Report

Base: 79.65% // Head: 80.00% // Increases project coverage by +0.34% :tada:

Coverage data is based on head (9098211) compared to base (c021a43). Patch coverage: 86.97% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6945      +/-   ##
==========================================
+ Coverage   79.65%   80.00%   +0.34%     
==========================================
  Files         412      413       +1     
  Lines       37585    37913     +328     
==========================================
+ Hits        29938    30331     +393     
+ Misses       7647     7582      -65     

Flag	Coverage Δ
nightly	80.00% <86.97%> (+0.34%)	:arrow_up:

Impacted Files	Coverage Δ
jwst/extract_1d/soss_extract/soss_solver.py	85.71% <77.27%> (+10.17%)	:arrow_up:
jwst/extract_1d/soss_extract/atoca_utils.py	69.90% <82.05%> (+5.75%)	:arrow_up:
jwst/extract_1d/soss_extract/atoca.py	78.41% <86.20%> (+2.85%)	:arrow_up:
jwst/extract_1d/soss_extract/soss_extract.py	90.01% <89.05%> (+13.30%)	:arrow_up:
jwst/datamodels/__init__.py	100.00% <100.00%> (ø)
jwst/datamodels/sosswavegrid.py	100.00% <100.00%> (ø)
jwst/extract_1d/extract_1d_step.py	76.51% <100.00%> (+3.32%)	:arrow_up:
jwst/extract_1d/soss_extract/soss_boxextract.py	89.55% <100.00%> (+2.25%)	:arrow_up:
jwst/extract_1d/soss_extract/soss_syscor.py	20.73% <0.00%> (-26.83%)	:arrow_down:
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[tapastro]

Converting from draft was done too quickly - this isn't quite ready yet.

[tapastro]

Updated regression tests with coverage for new ATOCA flags - run here: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/470/

EDIT: this run stalled out and after 4 hours, I killed it off. Running them locally to see if I can reproduce the hang...

[tapastro]

Updated tests and responded to review comments. Running another regtest set: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/476/

EDIT: 475 was running on outdated artifacts, so aborted and started 476.

[tapastro]

@hbushouse This could use a review.

[tapastro]

Cleaned up the review comments, apart from the testing coverage - I added a regression test that should add significant coverage, but Codecov will only show once it enters our nightly runs. I am happy to add more tests if it shows some gaps.

[hbushouse]

Is there a recent regtest run for this?

[tapastro]

476 was the last run, from which only a merge to master and docstring updates have been added since. I can run another, though.

[tapastro]

Started another: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/486/

[hbushouse]

Do these look like expected differences? https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/486/testReport/jwst.regtest/test_niriss_soss/

[tapastro]

Do these look like expected differences? https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/486/testReport/jwst.regtest/test_niriss_soss/

The test_niriss_soss_extras truth files were generated on my machine, so platform differences are expected - they should be fine. The test_niriss_soss_stage3 results should also change, as it now uses ATOCA instead of box extraction.

The stage2 difference worried me a bit, but I inspected each output spectrum by eye and there is no appreciable difference. There are also some changes to SOSS-specific extraction parameters SOSSDLTX, SOSSDLTY and SOSSROTN that could likely cause some slight deviation in output.

So in all, I think the changes are expected.

[hbushouse]

Looks good except for the usual suspects in the CI tests, so I'm merging.