JP-2286: Refactor the Bayesian evidence step of residual fringe correction

[WilliamJamieson]

Description

This PR switches the Bayesian evidence step of the residual fringe correction from using the BayesicFitting library to using astropy based models and fitters. Note that this will change the results of the residual fringe correction, and these changes should be investigated throughly. The changes are due to differences in how the numerical approximations of the Bayesian evidence are computed (scipy/astropy vs BayesicFitting based computations).

Note that this depends on PR #6475 (and should be merged after that PR is merged).

Checklist

• [ ] Tests
• [ ] Documentation
• [ ] Change log
• [ ] Milestone
• [ ] Label(s)

[codecov[bot]]

Codecov Report

Merging #6491 (8d31e37) into master (2f0bda5) will decrease coverage by 0.90%. The diff coverage is 23.87%.

@@            Coverage Diff             @@
##           master    #6491      +/-   ##
==========================================
- Coverage   78.33%   77.43%   -0.91%     
==========================================
  Files         409      411       +2     
  Lines       34840    35821     +981     
==========================================
+ Hits        27291    27737     +446     
- Misses       7549     8084     +535     

Flag	Coverage Δ		*Carryforward flag
nightly	78.32% <14.28%> (ø)		Carriedforward from 3226af4
unit	55.51% <24.18%> (-0.15%)	:arrow_down:

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
jwst/residual_fringe/utils.py	10.19% <19.04%> (+1.04%)	:arrow_up:
jwst/residual_fringe/fitter.py	21.18% <21.18%> (ø)
jwst/residual_fringe/fourier.py	32.25% <32.25%> (ø)
jwst/cube_build/ifu_cube.py	72.58% <0.00%> (-11.68%)	:arrow_down:
jwst/associations/pool.py	86.84% <0.00%> (-7.90%)	:arrow_down:
jwst/lib/v1_calculate.py	18.91% <0.00%> (-7.17%)	:arrow_down:
jwst/associations/lib/product_utils.py	94.73% <0.00%> (-0.10%)	:arrow_down:
jwst/datamodels/make_header.py	91.36% <0.00%> (-0.03%)	:arrow_down:
jwst/timeconversion/time_conversion.py	3.24% <0.00%> (+0.02%)	:arrow_up:
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2f0bda5...8d31e37. Read the comment docs.

[hbushouse]

How does this PR relate to all the other ones currently open for residual fringe work, such as #6475 , #6761 , and #6739 ? Can this one be closed, because it's obsolete?

[WilliamJamieson]

How does this PR relate to all the other ones currently open for residual fringe work, such as #6475 , #6761 , and #6739 ? Can this one be closed, because it's obsolete?

Not quite yet. This is the second "phase" (it refactors the second step/part) of the residual fringe correction refactor. The other PRs are related to the first phase of the residual fringe correction.

[jemorrison]

@WilliamJamieson were on we on this PR ? The last time I checked you where looking into the why the new code took so long to run. Patrick K. has bug fixes for fringe correction using the flight data. I was going to create a branch for him and he was going make the changes, then I was going to review it. I was wondering if this PR should be merged first or if we should just work in parallel ?

[hbushouse]

Any updates on the status of this? The residual fringe correction step is now available in the cal pipeline. Is this work meant to be an enhancement to the baseline algorithm, in order to remove a dependency on BayesicFitting (which I assume is in use in the baseline version)?

[WilliamJamieson]

Any updates on the status of this? The residual fringe correction step is now available in the cal pipeline. Is this work meant to be an enhancement to the baseline algorithm, in order to remove a dependency on BayesicFitting (which I assume is in use in the baseline version)?

The issue is that astropy has extreme bottle necks on some of the fitting involved, which means that this causes the residual fringe to take >5x longer than the current code. It will require extremely careful benchmarking to narrow down the issues. For now I'm keeping it around because it works (but it is much slower), or for the event that we absolutely need to drop the BaysicFitting dependency.