Eureka
Eureka copied to clipboard
kevin218
Exponential Ramp Model Degeneracies
Instrument
Light curve fitting (Stages 4-6)
What happened?
So I've noticed that using all three terms in the exponential ramp model consistently gives me terrible fits due to some super strong degeneracies between the parameters, but using only r0 and r1 (and r3 and r4 for the second ramp) gives great results. @kevin218 do you think it is still worth keeping the ability to fit the r2 and r5 terms, or should we just remove them?
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
No, the function is written as intended. However, users should never use all three terms to fit the exponential ramp. One of r0 and r2 should be fixed, the other free. The correlations between [r0,r1] are different than the correlation between [r1,r2], despite achieving identical fits. I believe fixing r0 minimizes correlations and achieves faster exploration of the parameter space, but don't quote me on that. The goal is to provide flexibility to the user and let them decide which parameters to fit.
Also, I'm not sure if you saw this, but the MIRI ramp is a decreasing exponential, while the NIR instrument ramps are rising exponentials. With a single function, it's easy to switch between the two forms by setting r0=+1 or -1.
Ah okay, I'll try using just r1 and r2 instead and see how that compares. This really needs to be documented somewhere though if we're keeping it as is. Yeah, the MIRI observations showed an idle recovery ramp instead which was quite easy to fit - I don't know that we should expect that for all observations though since I think the MIRI detector hadn't been used for quite a long time before those observations were taken. But our framework handles the ramps well regardless
Thinking more on this, I now realize that there is an exact degeneracy between r0 and r2 -> r0=exp(r2) (I hadn't looked closely at the function's equation earlier). r0 isn't too hard to guess using the amplitude of the ramp (around r0 ≈ 0.0025 for the MIRI observations which show a ramp amplitude of 0.005), but r2 is very challenging to guess and put good priors on (without just guessing r0 and then solving the equation above).
To test for faster parameter explorations one way or another, I ran two dynesty fits to a MIRI white-light phase curve and found that using [r1, r2, r4, r5] took 20m54s while using [r0, r1, r3, r4] took 21m02s (all other settings were held constant and the prior volumes were roughly equal but slightly favouring [r1, r2, r4, r5]).
If it's just a matter of personal preference, we can definitely keep things as is, but it does seem that having r2 and r5 provides no meaningful benefit and it opens users up to potential confusion (by allowing the possibility of fitting all of r0-r5).
Can you try the same experiment with EMCEE? My experience has been different with mcmc.
I know Yoni recently encountered this degeneracy when working on MIRI data for the first time (not sure he would've figured out what to do had I not told him, given the current state of the docs and the lack of any warnings/errors from the code). We should either change how the code is written or add an error message that says you can't freely fit all three parameters at the same time.
This issue is largely taken care of by our docs page at https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5-fit-parameters, but I suggest we add bolding like
Users should not fit all three parameters from each model at the same time as there are significant degeneracies...