Performance Metrics

[lkreidberg]

here are some suggested performance metrics to test how Eureka! is performing compared to other codes:

• plot of pointing stability over time (drift in x and y direction relative to first exposure)
• plot of background counts over time (median, or by wavelength channel?)
• from Kevin: MAD calculation that’s in Stage 3:

normspec = optspec / np.mean(optspec, axis=0)
for m in range(n_int):
        ediff[m] = 1e6 * np.median(np.abs(np.ediff1d(normspec[m])))
MAD = np.mean(ediff)

• rms of the light curve fits compared to the photon noise (broadband and spectroscopic light curves)
• Allan deviation plots for each wavelength channel to test for correlated noise (Kevin/I have a code snippet that can do this - ask one of us if you want it)
• goodness of fit for the lightcurves. No metric is perfect, so it would be great to have multiple options: chi2, BIC, AIC, and Bayesian evidence (this last is easy to get with dynesty, which should be v easy to implement now that Sebastian has emcee working)
• from Aarynn: posterior distributions of fit parameters
• from Aarynn: transmission spectrum