Unexpected model selection behaviour if models extinct early

[ljschumacher]

Bug description When testing model selection, history.get_model_probabilities() often only returns one (of two) model probabilities. This seems to be because the second model is "dying" early on, i.e. on or before the first run (during calibration?). Sometimes I get model probabilities for the first two populations, sometimes not.

Expected behavior I would expect either all model probabilities to always be returned, even if they are zero, or at least to do so for the first generation. Otherwise there is no way to tell if the inference was actually run on multiple models. Or is there a way to access extinct model probabilities "under the hood"? This is not clear from the documentation.

Environment pyabc 0.10.14 Python 3.9.4 Mac 10.15.7

[EmadAlamoudi]

Hi @ljschumacher , thanks for reporting that, do you have a toy example to show the behavior that you described?

[ljschumacher]

Not a minimal one right now I’m afraid… I’d be happy to share my code or try and trim it down, but I also noticed that I may have a mismatch in the dimensionality of my summary statistics (which is weirdly not throwing an error). I’ll try and fix that first in case that’s causing the problem, but it may be 2 months before I get to it.

On 12 Aug 2021, at 09:39, Emad Alamoudi @.***> wrote:

Hi @ljschumacher https://github.com/ljschumacher , thanks for reporting that, do you have a toy example to show the behavior that you described?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ICB-DCM/pyABC/issues/479#issuecomment-897455280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ75UGSXZYVKWZ47NUAR3DT4OCEVANCNFSM5B3VFTFQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[EmadAlamoudi]

ah, no worries, and please feel free to write here any update or if additional help is needed.

[ljschumacher]

after fixing my summary statistics error I can confirm that this doesn't seem to happen anymore. For completeness, I tested it with MultivariateNormal and Local(k=0.25) transition kernels.

I suspect something may be not quite right about the Local kernel implementation, as I get bigger effective sample sizes (by almost an order of magnitude) despite slightly lower acceptance fractions. Is this expected behaviour? (sorry, I realize I'm going off-topic here)

1. Local kernel run: INFO:Sampler:Parallelizing the sampling on 8 cores. INFO:History:Start <ABCSMC id=1, start_time=2021-09-30 21:00:56.482070> INFO:ABC:Calibration sample t=-1. INFO:Epsilon:initial epsilon is 40.27474661660461 INFO:ABC:t: 0, eps: 40.27474661660461. INFO:ABC:Acceptance rate: 4000 / 8076 = 4.9529e-01, ESS=4.0000e+03. INFO:ABC:t: 1, eps: 38.247117876294986. WARNING:pyabc.util:Unusually many (model, parameter) samples have prior density zero. The transition might be inappropriate. INFO:ABC:Acceptance rate: 4000 / 14590 = 2.7416e-01, ESS=1.7419e+03. INFO:ABC:t: 2, eps: 37.332669231396544. INFO:ABC:Acceptance rate: 4000 / 32535 = 1.2294e-01, ESS=1.7909e+03. INFO:ABC:t: 3, eps: 36.715068226185004. INFO:ABC:Acceptance rate: 4000 / 144071 = 2.7764e-02, ESS=1.8391e+03. INFO:pyabc.util:Stopping: maximum number of generations. INFO:History:Done <ABCSMC id=1, duration=1:22:50.839967, end_time=2021-09-30 22:23:47.322037> Model probabilities: m 0 1 t
   0 0.670500 0.329500 1 0.984903 0.015097 2 0.999904 0.000096 3 0.999997 0.000003

Ran in 4975.381s

2. MVN run: INFO:Sampler:Parallelizing the sampling on 6 cores. INFO:History:Start <ABCSMC id=2, start_time=2021-10-01 08:40:27.060829> INFO:ABC:Calibration sample t=-1. INFO:Epsilon:initial epsilon is 40.35282827484079 INFO:ABC:t: 0, eps: 40.35282827484079. INFO:ABC:Acceptance rate: 4000 / 7809 = 5.1223e-01, ESS=4.0000e+03. INFO:ABC:t: 1, eps: 38.31956669581477. INFO:ABC:Acceptance rate: 4000 / 14003 = 2.8565e-01, ESS=2.3371e+02. INFO:ABC:t: 2, eps: 37.380438434022935. INFO:ABC:Acceptance rate: 4000 / 27772 = 1.4403e-01, ESS=2.7229e+02. INFO:ABC:t: 3, eps: 36.713963499496316. INFO:ABC:Acceptance rate: 4000 / 107916 = 3.7066e-02, ESS=3.8676e+02. INFO:pyabc.util:Stopping: maximum number of generations. INFO:History:Done <ABCSMC id=2, duration=0:20:24.372630, end_time=2021-10-01 09:00:51.433459> Model probabilities: m 0 1 t
   0 0.646250 3.537500e-01 1 0.990464 9.536302e-03 2 0.999990 9.923872e-06 3 1.000000 2.763378e-13

Ran in 1226.469s

The good news is that the results are qualitatively similar. I'm also surprised how much quicker the MVN is here, which is contrary to previous experience, but that was an older pyabc version, and also this wasn't a careful speed test. Mainly I was just confused about the ESS outputs.

[yannikschaelte]

good to hear more works now first of all.

I suspect something may be not quite right about the Local kernel implementation, as I get bigger effective sample sizes (by almost an order of magnitude) despite slightly lower acceptance fractions. Is this expected behaviour? (sorry, I realize I'm going off-topic here)

In most applications we used MVN, so it is certainly not entirely impossible something is wrong about the implementation of Local, although I remember it working well on test problems. The ESS calculation is based on the ratio prior / proposal (see e.g. here), so the proposal does affect it, however one would expect it to do so rather via a different choice of samples. If a local kernel represents the distribution better and thus gives more tailored samples covering the distribution more evenly, one could indeed expect the ESS to be higher compared to MVN. There can however also be other reasons, not least of all just random chance.

The good news is that the results are qualitatively similar. I'm also surprised how much quicker the MVN is here, which is contrary to previous experience, but that was an older pyabc version, and also this wasn't a careful speed test. Mainly I was just confused about the ESS outputs.

The local kernel may be calculating a substantial number of covariance matrices, as you have a relatively large population size, which may indeed be cost-intensive and could be circumvented by kernels with less localization, e.g. only using up to a fixed number of covariance matrices. To check whether this is what takes time, logging the time spent in fit() is something we should implement in pyabc anyway.