Does using ss3 opt instead of safe actually speed up model runs?

[k-doering-NOAA]

I just ran Simple locally with opt and safe versions of ss3 and found that ss safe actually runs a few seconds faster than opt:

SS safe speed (with hess): 1 min 31 s SS opt speed (with hess): 1 min 34 s

Note that it would be good to run beyond Simple to get a better idea of how much speed is gained by running with opt. Seems like none?

Test we could do:

• Run all the test-models on GitHub actions with safe and opt, compare the run times.
• Run simulations using ss3sim (say, run 100 models) and record the time it takes for the simulation.

If there is not much of a gain, potential paths forward are:

1. Work to speed up opt somehow
2. Stop compiling opt altogether so we have less binaries to maintain

Also, I thought there was an issue opened already about this, but couldn't find it. Feel free to add its issue number on this new issue and close the old one if found, as I don't think it is as specific.

[chantelwetzel-noaa]

Thanks for looking into this issue. I typically always run the safe executable. There is a potential "urban legend" story that I heard of someone using only the opt executable in their assessment and then a review identified quite a few bound issues during a review panel that weren't triggering warnings in the opt executable. I don't know if any of this is actually true and if it was, I could see people using the safe version during model development and then switching to the opt version when doing the final diagnostics (sensitivities, profile, and retrospective runs). This of course assumes a meaningful time savings are gained by using the opt version. I would suggest testing a more complex model to understand the potential time savings with more complex models an think sablefish model in the model test suite would a good one to try.

[k-doering-NOAA]

Thanks @chantelwetzel-noaa ! I have also always used safe, even for simulation studies (I think this is also true for @kellijohnson-NOAA and @iantaylor-NOAA )? You are right that testing a more complex model would be more meaningful!

[iantaylor-NOAA]

I haven't used opt in years. If I were doing a simulation analysis, I would consider it, but only if I could figure out a way to make sure I tested with safe after every change in the configuration.

More information on the speed differences would be surely useful to judge whether to maintain the fast executable and choose when it to use it. Including the slow models like Sablefish and Spinydogfish_2011 would be helpful too.

[k-doering-NOAA]

A user recently emailed with some convergence issues and we figured out that it was due to using the opt version. The model ran through (but slower) using safe.

This made me also realize that all our tests currently run just the safe version of SS3 and we don't do any testing of opt at all. It doesn't seem to be good practice to distribute something that we don't test!

All of this makes me wonder if we should even be compiling and distributing the opt version at all?

[iantaylor-NOAA]

@k-doering-NOAA, can you run the test models with ss_opt and compile the speed differences? That would seem like enough information to help judge whether it's worth continuing to distribute it.

Then again, the changes in ADMB 13.0 could shake up any such comparison, so maybe we should run the save vs opt test on ADMB 13.0.

[k-doering-NOAA]

Yes, that's a good point, I can set that up. I think I could set up both versions, using admb 13 and using current admb pretty easily.

[k-doering-NOAA]

Interesting results using SS3.30.19 and ADMB 13.0 - in most cases, it looks like the opt runs were not faster than the safe ones. Given we are transitioning toward admb 13.0, maybe this is good evidence for not continuing with opt? However, changes are still happening before the official admb 13.0 release to improve performance, so we may want to rerun this again after the official release.

Model	opt, ss3 v 3.30.19 and admb 13.0	safe, ss3 v 3.30.19 and admb 13.0
BigSkate_2019	0 hours, 3 minutes, 0 seconds.	0 hours, 2 minutes, 25 seconds.
Empirical_Wtatage_Age_Selex	0 hours, 0 minutes, 53 seconds.	0 hours, 0 minutes, 40 seconds.
growth_morphs	0 hours, 2 minutes, 35 seconds.	0 hours, 1 minutes, 58 seconds.
growth_timevary	0 hours, 0 minutes, 31 seconds.	0 hours, 0 minutes, 24 seconds.
Hake_2018	0 hours, 0 minutes, 43 seconds.	0 hours, 0 minutes, 29 seconds.
Hake_2019_semi-parametric_selex	0 hours, 0 minutes, 46 seconds.	0 hours, 0 minutes, 29 seconds.
KelpGreenling2015	0 hours, 2 minutes, 55 seconds.	0 hours, 2 minutes, 16 seconds.
Sablefish2015	0 hours, 22 minutes, 1 seconds.	0 hours, 16 minutes, 36 seconds.
Simple	0 hours, 1 minutes, 4 seconds.	0 hours, 0 minutes, 49 seconds.
Simple_Lorenzen_tv_trend	0 hours, 2 minutes, 0 seconds.	0 hours, 1 minutes, 34 seconds.
Simple_NoCPUE	0 hours, 0 minutes, 56 seconds.	0 hours, 0 minutes, 44 seconds.
Simple_with_Discard	0 hours, 1 minutes, 21 seconds.	0 hours, 1 minutes, 3 seconds.
Spinydogfish_2011	0 hours, 6 minutes, 7 seconds.	0 hours, 4 minutes, 56 seconds.
tagging_mirrored_sel	0 hours, 1 minutes, 44 seconds.	0 hours, 1 minutes, 22 seconds.
three_area_nomove	0 hours, 2 minutes, 2 seconds.	0 hours, 1 minutes, 38 seconds.
two_morph_seas_areas	0 hours, 4 minutes, 37 seconds.	0 hours, 3 minutes, 32 seconds.
vermillion_snapper	0 hours, 2 minutes, 7 seconds.	0 hours, 1 minutes, 31 seconds.
vermillion_snapper_F4	0 hours, 1 minutes, 55 seconds.	0 hours, 1 minutes, 21 seconds.

[k-doering-NOAA]

Also, I did run opt on our current main branch and admb 12.3, but it ran into some issues with doing the comparison (I suspect maybe because a model didn't finish properly?) I am still looking into it.

[k-doering-NOAA]

Not sure why the comparison failed (although I can spend more time looking, but we do see faster run times for opt when using it with admb 12.3 and the main branch:

model	opt, main branch, admb 12.3	safe, main branch, admb 12.3
BigSkate_2019	0 hours, 2 minutes, 15 seconds.	0 hours, 3 minutes, 8 seconds.
Empirical_Wtatage_Age_Selex	0 hours, 0 minutes, 32 seconds.	0 hours, 0 minutes, 48 seconds.
growth_morphs	0 hours, 1 minutes, 32 seconds.	0 hours, 2 minutes, 33 seconds.
growth_timevary	0 hours, 0 minutes, 20 seconds.	0 hours, 0 minutes, 27 seconds.
Hake_2018	0 hours, 0 minutes, 23 seconds.	0 hours, 0 minutes, 35 seconds.
Hake_2019_semi-parametric_selex	0 hours, 0 minutes, 25 seconds.	0 hours, 0 minutes, 40 seconds.
KelpGreenling2015	0 hours, 1 minutes, 57 seconds.	0 hours, 2 minutes, 43 seconds.
Sablefish2015	0 hours, 13 minutes, 56 seconds.	0 hours, 17 minutes, 31 seconds.
Simple	0 hours, 0 minutes, 40 seconds.	0 hours, 1 minutes, 0 seconds.
Simple_Lorenzen_tv_trend	0 hours, 1 minutes, 23 seconds.	0 hours, 1 minutes, 59 seconds.
Simple_NoCPUE	0 hours, 0 minutes, 37 seconds.	0 hours, 0 minutes, 53 seconds.
Simple_with_Discard	0 hours, 0 minutes, 50 seconds.	0 hours, 1 minutes, 13 seconds.
Spinydogfish_2011	0 hours, 4 minutes, 30 seconds.	0 hours, 6 minutes, 38 seconds.
tagging_mirrored_sel	0 hours, 1 minutes, 11 seconds.	0 hours, 1 minutes, 49 seconds.
three_area_nomove	0 hours, 1 minutes, 23 seconds.	0 hours, 2 minutes, 0 seconds.
two_morph_seas_areas	0 hours, 2 minutes, 35 seconds.	0 hours, 4 minutes, 0 seconds.
vermillion_snapper	0 hours, 1 minutes, 13 seconds.	0 hours, 1 minutes, 49 seconds.
vermillion_snapper_F4	0 hours, 1 minutes, 6 seconds.	0 hours, 1 minutes, 30 seconds.

[k-doering-NOAA]

I think this test of opt vs safe could be repeated when there is another test version of admb 13 to test, and should be run once more once admb 13 is release. This can be done on github actions by modifying the run-with-est workflow with safe (as is usual), then the run-with-est workflow with opt. The table is created by downloading the logs from these runs and using a find in file search for the model names (to get the names in order ) and then key workds for the run time, and doing some text splitting in excel before copying/pasting into a markdown table.

I'm happy to help with this as needed!

[Rick-Methot-NOAA]

Let's check the compile script. Possible that both are optimized and neither is in "safe" mode