Backwards compatibility discussion and issues

[James-Thorson-NOAA]

Based on discussion with @Cole-Monnahan-NOAA @ChristineStawitz-NOAA, @ericward-noaa, and others, I am opening a new issue regarding whether/how VAST should seek to maintain backwards compatibility, and also how to document changes that might affect this. Preceding discussion starts in #244 around here, but that issue was originally about a specific change affecting backwards compatibility upon upgrade to R version 4.0.0.

In response to recent comments from @ChristineStawitz-NOAA, I intend to start a new NEWS.pdf document listing changes between numbered releases.

In response to comments from @ChristineStawitz-NOAA and @Cole-Monnahan-NOAA, I will consider dropping code related to earlier features; this will likely involve some ongoing discussion before I decide exactly how to proceed.

In response to @ericward-noaa, I hope to learn more about Rocker etc.

Anyone with thoughts on backwards compatibility (including me in the future while mulling), feel free to post them here.

[kellijohnson-NOAA]

First, thank you @James-Thorson for trying to accommodate so many users and their specific needs. Given that, please don't take my opinion too much to heart because, after all, it is just an opinion.

Backwards compatibility from a "user" standpoint is not that great with VAST. The development of the interface is too dynamic to keep up. I have resorted to doing essentially what @ChristineStawitz-NOAA suggested to maintain downstream code where I reference specific commits for your various packages from a given point in time and use that for a year, then attempt to upgrade once the year is over. I need to know that my code is not going to break mid assessment season and the previous approach was the only way I thought I could ensure that.

Second, when I do go to upgrade, it is difficult to work though the examples / tests that are available because you use a mix of previous code with new code depending on what folder you are looking in. In my mind, this is where the beauty of version control comes in. Delete all of the old files and suggest users go back in the commit history to find them. I am on day 2 of trying to upgrade VASTWestCoast, which is my own fault, but I am still running into issues. The point here being that your desire to maintain backwards compatibility isn't really working on all fronts. I am fine with estimates changing as long as I know why and I think the change log is a great idea. I just have to be able to explain changes to the SSC, and if the changes result in a better, i.e, more statistically sound model, then by all means bring on the changes. Thank you for providing and maintaining VAST and I am happy to chat more offline if that would be helpful.

[James-Thorson-NOAA]

Thanks for the feedback @kellijohnson-NOAA. I appreciate feedback like this to for medium and long-term guidance about priorities.

To clarify a few things about my intent:

1. The Wiki examples are intended to demonstrate the current interface, and I think they all use the new fit_model interface. The integrated-test scripts by contrast use a mix of high-level or mid-level interface functions, to test that the prior mid-level interface still works.
2. I definitely agree about sticking with a particular release number during an application cycle, and updating as suits that cycle. I often save the ZIP associated with a numeric release, and save that in my working script so that I can ensure that I have a permanent copy associated with each project (in case that's helpful to know)

I definitely hope to tighten up my backwards compatibility testing based on this feedback.

[ChristineStawitz-NOAA]

Hi Jim,

Thanks for the guidance and commitment to testing. We've been using the new Github actions to check our RMAS package https://github.com/nmfs-fish-tools/rmas/actions?query=workflow%3AR-CMD-checkwith great success (well success on the testing but not always passing the test :) ) They have a couple of pre-set workflows for checking that your package installs properly across Windows, Mac, and Linux devices. I would start on implementing actions to check package install across the latest versions - it's pretty straightforward to create the YAML to do so and then every time you push to the repo, it will auto-check that all of the dependency and package loading is working properly. I know Kathryn also has a great workflow for testing SS using Jenkins/AWS which accomplishes the same thing but this is a bit easier to set up IMO and it doesn't require your own AWS instance (Github provides a cloud instance for you).

Let me know if you want more info on how to do this.

Thanks, Christine

On Fri, Aug 7, 2020 at 4:20 PM Jim Thorson [email protected] wrote:

Thanks for the feedback @kellijohnson-NOAA https://github.com/kellijohnson-NOAA. I appreciate feedback like this to for medium and long-term guidance about priorities.

To clarify a few things about my intent:

1. The Wiki examples are intended to demonstrate the current interface, and I think they all use the new fit_model interface. The integrated-test scripts by contrast use a mix of high-level or mid-level interface functions, to test that the prior mid-level interface still works.
2. I definitely agree about sticking with a particular release number during an application cycle, and updating as suits that cycle. I often save the ZIP associated with a numeric release, and save that in my working script so that I can ensure that I have a permanent copy associated with each project (in case that's helpful to know)

I definitely hope to tighten up my backwards compatibility testing based on this feedback.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/James-Thorson-NOAA/VAST/issues/245#issuecomment-670780877, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALNPO3JC4P22RWVG36J66ITR7SD2BANCNFSM4PX7QZYQ .

--

Christine C. Stawitz, PhD. (pronouns: she/her)

Stock Assessment Model Developer,

Contractor with ECS Federal in support of NOAA Fisheries Office of Science and Technology | U.S. Department of Commerce

Mobile: 206-617-2060

www.fisheries.noaa.gov