Reviewer 2 comments

[labarba]

Recommendation: Accept If Certain Minor Revisions Are Made

Comments: This is an interesting paper that raises a number of important issues while also telling a good story about the authors' experiences and frustrations, in a manner that is enjoyable to read as well as highly informative. So I think it will make a very good article for CiSE.

However, to some extent I think the authors conflate issues in a way that might be confusing or misleading for some readers.

• [x] The unsteady flow problem considered here is highly sensitive to perturbations, and the determination of a quantity such as the optimal angle of attack seems to be ill-conditioned. This means that small changes in the method, grid resolution, boundary conditions, etc. can have a significant effect on the results obtained. This is a classic issue in numerical analysis and fluid dynamics that has been a core subject of study for decades, and in some sense has little to do with the recent surge of interest in "reproducible research". Many of the issues raised in the paper regarding this sensitivity and the ways in which different numerical methods and implementations react to it are essential for anyone to be aware of who works on this type of problem, and should properly be considered as part of any original publication in this field, not only as an issue that arises when reproducing past work. I am sure the authors are aware of this, but this may not come across to readers from other fields. References are made to the extreme sensitivity of this particular CFD problem, e.g. in in the postmorten on p. 7 and the lessons learned on p. 9, but I think it would be valuable to discuss this in the introduction to the article.
• [x] My fear with the current presentation is that readers might get the impression that reproducibility in CFD is hopeless so they may as well not try, whereas the real lessons I think are that (a) this particular type of problem has sensitivity issues that one should be aware of before publishing any results -- don't believe what comes out of one code at one grid resolution without doing careful sensitivity studies and (b) this problem adds additional complication to trying to make ones own work reproducible (even by the same group with the same code later) due to changes in hardware or software dependencies.
• [x] And I think it should also be pointed out that (c) not all CFD problems are nearly this sensitive -- for other types of flow many of these problems would not arise and it would be much easier to reproduce/replicate results, and (d) even for this sensitive problem, it was invaluable that the original code was kept under version control and properly archive since this greatly facilitated this replication study, and allowed the determination of exactly what differences in software led to differences in newer results compared to those originally published. This behavior should be encouraged in the interest of advancing science.

It is also commendable that the code to support this study is all available on GitHub, although I have not attempted to repeat the experiments myself.

Additional Questions:

• How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section. : Very Relevant
• To what extent is this manuscript relevant to readers around the world? : The manuscript is of interest to readers throughout the world

1) Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical. : Points out many difficulties in trying to replicate/reproduce previous results obtained with computational fluid dynamics, both using the same code as the original study and using other software. There are many illustrations of pitfalls that can arise. These observations should be of interest to anyone doing CFD in a similar unsteady regime, whether or not they are interested in "reproducible research".

2) Is the manuscript technically sound? Please explain your answer in the Detailed Comments section. : Yes

3) What do you see as this manuscript's contribution to the literature in this field?: It contains an excellent summary of issues that are illustrated through a real-world example, and well described.

4) What do you see as the strongest aspect of this manuscript?: It raises awareness of many difficulties in doing such research and obtaining results one can trust, as well as difficulties in trying to replicate experiments years later, even with the same software.

5) What do you see as the weakest aspect of this manuscript?: As described further in m y detailed report, I think it should be pointed out that this unsteady problem is highly ill-conditioned and that not all CFD problems are so sensitive, and also that the issues raised in this paper go beyond "reproducible research" to how one properly does research on such problems.

1. Does the manuscript contain title, abstract, and/or keywords?: Yes
2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes
3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: References are sufficient and appropriate
4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Could be improved
5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Satisfactory
6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Satisfactory
7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory
8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Easy to read
9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of immediate and continuing interest to CiSE readers

Please rate the manuscript. Explain your choice in the Detailed Comments section.: Excellent

[labarba]

References are made to the extreme sensitivity of this particular CFD problem, e.g. in in the postmorten on p. 7 and the lessons learned on p. 9, but I think it would be valuable to discuss this in the introduction to the article.

• Addition to the introduction addresses this referee comment. See change history.

[labarba]

… for other types of flow many of these problems would not arise and it would be much easier to reproduce/replicate results

In response to this comment by the referee, we do think that most CFD problems of interest today are unsteady and present challenges like the ones we illustrate in the manuscript. Flow situations that present no challenge—e.g., laminar, steady, simple geometry—are routine and not very interesting!

Even for steady flow, if the Reynolds number is high, the replication challenge is steep. For example, the AIAA Drag Prediction Workshop has been ongoing since 2001, demonstrating wide differences between results with different (yet trusted) codes. (We added to the manuscript a mention and citation for the AIAA effort.)

[labarba]

the real lessons I think are that (a) this particular type of problem has sensitivity issues […] and (b) this problem adds additional complication to trying to make ones own work reproducible

We have added more discussion about the physical flow situation, mentioning that it is subject to various instabilities. Numerous flow situations of interest have these instabilities, and we agree with the referee that they add extra challenges to replication. But this is quite often the case in CFD. We don't see that it's possible to dissociate the sensitivity of the flow physics with the reproducibility challenges that they bring. The message of the paper is that replication and reproducibility in CFD can be hard—unless one has a very uninteresting flow.