unsupported test module used in section 6.3

[powderflask]

Section 6.3 provides an example with the following code:

   import test
   print('testing square function')
   test.testEqual(square(10), 100)

(https://github.com/RunestoneInteractive/thinkcspy/blob/master/_sources/Functions/UnitTesting.rst)

Python documentation on using the test module is quite clear:

The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.

https://docs.python.org/3/library/test.html

This code sample does not work in recent Python release (e.g., 3.5 +), causing confusion for students who try to run it in other environments.

The Code Lens on this example also crashes with: ImportError: test not found or not supported

Thoughts on resolving issue: The unittest framework uses classes, and so is not well-suited at this introductory level.

I recommend either:

1. don't introduce a testing framework at this point - introduce the concept of unit testing with a conditional or assert statement.
   Issue: conditional statement has not been introduced yet.
2. introduce doctest in this section, and introduce unittest framework later. https://docs.python.org/3/library/doctest.html

I'm happy to make a PR for this, but wanting some direction and advice from core team before working it up.

[bnmnetp]

The purpose of us creating our own test module is to address the problems with unittest that you outline. As well as the fact that skulpt (the javascript implementation of Python that we use in the browser) does not support doctests.

You can't please everyone with one textbook, and there are plenty of people who have suggested we should introduce unit testing even sooner than we do.

I might be open to renaming test to something else, or perhaps implementing a baby version of pytest -- which does use assert. But it would require more discussion with other authors and contributors.

Brad

Tagging @presnick and @bjones1 for further input

[powderflask]

Ahhhh! Ok. My confusion is simply down to name overloading with private, internal-use Python module also named test. Got it. This makes sense, and no trouble for me to talk around, but wonder if it might avoid future confusion to simply introduce unit tests using a simple assert instead: assert square(10) == 100, "Expected square(10) to return 100, got: %s", square(10) then introduce more robust unit testing tools later.

Thanks Brad. BTW - you may not be able to please everyone with one text, but this text certainly pleases me and every student I've ever taught with it. Awesome.
Just looking for ways to make it even better :-)

[bnmnetp]

Thanks!

I think the move to simple assert statements would be good as it would introduce a standard Python construct. The downside is that the tests would stop after one failure.

[bjones1]

I'm in favor of simple assert statements. For beginners, I think that's a reasonable compromise vs. learning to use a test framework.

[powderflask]

I piloted using simply assert statements instead of unit tests and found it worked perfectly and had several additional benefit:

• students achieved much higher comprehension of Boolean expressions, and were much more likely to apply Boolean expressions without redundancy (e.g., return <Boolean expression> instead of using conditional return and less likely to write things like: if <Boolean expression> == True)
• students had fewer hurdles understanding subsequent use of assert statements to enforce function pre-conditions
• I was able to use assert statements in other circumstances, e.g., to help students reason about more complex algorithms
• I was able to use assert statements as specifications for pure functions (e.g. TDD) and as a result able to remove ambiguity from lab exercises and exam questions. Students seemed to really appreciate and respond to the clarity of these specifications.
• On quizzes and exams I had students complete an assert statement when tracing code for a particular input, thus removing any ambiguity from the question and their responses.

I am sold on this approach, and more than happy to take on the task of replacing the custom unit test framework with asset statements throughout text. Following that, I would like to add some asserts to various places / exercises where they will clarify / remove ambiguity (as a separate PR).

Please confirm this is a useful direction - I will have time to work on this in the coming months. Thanks once again for the true gift that is this book - I've never received so much overwhelming positive student feedback about a textbook. Just awesome.

[presnick]

I’ve been mostly using assert this semester as well and have liked it, for some of the same reasons described here (especially exams).

Another advantage is that you can assert things other than equality of two values, which is all we have with test.testEqual.

I think the big change would be to rewrite the test cases chapter using assert, possibly trying to write it in a modular way so that some sections could be introduced before functions if an instructor wanted to do that.

If you want to take a crack at it, I’m willing to collaborate on it with you over the summer.

On Sun, Apr 7, 2019 at 5:11 PM Joseph [email protected] wrote:

I piloted using simply assert statements instead of unit tests and found it worked perfectly and had several additional benefit:

• students achieved much higher comprehension of Boolean expressions, and were much more likely to apply Boolean expressions without redundancy (e.g., return <Boolean expression> instead of using conditional return and less likely to write things like: if <Boolean expression> == True)
• students had fewer hurdles understanding subsequent use of assert statements to enforce function pre-conditions
• I was able to use assert statements in other circumstances, e.g., to help students reason about more complex algorithms
• I was able to use assert statements as specifications for pure functions (e.g. TDD) and as a result able to remove ambiguity from lab exercises and exam questions. Students seemed to really appreciate and respond to the clarity of these specifications.
• On quizzes and exams I had students complete an assert statement when tracing code for a particular input, thus removing any ambiguity from the question and their responses.

I am sold on this approach, and more than happy to take on the task of replacing the custom unit test framework with asset statements throughout text. Following that, I would like to add some asserts to various places / exercises where they will clarify / remove ambiguity (as a separate PR).

Please confirm this is a useful direction - I will have time to work on this in the coming months. Thanks once again for the true gift that is this book - I've never received so much overwhelming positive student feedback about a textbook. Just awesome.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/thinkcspy/issues/101#issuecomment-480630078, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBvzoMVTMVgYzkytkOW5aFvz-XoAvPPks5vel8RgaJpZM4Z8ybC .

[bjones1]

Can skulpt run pytest? That's a great framework, and it uses assert for all the test cases. Perhaps that would work nicely? (We also use pytest on the server for unit testing).

[presnick]

I think for fopp just assert is good. We could have a supplemental chapter on collecting and running tests with pytest.

On Sun, Apr 7, 2019 at 9:27 PM Bryan A. Jones [email protected] wrote:

Can skulpt run pytest? That's a great framework, and it uses assert for all the test cases. Perhaps that would work nicely? (We also use pytest on the server for unit testing).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/thinkcspy/issues/101#issuecomment-480653176, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBvznxuBTJKLf8nrAhkKl29Wkj87W_Lks5vepsRgaJpZM4Z8ybC .

[bjones1]

Good point. Simple asserts are nice.

[bnmnetp]

I'm always happy to have help @powderflask ! And thanks for the kind comments, it makes the work worth the effort to know we are having an impact.

I want to be clear about what you are proposing to do. Are you proposing to replace all of the current uses of the unittest.gui module that are used to autograde problems in the Exercises subchapters? Or are you talking about chapter 18 and replacing the use of the test module in that chapter?

One good thing about the unittest module is that it does connect to the backend in a way that relieves the grading pressure on instructors, and I would hate to lose that. In fact I had hoped to find some help this summer to write unittests for more of the Exercises.

[bnmnetp]

@bjones1 as far as I know, nobody has ported pytest to skulpt. I did the unittest port myself and since it was written in pure python it was not too hard. skulpt does support decorators and assert so it might not be too hard.

[presnick]

I was assuming this was a rewrite of chapter 18, not a change to how we implement autograding...is that right?

On Mon, Apr 8, 2019 at 10:09 AM Bradley Miller [email protected] wrote:

I'm always happy to have help @powderflask https://github.com/powderflask ! And thanks for the kind comments, it makes the work worth the effort to know we are having an impact.

I want to be clear about what you are proposing to do. Are you proposing to replace all of the current uses of the unittest.gui module that are used to autograde problems in the Exercises subchapters? Or are you talking about chapter 18 and replacing the use of the test module in that chapter?

One good thing about the unittest module is that it does connect to the backend in a way that relieves the grading pressure on instructors, and I would hate to lose that. In fact I had hoped to find some help this summer to write unittests for more of the Exercises.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/thinkcspy/issues/101#issuecomment-480847527, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBvzolkQAZjYGtRsKtki9OAYbAUmdg-ks5ve021gaJpZM4Z8ybC .

[powderflask]

Yes - sorry for confusion. I'm proposing only to change student-facing content. For example, content in 6.3, 7.8. I have not yet done an exhaustive search, but recall seeing it used in other places too. I'm thinking 6.3 should introduce the idea of a more robust unit testing framework, and then show how the humble assert statement will serve us for now. Forgive me for ignorance, but I'm not sure what "Ch. 18" is -- is that the Labs chapter?

Once those changes are made, I will propose to add assert unit tests to some of the examples and exercises (as a form of specification) throughout the text. That is a separate proposal, but also deals only with student-facing content, not changes to how testing is done on the backend.

[bnmnetp]

Great, thanks for the clarification. And now I can clear up some more confusion.... we are actually talking about two different books.

https://runestone.academy/runestone/static/fopp/index.html is what Paul and I thought you were talking about. It is our "successor" to the thinkcspy book, which many many schools still use. Both would benefit from your proposal.

In the fopp book it is Chapter 18, which is an evolution/combination of 6.3 and 7.8 .

Brad

[constantinol]

Pull request #167 uses assert method instead of the previous test module framework.