Do you use automated testing/grading for assignments?

[afeld]

Do you use any autograder or automated testing with with students? Do they write the tests, or do you? Are they able to see the tests or the output? What software do you use to run them?

[hpssjellis]

Generally I don't trust the students to do any real online testing. They creatively cheat even when I am watching. I have however spent a ton of time making an online Physics 11 pratice exam at http://142.33.222.23/ellis/cpTests/ph11Exam.html which randomly sets questions and gives answers at the bottom of the page.

[afeld]

Sorry, I could have been more clear: I meant automated tests for their code (of the JUnit, RSpec, QUnit, etc. variety).

[hpssjellis]

Oops. Can you delete my post if you think it is not relevant. I don't use browser testers since I compile a working chrome javascript page into an App using Phonegap.com. If it doesn't work then something is wrong.

On Thu, Jun 19, 2014 at 8:43 PM, Aidan Feldman [email protected] wrote:

Sorry, I could have been more clear: I meant automated tests for their code (of the JUnit, RSpec, QUnit, etc. variety).

— Reply to this email directly or view it on GitHub https://github.com/education/teachers/issues/3#issuecomment-46634211.

[blake41]

We have automated tests for almost all our assignments. In some cases as the course progresses they write their own tests. I'd like them to write more of their own tests but the automated testing allows them to get immediate feedback.

[georgeyiu]

We use JUnit for much of our Java testing. We provide a subset of our autograder suite to the students at the beginning of the project and require that they extend that suite and write more tests. If the project is well spec'ed, you can incorporate some of the better student tests into the autograder suite for future terms in the course. Another great method of encouraging testing is writing a version of the master solution code that includes several bugs and requiring students to write tests to break them (blindly, without telling them the bugs of course).

[alexjacoby]

I'd love to hear if/how anyone is using webhooks to automate testing of student submissions. I've skimmed the documentation for the automated testing services but I'd love to hear what people think works best for this situation.

[delucas]

Hey @alexjacoby! I'm using a custom tool (a daemon that pulls homework from a queue, which is populated when my students make a push into their GitHub repositories with the answer). It pulls the repo and then runs a series of automated tests (JUnit, PMD, Checkstyle and Cobertura). Then it takes the output and converts it into a unified score. What do you think to use? Travis-CI seems good if you have a nice custom script that gets a score out of the test results.

[afeld]

Should have mentioned before, but there's a section of the classroom guide about automated testing, as well as a description of how I'm using Travis etc. in one of my classes.

[neurodude]

Any Idea how to integrate travis-ci with unit testing for the student submissions. I have been trying to work out how to run the unit tests without giving them the unit tests to begin with. Any thoughts?

[afeld]

I don't know of any way to do that through Travis – would probably need to set up something custom if hiding the tests is important to you.

[neurodude]

In the classroom guide it mentions auto grading. In that context it says:

Unit tests

Tests can be written by you, and/or by the students
Hidden from the student (a.k.a. an "autograder") or not

Any idea what they were referring to or how to set that up?

[afeld]

Right. The basic idea is that you need (private) scripts run when the code is submitted, which can be triggered via web hooks. https://travis-ci.org/ can't be used because there's no way to upload the hidden tests, but any of those other options (hosted by you, at least) should be able to support it. I don't know of any specific guides for setting something up w/ GitHub, but searching for "set up autograder" had a bunch of other examples.

@armandofox Saw your blog post about autograding at scale... what did you end up doing?

[neurodude]

How would I do it if I was ok with them seeing the tests? I have now a gradle build building and testing junits. Do I have to post the whole file structure? Can I just put the tests somewhere and have them run?

Also, It appears that google is different for you and me. I don't get other examples when I search "set up autograder"?

[efthym]

Hi All,

I'm preparing to use github for a University course following the sandbox approach. I got myself fairly familiar with teachers_pet and it worked fine for me so far.

Now, I would like to add the capability of running Travis-CI when students push their work to their (private) repos so that an automatic script in Ruby checks their work and they get to know if their submission meets at least basic requirements. I'm happy to give the script, which contains the expected results, to the students, but I plan to run other tests and visually inspect their work.

I know teachers_pet push_files copies everything from a local repo to multiple student repositories so I can get .travis.yml files into their repositories. However, I'd like them to be able to see the passing/failing "badge" in their own README.md files, so they can instantly know if their work is OK. Unfortunately, these badges need URLs which contain unique tokens for private repositories... Is there an easy way to customise each student's README.md file so it contains their own Travis-CI token? (assuming there's a way of getting it automatically).

Do you have any other suggestions for showing build status/automarker result in a private student repo?

Thanks,

Aris

[motoxman36]

You guys might be interested in this...

https://www.gradetrain.com/blog/teaching-programming-youre-doing-it-wrong

[laprej]

So if I want my students to use Travis while using the sandbox approach, I should have them manually enable it on each repo, correct?

[sre42]

Hi Aris @efthym Were you able to figure out your problem? As of now, Travis CI has granted educational access to my class repository, so students can head over to travis-ci.com and run tests on the private repository.

@laprej Yes I believe students will have to log in into Travis-ci.com and enable the build for each repository.

[sre42]

As of now, I am doing the following(for my advanced C++ class):

• Use GitHub classroom to create individual private repositories
• These repositories have CMake and Google's Test library incorporated, so the students can use CLion to test locally.
• The Students can also head to Travis-ci.com and enable cloud testing for themselves.
• My TA's can see the list of all the private repositories and hence see if they pass or fail.
• Here is an example of a C++ factorial program that uses Google test and Travis CI:

https://github.com/csc340-03-spring-2016/factorials-cpp-gtest-travisci.git

Still to figure out:

• Grade within Travis to determine score based on number of C++ tests passed
• Use Travis to clone and diff the test folder to determine plagiarism(changes to the test folder)
• Hide certain tests.

[efthym]

Hi Omar @Omarasifshaikh No I haven't done anything on this front yet. I think @laprej is right, though I'm not sure if it requires that students get the student pack, so they can have private repos of their own on GitHub.