Autograding

[andrewbredow]

The initial plan for this will be two parts:

• [x] Automated setup and feedback
• [x] Automated test runs

The initial target will use GitHub Actions and will involve a custom workflow.

[jeffrafter]

Autofeedback Example

The autofeedback example consists of three parts:

1. A template repository for the assignment which includes:
   • The assignment README
   • Automated tests for the assignment
   • A check workflow for running the automated tests
   • A setup workflow for setting up the template when used
2. A setup action that:
   • Creates a new commit (using template files)
   • Creates a branch for the new commit
   • Opens a pull request (using the PULL_REQUEST.md template)
3. A student repository (built from the template)
   • On each new push automated tests are run

Template repository

Here's a quick explanation of all of the files:

File	Description
README.md	Contains a description of the assignment, instructions for getting started and links to the pull request
add.test.js	An automated test that will pass or fail based on the code the student writes
jest.config.js	The automated test configuration
package.json	The package.json in this case is only used for running the tests and installing the dependencies for this assignment (in this case there are no dependencies other than Jest for running the test)
package-lock.json	see package.json
.gitignore	General .gitignore, most importantly there are entries for node_modules and add.js The entry for add.js is removed when setting up the template (see .github/template/files/.gitignore below)
.github/workflows/check.yml	This workflow runs the automated tests
.github/workflows/setup.yml	This workflow only runs once, on the first push of the newly created repository. It automatically generates a commit, a new branch, and a pull request for that commit so that students can begin working immediately
.github/template/PULL_REQUEST.md	The template for the pull request that will be opened. This let's teachers get as creative as they want
.github/template/files/add.js	All of the files in this files directory (including add.js will be used to create the commit for the pull request branch.
.github/template/files/.gitignore	This .gitignore is the same as the main .gitignore except it allows add.js to be added to the repo.

There are a lot of pieces for creating a template here. It would probably be helpful to put together a generic "assignment" template repository for teachers (in different languages).

Try it out as a student

To get the student experience (short of creating a full assignment and accepting it) follow these steps:

• Go to the addition assignment example and click Use this template.

• Name the repository example-addition-assignment then click Create repository from template

• The repository will be generated, wait about 15 seconds and the Actions will kick off.

• After about 30 seconds a pull-request will automatically be generated (you might need to refresh to see it in the Pull requests tab).

• Work on your homework right in the blob editor by viewing the changes in the pull request and choosing to Edit.

• When you save your changes a new commit is created and the automated tests are run.

The setup action

If you want to see the setup action: check it out

TODO

• The .gitignore trick is weird
• Verify the notifications
• ~~Don't run Check on master~~ Fixed
• ~~Pull requests tab doesn't auto-refresh~~ Coming soon from repos
• ~~Draft pull request isn't working~~ Fixed