Standalone autograders, autodetecting file names, package, etc

[dbp]

For whatever reason, I've found it difficult to wrap my head around making autograders with this: I think it's the copy repo, edit files, etc, workflow that for whatever reason is confusing me.

Working backwards, I started from wanting a single file to be an autograder for an assignment. They live in the repo for the course, and so should be self-contained, depending on the gradescope code in a library (and depending on shared pre-built docker images seems as bad / worse, given how poorly they seem to be maintained as infrastructure).

This is easy --- if you move the lib-grade.rkt code into a package (which I tentatively named autogradescope, though haven't published, as I wanted to open this issue first):

#!/usr/bin/env racket
#lang racket
(require autogradescope)

(require rackunit) 
...
;; (define-var ...
;; (define-test-suite ...

The next thing that caused trouble for me was the hardcoded file names -- students would often submit slight variations (capitalized, etc), and while one could say this is a learning opportunity, it doesn't strike me as an important one. So, rather than (define-var foo from "file.rkt"), I changed it so that it finds whatever file in the submission directory has the right extension (which is configurable, but defaults to .rkt). If students are submitting multiple files, obviously this doesn't work (and perhaps that was the original motivation for the current design?).

(set-submission-extension! ".rkt") ; this is the default, so not actually needed.
(define-var my-function) ; these are found in the first file in the submission with the correct extension. 
(define-var other-function)    
...

The last thing that I changed was how testing of the autograders is done (i.e., on our own computers, before we send them to Gradescope). While there is certainly value in running on the same image (or, hopefully the same image) that runs on Gradescope, I suspect most of us rely on the fact that Racket 8.9 is Racket 8.9, across platforms (as, indeed, we need our testing to match what our students are doing), and so running natively should be "good enough", and can be a lot lighter weight: an environment variable can hardcode the path to the file where definitions are loaded from (and, when it is present, the results are printed to stdout):

$ SUBMISSION=./reference-soln.rkt ./run_autograder
{"score":"96.42857142857143","tests":[{"output":"Execution error in test named «blah»"}]}

Obviously, these are a bunch of changes, and some actively conflict with how this library currently works. At the same time, they aren't mutually exclusive (e.g., there can easily be two forms of define-var), and if you were interested in turning this into a library (to support my first goal: having the autograders be single files, not depending on prebuilt docker images), I could certainly do a more careful job merging. On the other hand, I'm also fine with having a fork (which is why I haven't called the library gradescope-racket). One thing though: could you stick a license on this code? :)