timout function for check does not work when code gets stuck in try/catch loop

[mjdv]

The assignment I am working on involves students writing a python file containing classes, which should not run any code when imported. (That is, all code except definitions should be within if __name__ == "__main__".) I therefore want to test whether or not importing the student's code is a process that terminates within two seconds.

I use the following Python check:

import check50
import os
import sys

@check50.check()
def exists():
    """Check that hangman.py exists."""
    check50.exists("hangman.py")

@check50.check(exists, timeout=3)
def can_import():
    """
    Checking that you can import the file without it hanging.
    """
    sys.path.append(os.getcwd())
    import hangman

When the student fails this test (i.e. has written non-definition code), and is using a try/catch block to catch faulty input from the user, the timeout no longer works. Consider the following example hangman.py:

from cs50 import get_int

while True:
    try:
        n = get_int("What is the length of the word? ")
        assert n > 0, "Word length needs to be positive."
    except Exception as e:
        pass

When running the check on this piece of code, it runs (apparently) forever.

Note that the problem really is with try/catch blocks. E.g., the following code is killed succesfully after 3 seconds (although it does print the get_int message repeatedly to stdout).

from cs50 import get_int

n = None
while True:
    n = get_int("What is the length of the word? ")

[cmlsharp]

I really, really don't recommend importing student code directly into your test. Technically, we provide an API function for this (check50.py.import_ which is basically a wrapper around what you did with sys.path) but it really isn't recommended to be used on "untrusted" code for a number of reasons. For starters, it lets the student run arbitrary code within the check50 process which can lead to all kinds of bad/weird things. For example, a student could, in principle, write a piece of code that, once imported, caused the check to immediately pass (though it would require some sophistication). Moreover, if exceptions are thrown by the code you import, that won't register with check50 as a failure. Instead, it will register as an internal error on check50's part (displaying the :| rather than the :( face and telling students to submit a bug report). check50.py.import_ instead of import directly does fix this latter issue though.

The "right" way to do what you are trying to do here is to use the run interface to start a new process.

@check50.check(exists)
def can_import():
    check50.run("python3 -c 'import hangman'").exit(0, timeout=3) # timeout is optional here, defaults to 5 seconds

That said, it is disconcerting that even a "wrong" check can make check50 run forever, so this is still worth investigating. Thank you for the report!