Add functionality to check for failures

[richierocks]

At the moment, all the tests center around testing if the student got something right. What we don't have a good facility for at the moment is testing for specific things that the student got wrong.

Example exercise

The pre-exercise code defines a vector of coin tosses from a biased coin.

coin_tosses <- ifelse(rbinom(1000, 1, 0.75), "head", "tail")

The goal is to calculate the percentage of heads in the sample of coin tosses.

# Calculate the %age of heads
percentage_heads <- ___

Here's the correct solution.

# Calculate the %age of heads
percentage_heads <- 100 * mean(coin_tosses == "head")

A common wrong solution is to calculate the result as a fraction, forgetting to multiply by 100.

# Calculate the %age of heads
percentage_heads <- mean(coin_tosses == "head")

The easy SCT is to check for the right answer.

ex() %>% check_object("percentage_heads") %>% check_equal()

It's harder to give a feedback message for the specific case of forgetting to multiply by 100. The only way I can think to do it is to do a test_expression_result on ranges of bad numbers.

ex() %>% 
  check_expr('percentage_heads > 1') %>%
  check_results() %>% 
  check_equal(incorrect_msg = "`percentage_heads` is too small. Did you forget to multiply by 100 to convert to a percentage?")

This feels a little bit clunky. I'm not sure what the ideal syntax is, but I think it's something worth exploring.

[machow]

Once this is implemented, you should close this test_not issue and announce it on the #datacamp channel in all caps :)

https://github.com/datacamp/testwhat/issues/32

[ismayc]

This is similar to a comment I made recently. Shouldn’t we have an easy way to determine what the common incorrect answers are for any given exercise and have that drive the types of tests we should be running on that exercise? It seems we should be using this type of data collection to make our product better.

[ramnathv]

@richierocks One possible approach could be to define an additional set of check_xxx functions that can check the value of an object and return an appropriate message. For example, we could write something like:

ex() %>% 
  check_object("percentage_heads") %>% 
  check_equal() %>%
  check_gte(1, "`percentage_heads` is too small. Did you forget to multiply by 100?")

[filipsch]

All great ideas, but I'd be curious to know what the syntax is, and how to unambiguously define these failures. I think @ramnathv's suggestion is a good start, but the idea requires more fleshing out to see if it can work for most use cases.

[machow]

If we're taking the approach of directly testing the students code / environment, we can probably shed custom functions for each operation and just use R expressions. One note to highlight some of the challenges of this appreach is that in @ramnathv 's example code, the last two SCTs need to be switched (since if percentage_heads is equal to the solution and not greater than 1, then the solution fails; could wrap in a test_correct(...check_equal..., ...check_gte...)).

ex() %>%
    check_object("percentage_heads") %>%    # used for its convenient feedback message
    check_richies_cool_func(percentage_heads > 1, "`percentage_heads` is too small. Did you forget to multiply by 100?") %>%
    check_equal()

# I don't remember what check object returns, so used %T>%
ex() %T>% 
    check_object('x') %T>%
    check_object('y') %T>%
    check_richies_cool_func(x > y, "x should be greater than y")

In the second example, I used check_object to catch when a variable wasn't defined, but it's fairly straightforward to pull that from the error generated from running the expression (e.g. x > y when x doesn't exist). That might be a bit too black boxy though :p

@ismayc there is the diff-viewer. I think teach is supposed to display a students last submission when they give feedback. (It's a good point, but sounds like it may be a separate issue from this one)

[hermansje]

You can combine the functions override_solution_code to override the solution with a specific incorrect submission and check_equal(... eq_func= ...) with a negating custom equality function to do a results-based negative check.