FITB: answer is always a regex - can cause unexpected/broken behavior

[ascholerChemeketa]

If a test is correct because it is the first in a block it behaves different than if it is correct because is has correct=Yes

              <statement>
                <p>The second, implicitly correct fillin breaks the widget.</p>
                <p><fillin mode="string" answer="double\s+x;"/></p>
                <p><fillin mode="string" answer="double\s+x;"/></p>
              </statement>
              <evaluation>
                <evaluate>
                  <test correct="yes">
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
                <evaluate>
                  <test>
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
              </evaluation>
            </exercise>

            <exercise label="test2">
              <statement>
                <p>Neither of these ends up correctly matching "foo();".</p>
                <p><fillin mode="string" answer="foo();"/></p>
                <p><fillin mode="string" answer="foo();"/></p>
              </statement>
              <evaluation>
                <evaluate>
                  <test correct="yes">
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
                <evaluate>
                  <test>
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
              </evaluation>
            </exercise>

            <exercise label="test3">
              <statement>
                <p>The correct regex recipe depends on if correct="yes" is specified.</p>
                <p><fillin mode="string" answer="foo\(\);"/></p>
                <p><fillin mode="string" answer="foo\\(\\);"/></p>
              </statement>
              <evaluation>
                <evaluate>
                  <test correct="yes">
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
                <evaluate>
                  <test>
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
              </evaluation>
            </exercise>

[ascholerChemeketa]

Tag @dbrianwalton

Which behavior is the expected/correct one?

[dbrianwalton]

Thank you for finding these. I think that the issue here is about the role of the @answer and the role of the test. @answer is intended to be a literal string that represents the answer. It will be used directly for the correct answer in a static build.

I did not anticipate that the text of @answer would itself contain regex expressions.

If a regex is required to test the entry, then the explicit regex needs to be provided.

So, for the first question, we should have @answer="double x;" in the #fillin, but there should be a test using the more complete regex in the test and should not be using @use-answer.

<strcmp>double\s+x;</strcmp>

The second case is problematic and I'm not sure what to do because it is intuitively encoded correctly but the parentheses are breaking the regex. Maybe there should be some escaping going on if @use-answer is chosen.

I'm not sure what the third case correct answer is meant to be and what is being matched. Are we trying to match foo();? Or are we trying to match the regular expression that matches a response of foo();?

I don't immediately see why implicit or explicit @correct is making a difference.

[ascholerChemeketa]

The @answer vs test differentiation makes sense. In RST it was common (for me at least) to author all the answers as regexes. A clean answer with regex test does make more sense in terms of markup.

Yes, that does leave the issue that whatever string is provided gets built into a regex. So if you specify the @answer="foo(bar);" hoping to match exactly that string, and use @use-answer, what it actually will match is foobar; as the parens are treated as a regex group. Any other regex special symbols are going to not match as the literal string.

Maybe this would be best fixed at the source - the assumption by Runestone that every string pattern is a regex. i.e. adjust Runestone to have a "string" type in addition to "number" and "regex". It could use trim() to handle whitespace instead of the regex pattern \s*. There would need to be a way to communicate that to RS of course, but that feels cleaner than trying to properly escape the @answer so it works as expected when interpreted as a regex.

[ascholerChemeketa]

Renamed issue to match what it has evolved to...

@answer is always a regex, which is not clearly documented. Two issues result:

• Special characters can cause @use-answer to match unintended strings
• If the string provided is an invalid regex, there is a JS error when you try to grade it and nothing happens.

Sample breaking problem:

            <exercise label="invalid-regex">
              <statement>
                <p>How do you increment the variable x?</p>
                <p><fillin mode="string" answer="x++"/></p>
              </statement>
              <evaluation>
                <evaluate>
                  <test correct="yes">
                    <strcmp use-answer="yes"/>
                  </test>
                </evaluate>
              </evaluation>
            </exercise>