Failure mode to be avoided

[dtkaplan]

Recall that I implemented a variation on learnrhash and have been using it in a large course at the US Air Force Academy. The main difference is that my hash code gets constructed by monitoring events in the tutorial session.

After having nine tutorial assignments for each of 160 students, it's become apparent that there is a user behavior that triggers unanticipated and unwanted behavior by the overall system. The problem occurs when a student does not complete the entire tutorial in one session, perhaps because he or she went off to do something else or because the server failed, etc. Often, the student has not yet submitted the hash code to the course support software.

When the student returns to the tutorial in a new session, the previous answers have been cached in the browser and re-appear to the student. However, these previous answers have not triggered a learnr event in the current session and so aren't encompassed by the hash code. (I think, but am not certain, that this will also happen when the hash is being constructed by polling the tutorial state on the server.) The student submits the current session's hash code, understandably believing that this reflects that state appearing in the student's document, which includes events generated in a previous session. But the hash only reflects events in the current session. Result: the student claims that they have submitted all their answers but in reality only a few were represented in the hash code sent to the course support system.

For approximately 1/3 of my students, this has resulted in a serious undercount of their answers in each tutorial document. So it's a mainstream behavior for my students.

We tried to train the students to submit the hash code more often, but this did not alleviate the problem. Lacking a proper solution, I've tweaked the learnr code to change the behavior when a correct answer is submitted. The original behavior is to show the feedback message and no other control. In my tweaked version, a "Refresh submission" button appears under the feedback message. That way, a student can go through his or her completed tutorial in a new session and quickly transfer the browser-cached correct answers into the hash code for that session. See this tutorial for an example.

I'm posting this issue so that you can avoid this situation and, if you find a better solution, can tell me about it. Alternatively, I might make a pull request to {learnr} to implement the "Refresh submission" behavior as an option.

[rundel]

Hi Danny - that is very interesting and it isn't something I've seen yet. We haven't tried deploying this for students yet but will be in the next couple of weeks.

Most of our testing was with short delays with closing and reopening the browser on either local sessions or shinyapps.io deployed versions. I wonder if there is some amount of time that needs to pass before this information is lost. I'm assuming that the students are reporting that their earlier answers are showing as entered in the tutorial?

I've just deployed a test on shinyapps.io and I'll try leaving it overnight to see if I can recreate the issue.

[davidkane9]

We are using a version of this code in tutorials this semester here: https://github.com/davidkane9/PPBDS.data/

I forget the exact reason why we decided we could not use learnrhash directly. But, goodness knows, we have no interest in maintaining this code base ourselves.

We have 160+ students this semester also. In all my testing, I am always able to get my answers back even after restarting, even after re-installing the tutorial. But, there have been numerous student reports of interim work disappearing. Note that I am not sure if this has anything to do with learnrhash per se. These reports all involve cases of work disappearing before the hash is ever made. So, I suspect that is more a learnr issue than anything. Here is an example tutorial:

https://github.com/davidkane9/PPBDS.data/blob/master/inst/tutorials/01-visualization/tutorial.Rmd

Note that we always need to set options(tutorial.storage="local")

Hope this is helpful! learnrhash code is wonderful!

[rundel]

I haven't been able to recreate this in my limited testing - it does sound like it is more of an issue with how learnr is caching results. learnrhash is not doing anything beyond grabbing the state objects from learnr and encoding it so I would suspect that for some reason the state is not being restored properly.

I'll escalate this up the chain to the learnr people and see if they have anything to add.

[rundel]

@dtkaplan Two things to confirm quick:

1. Are you using learnr 0.10.1 or a development version? I think a lot of changes have happened in the latter recently and I haven't tested against them.

2. Are you sure that students are seeing their previous work in the restored sessions but these answers are missing from the hash? Based on my understanding of how learnr works it shouldn't be possible for their answers to be restored without the state objects being updated as well.

[dtkaplan]

1. I'm using a fork I took of learnr 0.10.1 three months ago. So it may well be that I'm missing a change that would be relevant to the situation.

2. The previous work is not appearing in the hash. I wouldn't expect it to, because I generate the hash not by reading the state objects but by monitoring the event stream.

I didn't understand that {learnr} is restoring objects from the client storage. Given that, perhaps I should modify my system to read that the objects rather than the events.

On Wed, Sep 9, 2020 at 6:04 AM Colin Rundel [email protected] wrote:

@dtkaplan https://github.com/dtkaplan Two things to confirm quick:

Are you using learnr 0.10.1 or a development version? I think a lot of changes have happened in the latter recently and I haven't tested against them. 2.

Are you sure that students are seeing their previous work in the restored sessions but these answers are missing from the hash? Based on my understanding of how learnr works it shouldn't be possible for their answers to be restored without the state objects being updated as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rundel/learnrhash/issues/6#issuecomment-689489412, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAECBLOB57YG4GJNYQLB44LSE5OK3ANCNFSM4RAJVKPA .

[rundel]

I don't think 0.10.1 would be an issue, I was more concerned that something had changed in a dev version.

I would need to go back an experiment with learnr again to see what is going on, when learnr replays the earlier answers does this trigger the same submissions events or not?