Exams: Create them, take them, view reports

[nucleogenesis]

Summary

I've tried my best to sort of fold together the existing data architecture code for Coach quiz reports and Learner quiz reports with the updated data structures.

I've kept front-end architecture shaped the same, but updated the shape of the data, basically testing out the reports and such, finding bugs, then fixing them. The changes were in relatively self-contained / purpose-built modules -- that is to say that I've tried to be sure that there are no chances of regressions.

The exam.utils module has been significantly updated. Now there are functions going from each data_model_version to the next (ie, v0-v1, v1-v2, etc).

There appear to be two things needed from this module at this time:

• Convert an exam to the latest version
• Fetch all of the exercises for the exam

Now there is a general function for the first part and the second function does the conversion and fetches the nodes.

References

Closes #12097

Reviewer guidance

Backward compatibility

A big part of this change involves ensuring backward compatibility. There are automated tests that do this, but I've not tested with any "data model version" below 2.

Seems like to get an exam < v2 you will need to create a quiz using Kolibri <=0.11 as V2 seems to have been added into 0.12.

• Create like 5 quizzes in Kolibri <= 0.11
• Take a few of the quizzes with multiple learners (but also leave at least one untouched) -- also, be sure to get some wrong for a couple users
• Start this PR's assets on the same KOLIBRI_HOME (stopping the old one first)

Here you should be able to do the following:

• View exam reports for all exams that have been taken, for all users
• View specific questions' reports in Coach
• View "difficult questions" list in Coach reports
• View quiz report as the learner
• Take the quiz if it has not been taken previously by a user

[github-actions[bot]]

Build Artifacts

Asset type	Download link
PEX file	kolibri-.pex
Windows Installer (EXE)	kolibri-0.16.2b1.dev0+git.1057.g064993bc-windows-setup-unsigned.exe
Debian Package	kolibri_0.16.2b1.dev0+git.1057.g064993bc-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.16.2b1.dev0+git.1057.g064993bc-0.4.1.dmg
Android Package (APK)	kolibri-0.16.2b1.dev0+git.1057.g064993bc-0.1.2-debug.apk
TAR file	kolibri-0.16.2b1.dev0+git.1057.g064993bc.tar.gz
WHL file	kolibri-0.16.2b1.dev0+git.1057.g064993bc-py2.py3-none-any.whl

[radinamatic]

Tested the latest EXE asset on Windows 10, using Firefox and Edge for coach and learners, respectively.

I can appreciate that the backward compatibility was the main scope here, and I had planned to test it upgrading from a previous version, but several issues were present even out-of-the-box, that made more sense to report them first, before falling down the upgrade rabbit hole.

1. Coach can create and save the quiz, but cannot re-open it to edit questions/resources (just the quiz title and assignees, same as in previous versions)

2. Expand all and Collapse all buttons are not working, and neither is opening the accordion panel to preview questions, neither with mouse nor with keyboard. I can see the TypeError: _vm.activeResourceMap[question.exercise_id] is undefined error in the console.

3. Learner can start the quiz, but they cannot complete it, the Submit button is nowhere to be seen. For the same reason they cannot see the quiz report.

   

   And no idea where the 👻 100 additional questions are coming from.

4. Coach can see the reports, but they all report learners have started as it's not possible to complete (see nº 3.).

5. Difficult questions are not displayed. I had 3 learners give incorrect answers to same 6 questions.

   

I have also some keyboard navigation specific issues, will report the summary later. It might be better to address these issues first, before testing the backward compatibility.

db-logs.zip

[rtibbles]

cannot re-open it to edit questions/resources

This continues to be intentional, although we could potentially add some leeway here in the case that the quiz has never been activated. The issues arise when a coach edits a quiz when students have already taken it, as without some sort of version control in place, we would not be able to properly parse responses from older versions.

[marcellamaki]

add some leeway here in the case that the quiz has never been activated

this seems like it could be useful, as with the new editing options, being able to edit/revise the quiz (before made visible to learners), seems more like expected behavior than with our current quiz selection/creation flow. But I don't know that it necessarily needs to be MVP. If it's not though, maybe some helper text at "save" would be useful.

[radinamatic]

Ditto to what Marcella said: we're adding more complex (enhanced) features to quiz creation, but we're not offering any options for revision and corrections? 🤔

Agreed that the limitation needs to be before the quiz is made visible to learners, but I can only imagine the frustration of the coach who invested time and effort to carefully craft the sections, add instructions and descriptions, select and order questions, and then they have to do it all over from scratch just because they discovered a typo after saving... 😭

Not to mention that error prevention is part of the WCAG 😉

[radinamatic]

At a minimum if we decide not to include it in the MVP, we must add the Please review carefully the quiz as you will not be able to edit it after saving modal, to warn the users, but honestly that just looks... well, just ...unpolished (to put it mildly). 😞

[nucleogenesis]

Users can duplicate a quiz, so in that case I think it makes even more sense to include the ability to "re-open" a quiz for editing within the quiz creation tool.

I think that in the medium-term it'd be ideal for us to revise some of the UX around this, but in the short term I think that we could get @tomiwaoLE 's thoughts on how/where to put the button / option to edit a quiz.

I think maybe having an "Edit quiz" button next to / near the "Start quiz" button might be worth considering along with adding text to the "Start quiz" modal (where we say 'this will make learners download 1234kb') to say "You will no longer be able to edit this quiz" or something?

[radinamatic]

(continued)

6. That quiz with 107 questions kept throwing error when the learner inputs the answer on the last questions and clicks the Next button (or presses Enter when navigating by keyboard).

   

Keyboard navigation issues

• When focusing to select the Edit section option and pressing Enter, the drop-down remains visible even after the side panel with Section settings is opened.

  

• Number of questions field works perfectly, the user can type and increase/decrease the number with arrow keys 💯 However, if they navigate further and focus the + button, it goes wild and each Enter key press adds (or concatenates) a number 1 to the number of questions... 🤪

  

  I did delete all those 1's to be able to proceed, but could that be why the quiz in question had 100 👻 questions?

• When focusing the Change resources button, pressing Enter and selecting resources to perform the change, when the work in that side panel is completed, whether user presses the focused Save changes button, or they decide to abandon and focus the Back ⬅️ button, the focus should return to Change resources button that initiated that whole workflow. As of now, the Close ✖️ button in the upper right corner is focused Instead.

• By the same logic, if the Options button opens the Section settings, when that side panel is closed, the focus should return to the Options button.

• The Section order section inside Section settings 🙃 is not keyboard navigable: the focus jumps right over it and lands on the Delete section button.

  

• The 🔽 and 🔼 buttons to reorder questions by keyboard are focusable, but they do not work, nothing happens after pressing either Enter or Space. Not sure if that is because the whole accordion is not working in this asset...

[nucleogenesis]

@radinamatic

I've fixed the accordion expand/collapse issue.

I also fixed the issue where you ended up with 107 questions when it was supposed to be 17... Javascript is fun because it ended up that rather than trying to add 10 + 7 (two numbers) it was instead adding 10 + "7" (a number and a string) which results in forcing the number 10 into a string "10" then adding a "7" to it for "107"... so now I've made sure it always is working with numbers 😅

You should be able now to create a quiz and test it. I'm honestly not super sure exactly why the number was being passed around as a string rather than a number -- like, I feel I should have noticed this with all of the quizzes I've been taking. I think that maybe it was to do with whether or not you click the +/- buttons to change the number or type in the number. When I selected the box and typed in 7 on the second section, it ended up thinking there were 107 questions. But as far as I can remember I virtually always click the buttons rather than type it in 🤷🏻‍♂️

Fun and silly bug :)

---

I mentioned to @marcellamaki in a chat today that I did find it weird that I couldn't submit a quiz so I looked back at 0.16 and it's always visible there.

I looked in the Figma thinking that I'd find a discussion around the fact that the new designs remove the previously always-visible "Submit quiz" button that you'd see in 0.16.

So I'll ask for them to weigh in on that separately from this PR.

[pcenov]

Hi @nucleogenesis, great to see so much progress made here! Wow! Today I've started testing scenarios for users upgrading from Kolibri 0.15.2 to this latest version. So far I am happy to report that almost everything is working perfectly fine and as expected. Here are a few issues that I was able to reliably replicate:

1. Missing value for 'File size to download' when starting a quiz which was created on 0.15.2 but not started until after the upgrade:

https://github.com/learningequality/kolibri/assets/79847249/19e6b35b-a305-4b37-92a5-c113d93713a6

2. Cannot view a question listed as a difficult question:

https://github.com/learningequality/kolibri/assets/79847249/f0987878-f8f4-47f9-a020-46f162cadc47

3. There's just an empty gray rectangle at the place of the section title for quizzes created in 0.15.2:

4. For one particular quiz I was able to stumble upon a difference in the number of questions answered correctly between the report I was seeing as a Coach in 0.15.2 and the one in the current version. I'll be further testing tomorrow to see if I can reliably replicate it:

Here are both home folders in case you need them:

kolibri16HomeFolder.zip kolibri15HomeFolder.zip