mumuki-laboratory
mumuki-laboratory copied to clipboard
mumuki
Adaptive flow
Practice guides should be suggested based on previous student's results
I think the suggestion should be on a per exercise basis, not guides. We should try it, but the atheneum should be able to determine whether it's enough or not, and provide a new exercise of the same subject if necessary.
Why not a guide?
Regarding choosing whether propose anothe exercise or not, It should be a quite simple algorithm - for example if less than 20% of exercises required more than 3 submissions, then skip the next practice guide.
Perhaps a guide would be fine, if the algorithm decides when to stop giving new exercises. Following your proposal, I imagine this scenario:
- Learning guide finished, but many submissions were required for the last exercises to work.
- The student is redirected to the practice guide, which has 10 exercises.
- First exercise requires 3 submissions, another exercise is provided.
- Second exercise requires 2 submissions, another exercise is provided.
- Third exercise requires 1 submission, let's say another one is provided, just in case.
- Fourth exercise requires 1 submission, no more exercises are provided.
- The student is redirected to the next learning guide.
BUMP.
I am proposing a simplified adaptive flow:
- Quick flow: Both exercises and guides may be flagged as practice - or maybe reinforcement. Then, when a student solves an exercise easily - let's define this as solving it in less than, say, 5 attempts - the next exercise should be the next non-practice exercise, bypassing any practice . Otherwise, he should be redirected to the available exercise - regardless it is a practice exercise or not. Likewise, when a student solves a guide with less than 5 average attempts per exercise, the next guide should be the next non-practice guide. Otherwise, she should be redirected to the next available guide.
- Retry flow: When a students solves an exercise with too much difficulty - let's say more than 10 attemps -, the when finishing a guide, the next exercises should be the difficult ones. A message similar to the following should be displayed: "It looks like you had troubles solving this exercise. Let's do it again in less attempts!". The call to action button should be 'OK, let's try again', but there should be a smaller button that allows the student to bypass this option.
And here: https://github.com/mumuki/mumukit-flow/blob/master/spec/mumukit/guide_flow_integration_spec.rb.
As you see, there are some pending flows and details, in it still lacks of documentation.
Looks nice, but I have serious doubts about implementing it with the content Mumuki has now, because "practice" guides sometimes introduce new concepts... and "learning" ones are sometimes too easy to consider the students actually learn by just solving them.
"practice" guides sometimes introduce new concept
I am considering introducing a third kind of content "intention" or "aim": reinforcement, so that we can distinguish pure reinforcement practice from practice that is intended to learn concepts. That way, we could have learning, practice and reinforcement content.
and "learning" ones are sometimes too easy to consider the students actually learn by just solving them.
My assumption is that student can solve the assignment using very very few attemps, she probably already knew it or by some reason is outperforming. Perhaps we should just adjust the definition of something that was "very easily" resolved.
Hi all, I like the "retry flow" idea. It is a natural one. I have students that do this on their own, they go back to an exercise they had trouble with, erase it and do it again. With respect to the quick flow I share faloi concerns. There is more than one possible conceptual problem associated with a single exercise. I think that to do this properly we should tag the concepts and common errors involved in the exercises and identify which was the student problem with the exercise (was it loop definition errors, syntax, bad conditions in a conditional, etc). If we don't identify the actual problematic concept or skill we might make her practice something she doesn't need.
We could adapt the "knowledge map" idea from https://www.cs.rutgers.edu/~tdnguyen/pubs/haldeman.sigcse.2018.pdf. A nice research project would be to tag exercise concepts and skills automatically (although we might need some manual annotation to start with). @mgomez4 what do you think?
Hi everyone, I have the same feeling that @benotti and @faloi with quick flow. We should consider that the student learn how to resolve some kind exercise but the concept not yet.
I'm agree with Luciana. We should first categorize the concepts that are included on the exercises and depend of that, identify what kind of errors students make. If we can do it automatically would be great!
OK, so I think the backward flow is quite well accepted :)
The forward flow is perhaps more controversial, but I think there are two different problems here:
- Making the student bypass exercises that she should actually solve.
- This is easily solvable by tagging exercises with a
reinforcement intention.
- This is easily solvable by tagging exercises with a
- Making the student do additional exercises she does not really need to solve
- This is more difficult, since it requires an extra layer of intelligence, and as you have already pointed out it requires some kind of concept map.
In order to solve the latter, I like the idea of the concept map, but I think that it can only became a realistic strategy if there is a way of generating that concepts map with automatic assistance - otherwise we can't expect content editors to massively tag their exercises.
Anyway, we still need content, and unless a submissions-count based solution solution is not ideal, it is a still a good starting point to justifying content creation to content editors while we develop a more intelligent concepts-map based solution.
Also, now I am convinced that creating a new intention for reinforncement exercises instead on relying on the current practice concept is not an option but a need, so supporting the forward flow does impact existing content and is backward compatible.
Finally, when mumuki guides are used as exams - where you the goal of the guide is different -, the forward flow in a well forward-flow-oriented guide allows to implement leveling tests. But I will open a separate issue for this.
Thus, my proposal is to
- implement the described forward flow - with the addition of the reinforcement flag - as a first iteration of a more advanced flow which allows us to quickly start adding reinforcement content and testing how it works
- then, implement a data-oriented, knowledge map as part of a research project - if @benotti and @mgomez4 agree :smile:. I definitely would like to see that feature on the platform.
Be careful, skipping an exercise also need concept tagging in my opinion. If you don't tag concepts you might make the student skip an exercise that practice something but also introduce a new concept.
In your proposal you also need tagging and you propose binary tagging: #reinforcement or #noreinforcement. I propose to use more than two values for tagging. Of course, the tagging I propose can also be done manually.
On Wed, Apr 11, 2018, 07:01 Franco Leonardo Bulgarelli < [email protected]> wrote:
OK, so I think the backward flow is quite well accepted :)
The forward flow is perhaps more controversial, but I think there are two
different problems here:
- Making the student bypass exercises that she should actually solve.
- This is easily solvable by tagging exercises with a reinforcement intention .
- Making the student do additional exercises she does not really need to solve
- This is more difficult, since it requires an extra layer of intelligence, and as you have already pointed out it requires some kind of concept map.
In order to solve the latter, I like the idea of the concept map, but I think that it can only became a realistic strategy if there is a way of generating that concepts map with automatic assistance - otherwise we can't expect content editors to massively tag their exercises.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mumuki/mumuki-laboratory/issues/386#issuecomment-380462934, or mute the thread https://github.com/notifications/unsubscribe-auth/ACbWuMxcJs_7gsnPHSBREfpqvO64-8ABks5tngzPgaJpZM4G0Ui2 .
Hi. I just read the thread quickly, and do not understand with detail all the things involved. (You have a lot of lore in this subject that a person droping-by lacks...) But I will give you my impressions, as I have been asked. :D
First, I am not sure that "number of retries" is a good measure in all cases. In the online courses with automatic evaluation I have taken, I usually have the concepts right at the first attempt, but making the evaluator to accept my solution requires some attemps (usually with petty details). So, I am not sure about how to measure that a student has understood a concept in an automatic way.
That established, I understood that the "forwards flow" you mention is about skipping "reinforcement" exercises, and the "backwards" one is about retrying old concepts because it is somehow determined that that concept was not understood. If that's not the case, what follows can be ignored. :)
Provided a good way to determine the reasons to flow, I agree with both forwards and backwards flows. However, I found that many courses are designed for an average student, and so slow students tend to fail more often, and quick ones are bored. Forward flow can solve the problem of quick students, but "backwards" flow doesn't necessarily helps slow ones. I think that the course must have some "longer paths" that slow students must follow, based on their performance on average excercises. For example, on teaching movement over the board, it can be provided an exercise to draw some complex figure (an S shape, for example), and if that fails (given the assessment method discussed before), provide several more examples of simple figures (one with one East, another with one West, then one with two Easts, another with two West, then one with an East and a West, another with a West and an East, etc.). The simple exercises (with one or two movements) should not be part of the main flow (so, technically it is not a "backwards" flow), but an alternative path that it is suggested to slow students.
I have no idea how difficult this may result to your current implementation, but I feel that if I can design a course with flows, I would rather prefer this kind of possibility than the backwards or forward. Moreover, alternative slower paths can have themselves alternatives for even slower students... :D
Side idea: it would be nice to be able to implement a Choose Your Own Adventure style of exercises when combining multiple choices and adaptive flow: e.g. if student selects option 3 in exercise 4, go to exercise 5, but if you choose option 2 you go to exercise 6. Obviously, the platform progress should be transparent, and student should not be able to see exercises that were not suggested based on theirs elections. @PiruKaren