Criteria for assigning lesson difficulty

[rivaquiroga]

Currently, our editorial guidelines only mention the existence of three difficulty levels but don’t explain the criteria for assigning them:

“To help readers evaluate which lessons best fit their goals and skill level, we provide “Recommended for ___ Users” information in the lesson YAML file. There are currently three tiers, which can be set with the following numerical codes: 1 (Beginning), 2 (Intermediate), 3 (Advanced).”

It will be great to have more explicit criteria that helps us editors decide the difficulty level of a lesson. And also for consistency across lessons.

Here are some ideas to discuss:

Beginner The lesson does not require prior knowledge. All the steps to achieve the lesson’s goals are described (i.e., the lesson doesn't expect the reader to infer intermediate steps that might be evident for more advanced users). The tool used in the lesson is usually easy to install (there are no “known issues”). Any configuration problems that readers might encounter are anticipated (and possible solutions are presented).

Intermediate The lesson requires some prior knowledge that is currently covered in another Programming Historian lesson. If there are no PH lessons that cover it, other learning resources are suggested. The PH tutorials or learning resources suggested are all that the reader needs to be able to follow the lesson (e.g. an intermediate lesson about R expects the reader to have R and RStudio installed, to know how to install R packages, etc. There are published PH lessons that explain how to complete those steps).

Advanced The lesson requires prior knowledge. Although the lesson might make explicit what the reader needs to know and suggests PH lessons/other learning resources, the knowledge required for completing the lesson is not something that can be understood just by following a couple of tutorials (mainly, because the reader needs to have a mental model of the topic). The lesson expects the reader to infer some intermediate steps that might be evident to someone with the prior knowledge, but not to someone that doesn’t have it.

Other things to discuss:

• Lessons that do not require prior knowledge but involve a lot of setup... should be considered Intermediate?
• Lessons that involve the use of tools that are known for their installation issues (e.g. selenium)... should be considered Advanced?
• Does a translation always have the same level of difficulty as the original lesson? What happens if for an intermediate lesson there are no PH lessons or suggested resources in the new language? Should we change the difficulty level to Advanced?

(This issue is related to #2312, but I’m opening a different ticket because we need to agree on the changes before editing the guidelines).

[svmelton]

Thank you for raising this, @rivaquiroga! I also want to add that I think making dependencies and set up requirements more prominent is important.

To your questions:

• Yes, I think that the level of setup involved increases the mental load, particularly because setting up one package or dependency often requires setting up others. (I set up Hugo yesterday, lol)
• It may depend on the tool, but yes, in general I would agree.
• That's an excellent question, and I don't have a good answer, but I think you're probably right. It could change, of course, if there's a new translation of a lesson or more resources appear.

[DanielAlvesLABDH]

Dear Riva, these are great ideas to discuss. In the PT team we faced some problems once we started translating lessons of intermediate and advanced level. Most of us don't have a technological background. I agree with the texts proposed for each of the level. Regarding the questions, I would answer yes to the first two. For the third I would suggest a translation note at the beginningof the lesson calling attentionto the fact that there are no previous lessons translated to the same language yet, but would keep the same level of difficulty.

[anisa-hawes]

Thank you, @rivaquiroga. This is really interesting. Perhaps this guidance could form an appendix to the Editorial Guidelines? It could be part of the reference documentation on our Wiki.

[anisa-hawes]

I've made a note within my re-draft of our Editorial Guidelines about adding in some guidance on how to define ‘difficulty` when we have agreed it here.

[anisa-hawes]

Hello all. In the discussion session which followed our Workshop last week, @acrymble and I received some feedback from educators which I think is relevant to mention in this thread.

Many asked about how to find lessons suitable for their students' ability level. We demonstrated the sorting facility that is available in our Lesson Index, and we pointed to how are lessons are designed to set out prerequisites of knowledge or skill within their opening paragraph and/or point to where that can be gained (via our existing lessons, or elsewhere).

However, there still seemed to be a sense that the Index is overwhelming and that these educators are struggling to easily identify/find suitable lessons. A few asked if we could point to a sub-set of lessons which teach "core" or "basic" skills.

I wonder if, while we think about how we define 'difficulty', we might also think about how we make it simpler to retrieve all the "Beginner" level lessons. For example, maybe we could we could think about tagging lessons with something like a "core skills" label? Or having three 'difficulty' level buttons underneath the 'Activity' and 'Topic' buttons?

Are there 3-5 lessons in each language we'd recommend to educators and students who are just getting started with digital humanities?

Also related: Issue 2228 which explains a suggestion from one of our IPPs to reveal difficulty information on our Lesson Indexes.

[drjwbaker]

Thanks for reporting this to us. Great feedback.

My read. In part, we've been here before (PH once had a pathway of lessons). More pertinently, the problem is that what is 'core' is in the eye of the beholder. For example, until recently, Carpentries thought - for years - that the basics of scientific computing were shell + git + python + sql https://software-carpentry.org/lessons/ Then they switched sql for r. I can't do anything without shell. So would vote for shell. But then I co-authored that PH article. So my point is - I guess - if we choose what is 'core' we should own the fact that that is a political choice about the type of DHy skills we - as a community of editors - think the community should embrace: call it 'editors choice', or similar. Alternatively, we could throw it open to our community to choose. Alternatively alternatively, we look at our traffic and assign 'core' based on what people use (which, I recall, would be mostly python). Maybe we don't do 'core' at all, and focus attention instead on what is 'trending' based on our traffic?

[acrymble]

It may be helpful to tag lessons that we know have successfully been used in workshop settings or classrooms.

[anisa-hawes]

Thank you, @drjwbaker. This is interesting and useful context.

If we would rather avoid making those decisions about which are "core" skills, and simply surface the relative 'difficulty' levels of our lessons, then maybe designing a way to click "Beginner", "Intermediate", or "Advanced" would be enough to help educators and their students find what they're looking for.

We could also consider sharing some notes with our readers about what the 'difficulty' level means. For example:

Beginner:

This lesson is suitable for novice learners – we will guide you through each step. You don't need any prior knowledge to get started, and any software needed is easy to install.

[drjwbaker]

I guess I'm actually for us curating places to start, because I can see the need. I guess my point is I'm happy so long as we are reflexive enough to make clear that they aren't the places to start, but the places to start that we have chosen (or, continue to chose through regular revision of the choices).

[anisa-hawes]

Understood. Thank you @drjwbaker. Maybe we can do all three:

• curate a set(s) of lessons which we think could be places to start
• make it easier for readers to filter lessons by difficulty
• make it clearer what our difficulty levels mean (to editors who assign them, and to readers who interpret them)

[hawc2]

Chiming in here, it would be great to standardize and clarify this for authors, readers, and for our internal editorial purposes. @anisa-hawes how far along did we get with updating our editorial guidance on this and related issue #2228?

I encountered recently a relevant example, when I was surprised to see this lesson, currently being published, listed as Intermediate difficulty, until the author explained they aimed for it to introduce difficult concepts to beginners: https://programminghistorian.github.io/ph-submissions/en/drafts/originals/interrogating-national-narrative-gpt

[anisa-hawes]

Thank you for adding your thoughts, @hawc2. Seems like we need to agree a clear set of criteria to define each level of difficulty. Riva has proposed a framework to start with, and we have some initial feedback. I've added this to the agenda for discussion at the next Managing Editors' Meeting.

My sense is that #2228 is more about the front-end of our website, and how our readers can interact with the lesson index to filter and find content that meets their needs.

[anisa-hawes]

Hello all,

I'm sorry to have left this Issue open and untended so long. I have collated some notes which attempt to bring together the suggestions Riva raised upon opening this thread, with some thoughts discussed at the last Managing Editors' Meeting.

I've tried to clarify the criteria, so that assigning difficulty: is more straight-forward for editors/authors, and more consistent going forwards for our readers. I've broken the decision down, focusing on:

• Prerequisite knowledge
• Handling of specialist/technical terms
• Complexity of install and set-up
• Support for troubleshooting
• Guidance for further learning (towards or beyond the lesson)

I have also included some notes on usability, sustainability, accessibility and inclusivity which I think are useful to consider alongside difficulty. I think all lessons (whether beginner or advanced) need to support these objectives. Complex ideas can be expressed using straight-forward language, and navigation of difficult topics can be supported by logical section-headings and good signposting.

I very much welcome feedback on the notes below! In particular, it would be useful to hear the thoughts of @rivaquiroga, @hawc2, @DanielAlvesLABDH, @spapastamkou and @marie-flesch

--

Notes on: Assigning lesson difficulty

---

When assigning lesson difficulty, it is useful to consider: how much prerequisite knowledge is expected; whether and how specialist or technical terms are used and defined; the relative complexity of install and set-up; whether trouble-shooting steps are included, outlined, or referenced; where and how knowledge beyond the lesson's scope can be learned (in existing Programming Historian lessons and other written documentation, or is applied experience necessary?).

difficulty: 1 Beginner

• No prior knowledge required
• All steps are clearly defined
• Specialist or technical terms are defined
• Software packages are easy to install (no “known issues”)
• Challenges that readers might encounter are anticipated, and clear trouble-shooting steps are included
• Further Programming Historian lessons (or external resources) for advancing new skills may be referenced

difficulty: 2 Intermediate

• Some prior knowledge is required
• Existing Programming Historian lessons (or external resources) to empower less experienced readers to gain that knowledge are identified
• Key steps are defined, all steps are outlined
• Specialist or technical terms established by beginner lessons are used in context, while any new terms are defined
• Software install and set-up may be subject to “known issues”
• Challenges that readers might encounter are anticipated, and trouble-shooting steps are outlined

difficulty: 3 Advanced

• Significant prior knowledge and applied experience required
• Confident ability to infer intermediate-level steps expected
• Specialist or technical terms are used throughout, new concepts are explained
• Software and packages may be known for their complexity to install and set-up
• Challenges that readers might encounter are anticipated, and trouble-shooting steps are referenced

Whether beginner or advanced, all lessons must be usable, sustainable, accessible and inclusive:

Usability

• Summarise learning outcomes in an opening paragraph
• Structure logically, use section headings for clear and meaningful signposting
• Keep within the word limit: 8,000 words (including code)

Sustainability

• Cite software versions, specify technical dependencies
• Encourage general methodological narratives over screenshots and GUI-specific instructions
• Anticipate challenges, support trouble-shooting

Accessibility

• Caption images concisely, and ensure they are well-described with alt-text
• Provide cut-and-pasteable code, rather than showing it in screenshots
• Keep accessibility in mind when choosing data, case studies, and software

Inclusivity

• Utilise open formats, open programming languages and free software
• Consider implicit bias in algorithms and tools
• Identify multi-lingual external resources and software documentation wherever possible
• Be explicit if recommending learning content that is not available in the lesson's native language

[DanielAlvesLABDH]

Dear @anisa-hawes many thanks for this. I think it covers the main issues and give us the necessary features for assign lesson difficulty level. I would include the question of permanent links to external resources in the sustainability section.

[spapastamkou]

"Utilise open formats, open programming languages and free software" ==> I think that this is (also) one of the basic criteria of sustainability

[anisa-hawes]

A new Wiki page has been created to document this criteria for Defining Difficulty. This page represents part of a series of guideline documents I am developing to support editors and Managing Editors in their work.