instructor-training
instructor-training copied to clipboard
carpentries
Objectives and Outcomes
Prompted by a discussion with @allegravia and reading this pre-print Allegra co-authored with Rochelle Trachtenberg, Jessica Lindvall, and Teresa Attwood [1], plus some of the other literature referenced therein [2, 3], I would like to discuss the possibility of adjusting the language we use around learning objectives.
Based on reading the materials linked above, it is my understanding that the statements we currently describe as learning objectives - in Preparing to Teach, the boxes at the top of each episode in The Carpentries lesson template, and elsewhere - would be more appropriately named learning outcomes. Nilson [3], for example, describes a learning outcome as
a statement of exactly what your students should be able to do after completing your course or at specified points during the course.
which seems to me to be precisely what we mean when we talk about objectives in The Carpentries material: action statements that describe the tasks learners will be able to achieve once they have completed the episode/lesson, that can be measured as an indication of the learning that has taken place.
On the other hand, How Learning Works [4], the main text we studied in Trainer Training, discusses objectives and makes no mention of outcomes. But the first resource [5] in the Papers list on the Reference page of the Instructor Training site is all about outcomes...
And of course, making this change would not be a minor undertaking: there will be pull requests required here, on the lesson template, in the lesson development study groups pages, and almost certainly other places I haven't thought of yet. But, if it brings our materials in line with the language used in the literature, I think it would be worth it (translates to: I would make these PRs).
What do others think? Would anyone else like to add their perspective?
References
[1]: Tractenberg, R. E., Lindvall, J. M., Attwood, T., & Via, A. (2020) Guidelines for curriculum and course development in higher education and training. https://doi.org/10.31235/osf.io/7qeht [2]: Nicholls G (2002) Developing teaching and learning in higher education. London, UK: Routledge. [3]: Nilson L (2016) Teaching at its best: A research-based resource for college instructors, 4E. San Francisco, CA: Jossey Bass [4]: Ambrose S et al (2010) How Learning Works: Seven Research-Based Principles for Smart Teaching. [5]: Baume, D (2009) Writing and using good learning outcomes. Leeds Met Press, ISBN 978-0-9560099-5-1
Practically
This is the right time to discuss big changes like this. @karenword has a draft revision to this episode in : https://github.com/data-lessons/instructor-training/pull/38 so, if we do choose to do this, those changes should be made over there, not here so that they're a part of the major revision that's going on.
Content
I like this a lot. I actually probably talk through this as learning outcomes because, in reading this, i now see that i've merged the two concepts mentally and probeably user them interchangeably.
The revision Karen is doing de-emphasize's Bloom's and makes the current objectives more prominent, but the outocomes framing may make it even more tangible for trainees..
Thank you for this suggestion @tobyhodges !
In my experience, these terms are often used interchangeably. Poking around a bit, I'm seeing a number of sites that purport to distinguish between them, but I don't see any particular standards being cited. Where distinctions are being made (example 1, example 2), they seem to orient around depth, with "objectives" being used more narrowly (e.g. skills acquired during a class or portion of a class) and "outcomes" being used more broadly (e.g. skill levels achieved as a result of the course overall). This distinction makes more sense in a long-format course setting.
So in all, I don't think the definitions, to the extent that they are agreed upon, make a big difference. I suppose you could argue that "outcomes" might be better applied to whole lessons (vs single episodes) or workshops. There is maybe a sort of a philosophical question about how modular we think our content is, but again that's assuming this distinction is firm.
There are two things I like about the term "learning outcomes" vs. "learning objectives":
- I think "outcomes" is more clearly centered on the learner and helps to keep 'teaching goals' from creeping in.
- I have a sense that "outcomes" are becoming more common in the language of instructional development programs, so use of this term is more likely to leverage prior knowledge vs creating confusion.
On this second point, I'd really like to hear more from the community. What language have you encountered locally or in other contexts when you have learned about instructional design? What makes sense to you?
To me, as a non-native speaker, outcomes seem more tangible, i.e., that's what I'll get. Whereas objectives sound to me like where I'm aiming towards. (This "feeling" may be due to the usual objectives that we set every new year... and they are never reached 🙊) I also think like @brownsarahm that I may have used them interchangeably.
Another thought: we could also consider embracing the distinction and covering both.
Pros:
- eliminates necessity of changing the language in use for "objectives" across our curricula.
- provides a vehicle to discuss broader outcomes for Carpentries workshops. I have been trying to do more to contextualize overall goals, particularly in motivation (updates in progress here), so explicitly considering broader 'outcomes' in addition to narrow 'objectives' could support Instructors in appreciating the bigger picture these detailed objectives are/should be feeding into.
Cons:
- this involves adding, not replacing, vocabulary and content...
In my experience at Australian universities, learning objectives tend to define the scope of what is to be learnt at the subject or unit level, and learning outcomes define the synthesised result of having learnt those objectives, and reflect more broadly the context of that subject in the course/degree program. In this example, the learning outcomes map to discipline-wide graduate attributes, which look an awful lot like persona development (rabbithole 1). Underneath the tip of this iceberg are the teaching strategies and rubric development for assessment (rabbithole 2).
Are these hammers too big for carpentries lessons though? I'm happy to go with outcomes if it helps us align to current standards in instructional design. For me changing to outcomes makes me want to think much harder about what forms of assessment and evaluation we make available to learners.
I like @karenword's suggestion for considering the broader contextual outcomes - this could tie in neatly with advocacy work by the EC and LCAG perhaps?
Also I think I should read through the curriculum development handbook again!
@tobyhodges do you have a sense of how you'd like to proceed on this from the curriculum development side?
In agreement with what David said above, I think the word "outcomes" is a more tangible. And in agreement with Sarah and Karen, I really like that "outcomes" keeps the processes of lesson design and teaching preparation centered on what the learner will get, rather than what the teacher wants. I also think that, as a word, "outcome" is a more accurate description of what we write for our lessons than "objective."
As luck would have it, we will be looking at objectives next week in the Lesson Development Study Groups program. If you can afford to wait a little longer to make a decision on this, I will add it as a discussion prompt for the groups, and post back here afterwards with a summary.
Absolutely can wait @tobyhodges -- this is the kind of decision that should be made slowly and carefully.
I think, having let this settle for a while, I am leaning towards the idea of adding an "outcomes" section that covers a full lesson, for example, and leaving "objectives" in place for each episode.
Outcomes could potentially address some of the 'hidden curriculum' - things like self-efficacy and appreciating errors as learning opportunities, in addition to a global summary of skills development. Adding broader goals like these may also pave the way for more direct connections with our long-term assessment program.
Eager to hear what the Lesson Development community has to say!
The views of participants in the Lesson Development Study Groups program were also quite mixed on this topic. I think we are quite far away from a consensus within the community about which term best describes what we are trying to capture in these statements about our lessons/episodes.
At the very least, I think we need a callout to Preparing to Teach, mentioning
- the existence of both terms,
- the overlap and interchangability of their usage in the literature, and
- an indication that we use Objectives throughout our material.
But I am also open to the idea of a broader Outcomes section for lessons, keeping Objectives for each of the episodes. However, I am not sure the terminology here is as important as the advice we give to lesson developers: for example, we could add to the Curriculum Development handbook some guidance for developers, to define broader Objectives for the lesson as a whole that capture "greater than the sum of their parts" skills that learners will pick up. I would hope that point would hit home regardless of whether we talk about Outcomes or Objectives.
On Preparing to Teach, it seems relevant to point to: https://github.com/carpentries/instructor-training/issues/1226 started by @Denubis.
At this stage I'm thinking about the usefulness of creating learner personas for instructor training, and what impact that might have on how we answer your questions on objectives and outcomes.
I like outcomes instead of objectives because it sounds less theoretical and less like "university professor speak." Specifically, I think that the word "outcome" is used more frequently and is therefore better understood.
However, using outcomes in some ways and objectives in other ways is going to be really confusing for me. I think it's possible I'm out of my area of expertise in commenting on this issue, but since I read all the way through the discussion I thought I'd just share my reaction.
@tobyhodges and I discussed this today and we determined that this is fundamentally a curriculum decision. We will continue the conversation here because this concept is first introduced in Instructor Training, but we will wait for a final call to be made by the Curriculum Team and will update here accordingly.