There are several ways to measure the impact of open educational resources (OER). While authors and publishers of traditional textbooks track metrics such as books sales, royalties, or course adoptions, OER creators usually have different goals in mind so may want to consider different types of metrics such as amount of savings to students. Deciding which metrics to track should not be a last minute decision. Try and consider the metrics you want to track closer to the beginning of a project. This guide provides some suggested frameworks and resources to consider that may meet your needs to assess impact.
'36 Indicators of OER Impact" by Janet Swatscheno, licensed CC BY 4.0. This graphic is adapted from "56 Indicators of Impact" by Holbrook, J Britt, Kelli R. Barr, and Keith Wayne Brown. 2013. Figshare. https://doi.org/10.6084/m9.figshare.707081.v3. Licensed under CC BY4.0
The Open Education Group's COUP Framework (2013) is an approach to studying the impact of open educational resources and open pedagogy. COUP stands for cost, outcomes, usage, and perceptions.
The adoption of Open Educational Resources can impact a range of financial and cost metrics for students and institutions. Proponents of OER frequently claim that using these resources instead of traditional publisher textbooks or digital materials will save students money in the post-secondary context and will save organizations money in the K-12 context. There may also be other financial impacts, like changes in bookstore revenues and tuition revenues. The Cost strand of our work provides empirical evidence about the magnitude and direction of the financial impacts of OER adoption:
Given the folk wisdom that “you get what you pay for,” some individuals and organizations worry that student learning will necessarily suffer when students use freely available, openly licensed resources instead of $200 textbooks. OER proponents claim that using these resources instead of traditional publisher textbooks or digital materials increases student access to critical learning materials and expands faculty’s academic freedom, consequently improving student learning outcomes. The Outcomes strand of our work provides empirical evidence about the magnitude and direction of the learning impacts of OER adoption:
The permissions provided by open licenses allow students to use OER in a range of novel ways – for example, updating a history textbook based on recent events. Likewise, the permissions provided by open licenses allow teachers to engage in new pedagogical practices. Proponents of OER frequently claim that improvements in student learning outcomes will be highly correlated with the degree to which students and faculty exercise the permissions offered by OER. The Usage strand of our work provides empirical evidence about the ways faculty and students use OER and the the degree to which impacts on learning outcomes covary with these uses. We operationalize the idea of ‘exercising the permissions granted by open licenses’ by determining the degree to which students and faculty engage in activities described in the DIME model of OER adaptation:
What do faculty and students think about, and feel toward, Open Educational Resources? How do they judge their effectiveness relative to traditional textbooks? Their rigor and coverage? Do they find the formats, structures, and other design features easy to use? Frustrating? What about other stakeholders, like parents or policy makers – what are their thoughts and feelings toward OER? The Perceptions strand of our work provides empirical answers to these questions.
Source: "The COUP Framework" by Open Education Group, is licensed under CC BY 4.0
For more information, check out:
The SCOPE Framework (2023) is an expansion of the COUP Framework outlined above. It includes social justice, cost, outcomes, perspectives, and engagement.
| Concept | Definition | Examples of Areas of Inquiry |
| Social Justice | A corrective and liberatory practice that acknowledges the history of systems of oppression along with its modern day legacies. This is explicitly addressed through the equitable distribution of resources, opportunities, and attainment of full social, political, economic, and human rights for all |
|
| Cost | Losses assumed to be either due to or avoided by open education |
|
| Outcomes | Presumed effects due to open education |
|
| Perspectives | Impressions and opinions of open education |
|
| Engagement | Fully participating and being actively involved in open education |
|
For the full framework, check out:
David Wiley's S3 framework (2020) focuses on three components to evaluate the impact of open educational materials: success, scale, and savings. S3 focuses on three questions:
Definition of Success: Completing a course with a final grade that allows the course to count toward graduation.
Measurement:
Definition of Scale: The proportion of students being reached.
Measurement:
Definition of Savings: The amount of money spent on course materials by the average student using OER compared to the amount of money spent on course materials by the average control student.
Measurement:
Adapted from: David Wiley, "S3: A Holistic Framework for Evaluating the Impact of Educational Innovations (Including OER)" Improving Learning [blog], June 23, 2020. Licensed under CC BY 4.0
Ebner, Orr, and Schön (2022) provide a framework for OER impact assessment which clearly connects to open educational practices. They highlight insights relating to OER impact assessment:
It is meant to be a practical approach to get started by looking at existing policies/strategies for purposes, arguments, or aims related to OER and shows how there is no one way to measure impact. The framework identifies all the aspects that may need to consider for their own context.
For more information, check out: