You can’t show a learning program’s success or monitor its progress without first dividing the program goals into individual learning evaluation metrics. In this post, we’ll discuss the second step of Watershed’s training evaluation method: Define performance metrics most appropriate to your organization.
We've outlined seven practical steps when evaluating learning programs. Here, we'll discuss the second step in the learning evaluation process—identifying relevant success metrics for your organization. To some extent, this step happens in tandem with Steps 3 and 4 (Discovery and Design), as the information gathered and decisions made in these steps affect one another.
How Do I Select Performance Metrics for Learning Evaluation?
When you define metrics, we recommend a collection of metrics that cover as many of the five areas below. Each program goal should have one key performance indicator (KPI) that can be monitored throughout the program. These KPIs will be the most important metrics of your evaluation.
The following list provides examples of evaluation metrics at each level:
1. Usage and quality of content
- Number of times content is accessed*
- How much and which parts of video content are skipped
- Attendance at events
- Time spent
- Difficulty and quality of particular assessment questions
- Survey responses
2. Delivery and discoverability of learning experiences
- Number of times the experience is experienced*
- Clicks or time taken to access the learning experience
- Survey responses
3. Improvements in knowledge and competency
- Assessment/simulation scores comparing progress before and after the learning experience
4. Improvements in job performance
- Peer/manager/mentor observations
- Customer feedback
- Data from business systems used to do the work
- Survey responses
5. Overall achievement of the program goals
- Number or value of sales made per month
- Average time taken to produce each widget
- Average customer satisfaction rating
- Employee retention %
*Number of hits applies to both levels as the reasons for high or low numbers could relate to either quality or discoverability. This metric needs to be used in conjunction with other data or research.
Expressing L&D Metrics
The specific metrics and how they're expressed will depend on both the program and the organization. This is particularly significant for the metrics relating to the overall program goal.
For example, if an organization is interested in short-term benefits and has confidence in the value of learning, the metrics might stand on their own.
In an organization that's more interested in long-term benefits and has low confidence in the value of learning, the metrics should be expressed in relation to industry benchmarks.
Recommended Reading
Whatever metrics you choose, plan to measure them over a period of time. It’s important to evaluate whether or not learners remember their learning and persist in their changed behaviors.
If you want to know if your learning program is successful, you must first determine threshold values for each metric to represent results that are expected, below expectations, or above expectations.
Keep in mind, these thresholds may change. For example, as your organization improves. Remember to link these thresholds to industry benchmarks or levels of return on investment/expectation as appropriate for your organization.
In addition to those metrics, define a set of quality criteria for your materials and resources, which can be applied before learners interact with them.
Criteria should reflect both production quality (e.g., Are images used appropriately?) and pedagogy (e.g., Does the solution include reminders to aid remembering?).
Once you're satisfied with your chosen metrics (and how they're expressed), review them with relevant stakeholders to make sure the metrics provide all the necessary information.
Recommended Resources
Step 2 Objectives
- Define the set of metrics to monitor and analyze the program.
Making It Happen
- Ensure your metrics cover the range of levels outlined above.
- Consider the needs of relevant stakeholders.
- Don’t worry about feasibility and cost of capturing data at this stage. You'll prioritize and focus your list of metrics later.
Up Next: Learning Evaluation Step 3, Discover
Our next blog post covers the third step in our seven steps of learning evaluation: discovering the effectiveness of your learning programs. Don't miss out and sign up to receive our latest blog posts.
About the author
As a co-author of xAPI, Andrew has been instrumental in revolutionizing the way we approach data-driven learning design. With his extensive background in instructional design and development, he’s an expert in crafting engaging learning experiences and a master at building robust learning platforms in both corporate and academic environments. Andrew’s journey began with a simple belief: learning should be meaningful, measurable, and, most importantly, enjoyable. This belief has led him to work with some of the industry’s most innovative organizations and thought leaders, helping them unlock the true potential of their learning strategies. Andrew has also shared his insights at conferences and workshops across the globe, empowering others to harness the power of data in their own learning initiatives.
Subscribe to our blog
