When we originally published our Learning Evaluation blog series, we covered four learning evaluation models representing a range of approaches and perspectives: Kirkpatrick, Kaufman, Brinkerhoff, and Anderson.
While there are many other learning models we’ve yet to cover, the one we’re asked about most often that’s not mentioned in our original series is Phillips' Learning Evaluation Model. So, what is it, and how should you apply it?
What's Phillips Model for learning evaluation
In a nutshell, Phillips' evaluation of learning model focuses on how to:
- collect data,
- isolate the impact of training versus the impact of other factors, and
- account for more intangible benefits.
(Tip: Find out even more about Phillips' Model for Learning Evaluation in Jack Phillips' book.)
Meet the fifth level of learning evaluation.
One of the most frequently quoted aspects of Phillips' model is the addition of a fifth level of evaluation to Kirkpatrick's Learning Evaluation Model, which is return on investment (ROI).
Phillip’s model states that after determining a learning program’s business impact at Kirkpatrick’s Level 4, you can translate that impact into monetary terms and compare it to the total cost of the program to calculate ROI.
These costs include program development and delivery, plus the labor cost of time for learners to complete the training.
Let’s do the math.
Consider this real-world example: You need a simple learning resource to improve product knowledge and, therefore, increase sales of that product.
Assuming the resource won’t need updating for two years, you observe (or predict—see below) a 10% increase in year-over-year sales.
That 10% increase in sales during the next two years is expected to bring in $50,000 extra profit. So, the benefit of training is $50,000—but that's before we factor in the following costs:
- Resource Development: You spend $10,000 after paying a third-party vendor to develop the learning resource and incurring internal costs to support the resource.
- Completion Time Costs: If 100 salespeople—whose individual hourly rate is about $40—take the hour-long training once a year, the total cost for those salespeople to complete the training is $8,000 (e.g. 100 people x $40 x 2 years).
- Implementation Costs: Other implementation costs—such as responding to helpdesk queries about the resource—come to $2,000.
With the total training costs adding up to $20,000, the ROI (or expected ROI) is calculated as 150%—which is more than double the amount that was originally invested.
Timing is everything.
A common critique of ROI calculations is that they’re often applied after the program has been delivered. If you calculate ROI and determine the program cost more than the value it delivered, it’s too late to make changes and your money has already been spent.
Equally, if the program was a success in terms of ROI, how does knowing that ROI figure help the business—especially if you already know the program was a success from the Level 4 metrics?
ROI, however, can be extremely useful when planning a learning program. As you determine your business goals and L&D program budgets, you can use that data to determine ROI and decide whether to go ahead with the project or revise the plan.
Then, as the project progresses, you can monitor your spending against budget and success against business goals to ensure you’re on track.
Recommended Reading
Don’t assume reduced costs = ROI
ROI can often become conflated with reducing costs—the theory being that if a new training program costs less than an existing program, then you have automatically achieved an ROI of the difference between the two costs.
This approach is not one advocated by Phillips, but rather a common misunderstanding of ROI. The formula used is:
The problem with this approach is that it fails to take into account the benefit of the training and assumes that both the old training and new training deliver the same benefit. But is that the case?
If you don’t also measure the benefit of the training, then your ROI calculation is invalid. The formula to calculate the real ROI is:
Keep in mind, you shouldn’t include development costs associated with the old training because those costs have already been paid; you're not getting that money back by switching to new training.
Up Next: An Intro to Blended Learning
Now that you know the basics of learning evaluation, it's time to explore the world of blended learning (i.e. learning that happens everywhere) and what they mean in practice. And be sure to sign up for Watershed Insights to have the latest news and updates delivered straight to your inbox.
About the author
As a co-author of xAPI, Andrew has been instrumental in revolutionizing the way we approach data-driven learning design. With his extensive background in instructional design and development, he’s an expert in crafting engaging learning experiences and a master at building robust learning platforms in both corporate and academic environments. Andrew’s journey began with a simple belief: learning should be meaningful, measurable, and, most importantly, enjoyable. This belief has led him to work with some of the industry’s most innovative organizations and thought leaders, helping them unlock the true potential of their learning strategies. Andrew has also shared his insights at conferences and workshops across the globe, empowering others to harness the power of data in their own learning initiatives.
Subscribe to our blog
