Real-World Learning Analytics: Chipotle Mexican Grill

What’s your data capture strategy for tracking learning experiences, and why did you choose it?

Our primary format is xAPI because of all the information that comes with it and the fact that it is more HMTL5-friendly than SCORM. We have mobilized our learning platform to tablets and phones, and our content needed to shift along with it. xAPI was an easy transition for our elearning developers and our platform vendors to adopt.

Our secondary format is the litany of business performance metrics within our organization. By tying in our learning experiences with performance metrics, we can see what people are learning and how they perform back on the job.

How do you decide when you have enough data (or too much data)? Do you have suggestions for detecting and cleaning up bad data?

You can never have too much data. The challenge is getting the right data. Once you start digging in and looking at the data you do have, you’ll realize what questions you should be asking, what data you really need, and how it needs to be formatted going forward.

You can never have too much data. The challenge is getting the right data.

This will be your guiding light as you then work to clean up any data that would be useful but isn’t currently useable because of where its location or format. The key, though, is to not focus on the data but instead on the questions you need data to answer.

Knowing exactly what data to look at and its meaning can be challenging. What are some recommendations or examples for learning benchmarks and KPIs?

Most learning organizations will start with Kirkpatrick’s four levels of evaluation. They then begin to get their Level 1 data and determine if people even like the training. Some get to Level 2, where they measure knowledge before and after the training to gauge training effectiveness. This level is where many learning organizations spend most of their time and where they often stop.

My advice is to start at Level 4 rather than Level 1. Every piece of learning should have some business result in mind. Ask yourself, “What is the business impact of this piece of learning?” and “What is supposed to happen once someone takes this training?”. 

Once you know that, you can identify the behaviors needed to accomplish that business impact. Then, determine what people need to learn to enable that behavior. Finally, ask them if they liked the training.

In this model, you can measure the business impact, behavior change, learning, and enjoyment simultaneously—and the meaning of each piece of learning is built in from the start.

What are a few small ways people can regularly improve their learning?

Learning programs are only effective if people attend the program, then follow through on the behaviors identified in the program. While you may or may not be creating new training every week, you can gather quantitative or qualitative data to understand why people are or are not taking the training. And more importantly, you can see why they are applying what they learned.

Knowing the answers to these questions will inform what you need to alter about your existing programs and what you need to keep in mind about future programs. The best part is that this can be as simple as going out into the business and talking to your participants to really understand what is happening.

There are many metrics to choose from when it comes to showing ROI. So how do you prove the value of learning in your organization? And which metrics do you consider to be the most powerful?

The two measures that I start with are always the main business drivers of the organization and the retention of employees. After attending my training program, are participants more or less likely to achieve the company's business goals (which are unique to each company), and are they more or less likely to stay with the organization?

Suppose I can create a training system where participants are more likely to achieve the company’s business goals and are more likely to stay with the company. In that case, I will have created undeniable business value for the company.

Sometimes the boss is the only obstacle standing in the way of doing great things. What are your suggestions for getting organizational buy-in with data?

The first suggestion is to understand what the boss is trying to achieve. Tying data to your boss’s goals is an easy way to get buy-in to the data. What are her goals, and what are the things getting in the way?

Data will be the key to proving your idea has merit if you can understand what she is trying to accomplish and know how she can achieve it.

Without data, your ideas are just ideas. They are subjective and may be overlooked. Data provides objectivity that trumps an idea every time.

Without data, your ideas are just ideas. They are subjective and may be overlooked. Data provides objectivity that trumps an idea every time.

When the data you provide has no obvious business value or doesn’t have enough value to garner your boss’s attention, look at your organization’s bigger picture to understand how your “great idea” fits into the larger scope of the company. If you can tie the results of your great idea to something of benefit for the company, you may have a shot at getting buy-in.

What are your goals or plans for your learning analytics program?

I have two primary goals for analytics in the future. The first is to combine learning analytics with an adaptive learning framework to automatically trigger the best learning path for an individual employee.

The adaptivity will be based not only on what employees need to know and which way they learn best but also on the learning paths of the most successful employees in the company. This way, each learning instance will provide the best learning path for an employee and the company.

The second goal is to specify business outcomes down to the individual level, then tie those outcomes to learning in order to truly tie business KPIs and on-the-job behavior to learning outcomes.

An example of this would be any audit in your business (i.e., a customer satisfaction survey or an operations audit). Rather than look at unit-level results or hold a general manager responsible for an entire unit’s numbers, I hope to understand who exactly was working at the time of the audit and track the results of that particular audit to the individuals who were there at the time of the audit.

If remedial training is required, an adaptive system would automatically trigger remedial training for those employees that need it. Likewise, should the team be rewarded for an outstanding audit, the adaptive system would trigger rewards for those employees working.

With this in place, it will be far easier to track performance and behavior at an individual level and move corporations away from an audit and punishment culture and into a more individual merit-based reward system.

L&D Spotlight: Applied Industrial Solutions

Next, we'll talk with Andy Webb, a talented L&D professional who knows how to develop strategies and solutions linked to growth, performance improvement, and business goals. Be sure to sign up for our blog to have the next post sent straight to your inbox.