Mastering Learning Analytics: Understanding Complexity Levels

A closer look: Learning analytics complexities

As you may recall from our previous post, the first dimension in which people talk about learning analytics is complexity—or the sophistication of the analytics.

Furthermore, analytics consist of varying degrees of complexity, which provide richer, deeper insights as higher levels of complexity are attained. And we created the Watershed Method to define these degrees of complexity by breaking them into four levels.

Level 1: Measurement

The most basic form of analytics is measurement—the simple act of tracking things and recording values. Analytics isn’t possible without first gathering this data, which typically occurs through:

• passive data collection, or passively accumulating data as a byproduct of another activity, such as using a system
• active data collection, or intentionally being involved in taking measurements to gather data

Level 2: Data Evaluation

Now that you have data, it’s time to start evaluating it, which is the process of finding meaning in the data you’ve measured. For instance, does your data tell you something good or bad?

Evaluation usually begins by applying descriptive analytics, which simply means using different techniques to capture and aggregate data to paint a picture of what’s already happening. These analytics help us understand the what questions. In fact, the vast majority of analytics practitioners use simple descriptive analytics.

Common descriptive analytics require relatively basic mathematical computations—such as averages, counts, rankings, ranges, and percent changes—and are often presented in reports, charts, and graphs.

Level 3: Advanced Evaluation

When datasets get large enough,* you can use advanced evaluation techniques—such as exploratory and inferential analytics—to search for relationships within your data and discover powerful insights.

You can then probe these correlations to help form theories about causation and other relationships between variables. And you can start to investigate why things happened, not just what happened.

Keep in mind, advanced evaluation requires trained statisticians to use sophisticated techniques, such as correlation and regression analysis. In addition, this level of analysis is often a journey where one bit of information leads to more questions and where multiple models and visualizations need to be combined to provide insight.

Data mining, artificial intelligence (AI), and machine learning technologies go a step further and can help uncover hidden patterns that humans could never find using manual methods.

Level 4: Predictive & Prescriptive Analytics

The first three levels of the Watershed Method help you understand what’s already happened. The fourth level of the method covers predictive and prescriptive analytics, which help you understand what will happen in the future.

• Predictive analytics attempt to answer the question, “given what I know about the past, what is the most likely outcome if Event X happens?”.
• Prescriptive analytics goes one step further. It says, “given that we know the most likely outcome of Event X is going to be Event Y, then when we observe Event X, we should take Action Z to optimize the subsequent result.”

These techniques often require extensive datasets* and advanced AI/machine learning algorithms to make nontrivial predictions effectively.

Applications in learning analytics

All four degrees of analytics—and their respective techniques—can be applied to learning in different contexts. However, in current practice, most learning analytics fall into the measurement and evaluation categories.

In fact, very few corporate learning and development (L&D) departments use predictive and/or prescriptive analytics. This is due mainly to the industry’s lack of available data and analytical capabilities. As xAPI continues exposing more data, though, we’re seeing greater use of advanced analytics.

* The size of a particular dataset will vary depending on what’s being evaluated.

Which way are we looking?

Analytics allows us to look both into the future and the past.

Measuring leading indicators allows us to evaluate our progress, anticipate our attainment of a future goal, and adjust course accordingly. Educators use “formative assessments” as leading indicators of student success and as a valuable tool for shaping learning experiences.

Measuring lagging indicators allows us to look back and understand whether we’ve hit our goals. Educators use “summative assessments” to evaluate students at the end of a period to assess whether students met the desired learning objectives.

Up Next: Learning experience analytics

Next week, we’ll be moving onto learning experience analytics, which helps you understand more about particular learning experiences or sets of learning experiences—such as online courses, mobile apps, or in-person classes.