How to Use Observation Checklist Apps to Capture Skills Data

What’s an observation checklist application, and how does it fit in a learning ecosystem?

Observation checklist apps (which usually run on mobile devices) provide interactive webforms to structure and systematize observations of either a training simulation or completion of a real job task. Using these forms, observers can capture data about how well employees completed tasks.

^{This checklist was created with Xapify, a skills development platform, and ensures medical professionals take appropriate infection control precautions.}

A checklist application may be a stand-alone product or may form part of a broader skills management platform.

A skills management platform collects information about learner skills and supports the development of additional skills through learning plans. The observational checklist application is used to assess skill competency.

Data may include a combination of:

• rating scores against performance metrics, such as product knowledge
• a record of process steps completed or missed
• errors selected from a list of common mistakes
• the time taken to complete specific tasks (this is especially relevant in a medical context)
• comments from the observer

^{This screenshot shows how you might visualize data from a checklist to identify which competencies are (and are not) most commonly demonstrated during observations.}

Depending on the specific features of the checklist app, you may or may not be able to integrate it with other platforms in your learning ecosystem.

For example, Xapify is a skills development platform that includes an observational checklist. Xapify can assign learning content to employees based on any knowledge or skills gaps identified during observations.

As a result, Xapify requires integrations with all the systems in which that training content resides. Other L&D observational checklist apps that do not have this functionality may not integrate with other platforms, aside from data extraction to the learning analytics platform.

Observation checklist software data requirements and what to ask your vendor

As you might expect, the most critical data to capture is the data generated from answers to checklist questions. This data should include identifiers for:

• both the employee and the observer,
• the question, and
• the observer’s response.

This data can be used at an individual employee level to see how someone performed a particular observed task. For example, you might use these reports to:

• Review with the employee as part of a coaching discussion.
• Inform decisions to assign training to the employee.
• Inform the employee’s manager in relation to the employee’s performance.

^{This Watershed dashboard presents some example data generated by Xapify using a checklist. It shows scores against three competencies by learner, average scores by competency, and comments made by the observer.}

The data also can be aggregated to give a view of competency and skills at a higher level. For example, you might use these reports to:

• Identify exceptionally high-performing employees to take advantage of their expertise or to inform reward and recognition schemes.
• Identify particularly low-performing employees to provide support or inform other decisions.
• Map levels of skill proficiency and skill gaps across the organizations.
• Inform decisions about the training required to address commonly identified issues.

The purpose of observational checklist apps is data collection, so you should expect apps you are looking at to have a variety of robust and well-documented options for data extraction.

Ideally, the app will have a good xAPI implementation (which is a particular selling point of Xapify). At a minimum, you should expect CSV data to be available via API. When considering a checklist app for workplace observations, ask the vendor if they:

• Have experience integrating with an LRS or LAP.
  • In particular, have they integrated with your LAP?
• Support xAPI tracking of checklists.
  • If not, do they support extracting the data as CSVs via API?
  • Can they provide the documentation?

• Ensure the employee and observer identifiers match (or are mappable to) the identifiers used in your other data.
• When editing checklists, make sure that you understand your revisions will impact the data. Discuss this with your LAP and checklist app vendors and run tests for yourself if you are unsure how edits will affect the data.
• When creating new checklists, review them carefully and consider piloting them before using them more widely. This way, you avoid having to make future changes that might impact the data.

If you edit a previously used checklist, you may run into issues with inconsistent data. For example, if you replace a question, you may not be able to compare the data from the original question with the data relating to the updated one.

Observation checklist applications and your learning ecosystem

You can use observational checklist data alongside data from other systems to compare employee performance. For instance, do employees who complete training perform better during observations than those who don’t complete training?

This information enables you to identify which training best supports good performance. Conversely, it also shows which training does not contribute to performance (or might even be harmful). By combining the data sets, you will be able to ask these kinds of questions:

• Which learners significantly improved between observations, and what learning did they complete in between? This information will help identify particularly impactful training, which can then inform the design and curation of new training.
• What digital credentials and skills do people who perform well in observations have compared to those who perform poorly? This information will help inform decisions about hiring, internal promotion, and training.
• Where are there gaps in training that either lead to or fail to prevent errors? This information will enable you to create, curate, or promote training to address these gaps.

The checklist data needs to be mappable to the rest of your learning data for this to work. That’s why the observational checklist app needs to use a mappable learner identifier.

Changes to the ecosystem are unlikely to affect checklist data.

The role of observation checklist apps in skills and compliance

Where proper completion of tasks and processes is a compliance requirement (e.g. in healthcare), data from the checklist app can be a crucial input to compliance reporting. This scenario is particularly common in healthcare settings.

For example, one Watershed client uses Xapify to ensure healthcare providers wash their hands at appropriate times. This data informs training needs and also may be relevant for compliance.

Three elements that can contribute to evidencing that a person has a skill:

1. Evidence that the person has put effort into developing that skill, such as a record of learning completed or hours of practice logged
2. Evidence that the person has mastered the skill, such as successful completion of an assessment
3. Evidence that the person has applied the skill in a real-world context, such as within a work project

Successful completion of observed tasks can provide good evidence of applied skills (and could even be used to inform the awarding of digital credentials). Use checklist data to report on applied skills by mapping skills to observations or parts of observations.

Up Next: HRIS Data Requirements for Learning Analytics

So far in this series, we’ve looked at platforms and applications that generate data relating to learning activities. But learning data can be much more interesting and useful when combined with data about learners from an HRIS system.

This data means you can compare the learning activity of different parts of the organization or restrict the data shown in a report to a particular department. In the next post, we’ll explore data requirements for HRIS data—including how to get that data into your LAP and how you can use that data. Be sure to scroll up and subscribe to Watershed Insights so you don’t miss out!