In nearly every business function, wheels are turning as the data and analytics train picks up steam. Many executives are on board with a heightened focus on data-driven decision making. They're expecting key questions to be answered, backed by data. Sales, marketing, customer service, and other key functions are working hard to advance their data and analytics strategies. If you work in learning & development, at some point you'll need to do this, too.
For too many years, L&D has generated a limited set of LMS and survey data. The data could only answer basic questions like how many courses and classes are out there, how many people are taking them, whether people are mastering the content, and what participants and other stakeholders think of the training. These data may show a lot of activity, but don't really explain L&D's value to the organization.
Things get even trickier when you move beyond training to a more comprehensive learning and performance ecosystem. In addition to the LMS and survey data, there are knowledge bases, social networks, performance support tools, adaptive learning, microlearning, augmented and virtual reality-based learning, serious games, learning record stores, and so on. All of these systems generate their own data.
How can L&D get value, and more importantly, deliver value from its learning data?
What is learning data?
In an academic context, learning data describes an individual student's academic and behavioral deficits and their progress toward mastery of academic content. In a business context, learning data describes the enterprise's interventions, whether they're being used by the intended people, and their impact on the success of the enterprise.
Learning data differs based on the nature of the content. For example, course data includes enrollments, completions, and credits. Knowledge-base data includes searches, content accesses, ratings, and reviews. A social network's data includes connections, memberships, posts, replies, likes, and follows. Serious game data includes milestones, points, and badges. Performance support data relates to the processes, steps, tasks, and subtasks for which people are seeking guidance.
Data is a powerful tool that can inform your decisions and actions, as well as those of your executive sponsors. Data can drive continuous improvement in your products and services. It can also provide evidence of impact on your company's success. To realize these benefits, you'll need a data strategy.
Establishing your learning data strategy
A data strategy describes the questions that must be answered and the data needed to make decisions and take actions that result in a business advantage. Here are seven steps to establish a data-driven learning strategy for L&D
Step 1: List the questions you want to answer
Start by formulating the key questions your executive sponsors would most like you to answer. How much learning is going on? How much are we spending on learning? Is the learning managed efficiently and cost-effectively? Does L&D's capacity match our organization's demand for learning? What business challenges are addressed by L&D's learning products? Is the learning having a positive impact?
Step 2: Determine what data you have and where it is
Identify data that can inform your answers to these questions and conduct a data inventory. Determine which data already exists and what is missing. Identify the systems and people that are the sources of the data. Identify the filters needed to focus in on relevant data; such as date range, user type, content type, etc. Be sure to consider your organization's personally identifiable information (PII) policies and regulatory requirements when considering how to best collect, aggregate, and report learning data.
Step 3: Design your dashboard and reports
Define the specifications for reports and dashboards that would help you answer the questions you identified in Step 1. Create a drawing or wireframe for a dashboard. These drawings and specifications will drive implementation requirements.
Step 4: Define your data architecture
You may need to collect learning data from multiple sources. For evidence of impact, you may want to look at enterprise metrics, such as key performance indicators (KPIs) or number of compliance incidents side-by-side with your learning metrics. You may want to establish a data warehouse that aggregates data from multiple learning systems into a central place where it can be organized, tagged, and queried. It could sit inside your IT department's business intelligence (BI) platform or within your own learning record store (LRS). You can use data visualization software to create dashboards that link to your data warehouse and other enterprise systems.
Step 5: Develop and implement mechanisms to collect the data
This may involve integrations that extract the data from source systems. It may also involve incentives, policies, procedures, and tools for people to move data out of their personal spreadsheets into an information system. You may start by collecting the data that is easiest to get and use a phased roadmap to collect the remaining data.
Step 6: Develop and implement a taxonomy to organize and tag the data
The filters you identified in Step 2 can be applied as metadata for tagging content and users. Think about any additional ways you'll need to slice the data. Put on your sales and marketing hat and define your customer segments. Consider how to best describe each customer segment, e.g., job role, department, region, or years of experience. Identify the product categories used by each segment. Consider how to best categorize your products, e.g., topic, format, distribution method, level of detail, etc. If you apply this taxonomy to your learning data, you'll be able to explore the people-content, people-people, and content-content relationships.
Step 7: Develop and implement reports and dashboards
Revisit and update your designs from Step 3. Think about the best at-a-glance view of your data. Don't clutter your dashboard with too much detailed data. Show no more than eight or nine items. A good rule is to be able to find any dashboard data item within five seconds. Use reports for detailed data.
Repeat...
Repeat Steps 2-7 iteratively until you've fully answered all the questions listed in Step 1. Periodically revisit Step 1 to expand or reframe the questions you want your metrics to answer.
Exploring your data
As you begin to explore your data, you may find a surprising pattern or trend, and hypothesize its meaning. To test your hypothesis, you may need to explore the data further or conduct interviews and focus groups with users.
For example, let's say you have built a Compliance Dashboard that shows compliance training results (learning data) side by side with compliance incidents (business data.) You notice that over the last six months, compliance incidents at work locations in one region have been much higher than those in the other regions. So you decide to see what the learning data shows.
Scenario 1: Training numbers are down in that region compared to the others. One would hope that training would result in fewer incidents. If, over the next six months training goes up and incidents go down, you may have compelling evidence of a positive impact on the business. You can explore this by talking with people in that region to learn more about what factors might be causing the improvement.
Scenario 2: Training numbers are equal to, or higher than the other regions. Uh-oh, you've found an inverse relationship between training and incidents! Definitely time to go talk with some people, perhaps in all regions. Find out whether the training is effective. Are there adequate examples and case studies? Are they in the right context? Are people engaging in the learning program or just clicking through? And what else is happening in the region with all those incidents?
While you're analyzing your data, take care to manage the expectations of your sponsors. Depending where you are on your roadmap, you may only have preliminary answers to some of their questions based on partial data. Be sure to explain any shortcomings of the data behind these preliminary answers, along with your plans for a more complete picture in the future.
Concluding thoughts
Informed decisions backed by good learning data will drive continuous improvement, reveal evidence of impact, and steadily increase the value delivered by L&D. If you haven't unlocked the value of your learning data, it's time to start working on your data strategy now.