Few would dispute the convenience, low cost, and high efficiency of learning through technology. Whether eLearning, blended-learning, or mobile learning, it is usually just in time, just enough, and just for the user, which is the ideal form of customization and convenience for participants. At the same time, for larger audiences, eLearning represents a tremendous cost savings.
Executives have learned to love eLearning for its convenience, its timeliness, and its low cost. Learning through technology makes it possible for large numbers of people to learn at the same time without leaving the workplace. What executive would not love this? The problem is that the results are not always there.
Some professionals measure the ROI for eLearning on the basis of cost savings alone, comparing eLearning to the facilitator-led counterpart. When large numbers of participants are involved, this cost savings is very dramatic. This is acceptable only if the outcome would be the same from learning through technology when compared to the facilitator-led alternative. Unfortunately, that’s not often the case.
The challenge
Consider this real example: Each year, the global sales team at a major computer company launches over a hundred (the numbers are rounded for convenience) new products, services, and upgrades. The challenge is to prepare the sales force to sell them. A few years ago, facilitator-led product training was offered regionally, monthly. Now, eLearning or mobile learning is developed for each product. For every module, Level 3 (application) and Level 4 (impact) objectives are developed. For one product, a Level 4 objective was for 80% of participants to sell the product in one month. For the classroom version, this would mean that 20 participants out of 25 would sell the product in the first month. At this rate, a 150% ROI is delivered when compared to costs. For the eLearning version, only 20% achieved a sale in one month. This would mean that five of the 25 participants would sell the product in the first month. However, because of the low cost of eLearning, the ROI is 450% (rounded). An obvious question is: Which is best? The answer is not so comfortable. A higher ROI is achieved with eLearning, but with less business impact. That is disturbing for the clients and the sponsors who must pay for this.
Based on dozens of ROI studies at the ROI Institute, the results at the application and impact levels are usually less than traditional facilitator-led versions. It does not have to be this way, but the studies suggest that’s the case.
Types of data
It is helpful to think about the types of data that you can collect from any type of delivery process, whether it is facilitator-led, eLearning, blended learning, or mobile learning. These are shown in Figure 1.
Figure 1: Status of measurement
Regardless of mode of delivery, these measurement options are possible. In our studies through the ROI Institute, we have seen mixed results in terms of Level 1, when comparing eLearning and facilitator-led learning. Reaction measures with technology are lower, while learning measures with technology are often higher when compared to facilitator-led. A significant drop off occurs at Level 3 (application). Participants in technology-based learning are less likely to actually use what they have learned. At Level 4 (impact), a reduction appears that parallels the Level 3 reduction.
Causes
Why is this the case? Our analysis points to these issues:
- Facilitator-led learning is more likely to have application and impact objectives (Levels 3 and 4). A facilitator can teach to those levels of outcome as he or she describes the on-the-job application and places emphasis on achieving impact.
- In facilitator-led learning, a personal bond is often developed with the facilitator and with other participants. This commitment to achieve results, beyond the classroom, is often an important part of the program and the learners are more likely to apply the content.
- Application tools, built into the program, are more likely to be a part of facilitator-led learning than technology-based learning. These tools enable the participants to actually use the content in the job setting.
- In technology-based learning, the participant’s manager is left out of the process, for the most part while in facilitator-led learning, the role of a participant’s manager is often clearly defined. In many studies, the manager is not even aware that the participant is in the technology-based program. Much research has shown that involvement before and after the program are the two most powerful interactions to transfer the learning to the job.
- Because technology-based learning is convenient and sometimes optional, learners do not always take it seriously from a learning perspective. Participants will take the required courses, and may take them even if they are not required. Because of the impersonal nature of the interaction with the tablet, smartphone, laptop, or desktop, it is less likely that the participant will actually take action and make a difference.
Not only have we seen this in studies with our clients, but we see it dramatically in our own work. We offer the ROI Certification as a facilitator-led five-day learning program. This same program is available completely online and it takes 26 weeks, with weekly modules that must be completed, live chats, and a message board. While the content is the same, the eLearning version dives more deeply into the content. With a test at the end of each module, the participants know that they have learned. However, the results at Level 3 are dramatically different. For those in the classroom learning, over 50% actually obtain the certification, which means they have to complete an ROI study to achieve results (Level 3). For the online learning version, the result is less than 5%. For the live version, about 700 people complete it each year and for the online learning about 300 complete it—a total of 1,000. We have a consistent complaint from those who participate in the eLearning version, “I know much about this methodology, but I’m not sure I know how to do an ROI study.” Although the same content and the same application tools are built in, what’s missing is the opportunity to interact with other participants, to work together on projects, to learn from each other, and to present their ROI plan in front of others. With the facilitator-led version, there is significant bonding with the facilitator. When there is a follow-up reminder, it is more personal because they know the facilitator and they do not want to disappoint him or her and the rest of the group.
What is your experience?
We encourage others to conduct studies and compare eLearning with facilitator-led programs, evaluating at Levels 3 and 4, to see the difference. These evaluations could be pushed all the way to ROI (Level 5), if necessary. The methodology that should be used to measure the application, impact, and ROI, is the ROI methodology. It is the most-used evaluation system in the world. Figure 2 shows the ROI methodology.
Figure
2: ROI model
Tips for designers and developers
To ensure that there is appropriate application and impact from technology-based learning, you need to take some specific steps. We know from our own experience that learning through technology can drive business impact. We have ROI studies to show this. To do so requires addressing several issues. Here are six important areas:
1. Develop application and impact objectives.
Application objectives describe what the participant should do with what they are learning. The impact objective describes the consequence of the application. These are powerful objectives that link the program to the bottom line. Although rarely presented in eLearning or mobile learning, they can be developed in almost any learning process.
2. Develop content to reflect application and impact objectives.
Because objectives have moved beyond learning, it is critical to position the content to focus on application and impact, essentially relaying the message that the program is not successful unless application occurs and there is a corresponding impact. This is reinforced with case studies, videos, demonstrations, exercises, games, and examples to help with application and impact.
3. Design for application and impact.
As you design the programs, build application tools into the process. Include action plans, checklists, guides, templates, and other tools. This is critical to ensure that application and impact occurs. These tools facilitate the use of what participants are learning and suggest that application and impact are absolutely necessary.
4. Create an expectation for results.
Sometimes it is simple to define success. Participants must know that success is not achieved when they have learned the content, but only if there is a corresponding application and impact. This effort goes beyond developing objectives and focuses on emphasizing and describing program success. Part of this may include defining responsibilities that shift the participant’s responsibility beyond participating in the program. Yes, it is important to participate in the program, complete it, and learn what is necessary to be successful. But it is more important to actually apply it and have an impact. This is part of their responsibility. One organization included a statement of responsibilities at the beginning of each technology-based learning program that defined what the CEO saw as the participant’s responsibility in the program.
5. Involve the participant’s manager.
Almost any research on the transfer of learning to the job, will underscore the importance of the participant’s manager in achieving results. For the most part, technology-based learning programs seem to ignore the manager. Sometimes, participants take programs on their own time and the manager does not know about it. Still, there are times when it is helpful to let managers know that their employees will be involved. Provide a few low-level activities to encourage manager involvement, both before participants actually become involved in the program and after they have completed it.
6. Apply the principles of learning transfer.
Perhaps there is no area within learning and development that is as important as the transfer of learning to the job. Most typical actions work well in the learning-through-technology environment. Several valuable references on transfer strategies are available to ensure that learners use the learning on the job, such as Beyond Transfer of Training: Engaging Systems to Improve Performance by Mary L. Broad and Making eLearning Stick: Techniques for Easy and Effective Transfer of Technology-Supported Training by Barbara Carnes.
Summary
ROI for learning through technology can be an elusive number, but it does not have to be. Unfortunately, many technology-based learning evaluations are revealing deficiencies with on-the-job success, making it difficult to convince management that eLearning or mobile learning is adding significant value. Learning through technology is convenient, timely, and in most cases, extremely low cost—sweet words to an executive. However these same executives want results. They need assurance that learning through technology will drive application and impact.
By following some prescribed steps, designers and developers can make a huge difference in the success of eLearning, pushing it to impressive levels of success for application and impact. When this is the case, then you can easily base the ROI on the cost savings generated when comparing learning through technology to a facilitator-led counterpart.