Providing eLearning is not sufficient; L&D professionals need a way to measure its effectiveness. Recognizing this need, Donald Kirkpatrick published a four-level model for evaluating courses in 1959, later updated as the “New World” Kirkpatrick Model. In a 1996 paper, Jack J. Phillips, founder of the ROI Institute, suggested adding a fifth level to the Kirkpatrick levels of evaluation to measure ROI, a company’s return on its investment in training.
The Kirkpatrick Model is still a widely used method of evaluating training courses. The model can be applied before, during, and after training, and it provides a way of assessing eLearning effectiveness; the fifth level also makes it possible to perform a cost-benefit analysis to determine whether an investment in training has paid off for the company.
The Kirkpatrick levels of evaluation are described below.
Level 1: Reaction
Level 1 looks at how learners respond to eLearning or other training. Learners might receive “smile sheets” or other means of soliciting their responses. Questions might pertain to whether they enjoyed the eLearning and found it engaging. Level 1 questions might also ask learners to describe how useful they found the training to be and whether they believe that they will apply it in their work. At level 1, managers also attempt to measure the degree of learner engagement with and contribution to the learning effort. Positive reactions are regarded as essential to continued learner engagement in subsequent phases of training.
Level 2: Learning
Level 2 evaluates learning of a skill, acquisition of knowledge, or adoption of an attitude presented in the eLearning. At level 2, learning is generally measured using a test or some other assessment instrument. The test can be formal, informal, or even a self-assessment. Ideally, learners will have taken a pre-test whose results can be compared with this post-learning assessment. Assessment might be written, based on observation, or gleaned from interviews with learners. The New World Kirkpatrick Model adds confidence and commitment to level 2, assessing whether the learner believes that she can perform the new skills and whether she intends to apply the learning on the job.
Level 3: Behavior
At level 3, managers evaluate behavior change. Are employees doing things differently at work since completing the eLearning? Are they applying the training? Have they adopted new attitudes or beliefs?
This can be difficult to measure, and, according to Ed Forest’s article on the Kirkpatrick Model (see References), level 3 evaluation begins only three to six months after training. L&D professionals or learners’ managers might build in on-the-job processes that encourage, reinforce, or reward the desired new behaviors and attitudes. These could take the form of performance support tools, possibly delivered in a similar way to the initial eLearning.
Level 4: Results
Level 4 evaluation of results relies on clear goals having been established prior to training. What changes were managers looking for? Did they identify specific outcome targets? If the training is intended to reduce churn at a call center, for example, what specific targets were identified concerning increased longevity of employees? How is success determined? Ideally, the targets are clear and measurable, enabling managers to compare pre- and post-learning performance or results. Phillips wrote, “Kirkpatrick defines Level 4 evaluation as the results linked to training. These results could take the form of reduced absenteeism and turnover, quality improvement, productivity, or even cost reduction.”
Level 5: ROI
Phillips goes on to state that level 4 lacks a specific monetary calculation; this is why he suggested adding a fifth level. Kirkpatrick level 5, in Phillips’s view, would measure the monetary benefits seen as a result of the changed behavior against the cost of creating and offering the training. “The fifth level of evaluation is developed by collecting level 4 data, converting the data to monetary values, and comparing them to the cost of the program to represent the return on training investment.”
Evaluating eLearning can be challenging but valuable; applying the Kirkpatrick levels of evaluation offers L&D teams and managers a way to determine whether their training efforts are paying off in terms that matter most—the company’s bottom line.
References
Forest, Ed. “Kirkpatrick Model: Four Levels of Learning Evaluation.” Educational Technology.
Kirkpatrick Partners. “The New World Kirkpatrick Model.”
Phillips, Jack J. “Measuring ROI: The Fifth Level of Evaluation.” Technical & Skills Training, April 1996.